Specification is important – this statement is clear to everyone. A widely used product, technology or language without a specification is useless. A specification without a testsuite is dangerous. A testsuite without markup and tests is impossible. This process is quite complex. However there are ways to simplify the markup stage.
As for Java Language Specification (JLS) and (JVMS) they are written in FrameMaker. Afterwards spec is exported to html and pdf. The markup is embedded into html version. My opinion is that markup information should be placed into (or connected with) the origin text. In our case it is FrameMaker document. I’m not sure that this is possible at all, but my guess it is. If not, maybe FrameMaker is not the best solution. As a result we will significantly reduce the amount of time and effort needed for transferring old markup and marking up new text. Moreover during writing the next revision of spec the auther together with tck team should markup all chenged and new assertions. I’d say the best way is when the spec writing and the markup processes are done at the same time. It is reasonable for the auther to point out the test developers what statements should be tested.
Conformance test development involves identifying assertions in a specification, writing conformance tests that check the identified assertions and linking the test to the assertion that it tests.
Lets start from the following points:
- assertion is marked
- actual assertion is difficult to view in the spec (currently there are only small assertion gif’s at the end of each assertion)
- complete assertions are only viewed by reading the html directly or looking at each individual test
- start of assertions are difficult to see in the html code
- providing a visual way to view the assertion easily is teh problem we are trying to solve.
The main point is to color the assertions (specification text itself) using html tags. The research was made which html tags to use. Div, span, table and font tags were looked at. The best solution is the font tag. So the text is surrounded with font tags. The class attribute of the font tag corresponds to the type of the assertion. F.e. if the asserion is new it is colored with red, to indicate, that tests need to be written, old assertions are colored with green to indicate that tests already exist. There should be a utility (script or java program) to scan marked up specification and automatically add the tags needed for coloring. The background color of the text will be determined by the title attribute color of the assertion. This method was implemented and works fine. For usability purposes, there should be a mechanism to hide coloring, f.e. a javascript.
A disadvantage of this solution is that the color is static since it is based on the title attribute. A second solution would be that the tool would check for an existence of a test (based on the assertion id or link in the assertion). If a test exists, we would do something to set the color of that assertion. It could be as simple as setting a title attribute. A disadvantage to this solution would be that the assertion coloring would be still static, but based on when the user run the scripts.
A variation on the given solution is that we would dynamically generate the coverage data when the spec is viewed in a browser. We would determine if a test exists in the test directory for a given assertion and color the assertion accordingly. This could be done through a javascript/vbscript using objects, that allow accessing file system. This method would be dynamic and should always have the latest assertion coverage status.
Here are some examples from JLS3 chapters “Conversions and Promotions” and “Interfaces” :
Assertions conv063, conv047, conv065, conv48, conv66 and conv049 are from previous version of spec, they weren’t changed and tests update is not needed – color is aquamarine (neurtal green). Conv155 and conv156 are new, new tests should be developed, assertions are colored in glaring red. Conv064 was changed, test update is necessary – color orange. Annot019 is a new one, tests exist, but they are needed to be changed – salmon color. Annot020 is new, but кудумфте tests exist – color light green.
The main advantage of spec coloring is that the spec is visualized. User can see the whole assertion and its title. One can tell by looking at the spec, where there are areas with low coverage, where some or lots of tests should be added or changed. There is basically the possibility to see how well a spec is marked up and how well it is tested.
It is very important to have a good process while writing the test suite. I will talk about the one that was used for JLS.
As mentioned before the final product is the number of tests. There is a relation between the tests and the specification. The assertion-driven process gives an idea of what each group of tests actually checks in the specification. Using this relation the developer can calculate the coverage, get the list of assertions on which the tests were not written, etc.
Assertion is a statement from a specification that can be tested. And the first step is to identify all assertions in the specification. After that the developer can write tests.
Example of assertions from the Java Language Specification:
A compile-time error occurs if the same modifier appears more than once in an interface declaration.
The binary name of a member type consists of the binary name of its immediately enclosing type, followed by $, followed by the simple name of the member.
A continue statement may occur only in a while, do, or for statement.
There could be many statements that are non-testable or involve uncertainty. Sometimes such statements include words like “possible” or “maybe”. It is not true that if a sentence has a word “may” it is not testable, but usually it is so.
Examples of non-testable statements:
We do not recommend such “mixed notation” for array declarations.
Situations where the class of an object is not statically known may lead to run-time type errors.
If, however, evaluation of an expression throws an exception, then the expression is said to complete abruptly.
There are many discussions and disputes about assertions. Some say that examples should not be treated as assertions. Others say that each statement is an assertions and there are two kinds of them: testable and non-testable. My personal opinion is that an assertion is certainly something testable. And in most cases examples are assertions just because the test can be written checking the particular example.
The process of identifying assertions in the specification is called markup. There are many approaches. But in any case the user must be able to get information on whether the statement is an assertion and somehow distinguish one assertion from another. There could be a separate repository with mapping of assertions and their id’s to statements. I like the idea of integrating the markup into the specification. This approach was chosen for the language area of the Java SE test suite. The JLS was written in FrameMaker. With export mechanisms the PDF and HTML versions were created. The html version was used during the creation of the test suite.
In JLS and JLS 2 some special anchors identified the beginning and the end of an assertion. Additional information was the assertionID and short summary of the statement. The end anchor was an image and a link to the test. The html view and the code view are shown on the corresponding illustrations. The assertion id’s are arr033, arr034, arr020, etc.
The general idea can be described as:
<a name=assertionID><!– shord description as html comment –>
assertion statement here
<img src=”pics/assert.gif”><a href=”path to test”>test ID which is the same as assertion ID</a>
If seperate statements in different parts of specification are tested by one test the first tag will be something like arr033_0, arr033_1, arr033_2.
This kind of architecture was used for JLS and JLS 2. It was slightly modified for JLS3, but the main idea was kept. I know some examples of approaches with non-static assertion IDs kept in a separate repository, where ID is some hash-value calculated based on the content. For several reasons it showed up to be not a very good solution. There is always a hard process migrating to the new version of the specification. But in my opinion it is much easier with the static ID’s embedded into the specification.
Specification is important – this statement is clear to everyone. A widely used product, technology or language without a specification is useless. A specification without a testsuite is dangerous. A testsuite without markup and tests is impossible. This process is quite complex. However there are ways to simplify the markup stage.
Conformance test development involves identifying assertions in a specification, writing conformance tests that check the identified assertions and linking the test to the assertion that it tests.
