Make it simple?

2plus2 Make it simple?Specification is important – this statement is clear to everyone. A widely used product, technology or language without a specification is useless. A specification without a testsuite is dangerous. A testsuite without markup and tests is impossible. This process is quite complex. However there are ways to simplify the markup stage.

As for Java Language Specification (JLS) and Java Virtual Machine Specification (JVMS) they are written in FrameMaker. Afterwards spec is exported to html and pdf. The markup is embedded into html version. My opinion is that markup information should be placed into (or connected with) the origin text. In our case it is FrameMaker document. I’m not sure that this is possible at all, but my guess it is. If not, maybe FrameMaker is not the best solution. As a result we will significantly reduce the amount of time and effort needed for transferring old markup and marking up new text. Moreover during writing the next revision of spec the auther together with tck team should markup all chenged and new assertions. I’d say the best way is when the spec writing and the markup processes are done at the same time. It is reasonable for the auther to point out the test developers what statements should be tested.



, , , , , , , , , , , , , ,
       

    Assertions and markup

    book Assertions and markup

    It is very important to have a good process while writing the test suite. I will talk about the one that was used for JLS.

    As mentioned before the final product is the number of tests. There is a relation between the tests and the specification. The assertion-driven process gives an idea of what each group of tests actually checks in the specification. Using this relation the developer can calculate the coverage, get the list of assertions on which the tests were not written, etc.

    Assertion is a statement from a specification that can be tested. And the first step is to identify all assertions in the specification. After that the developer can write tests.

    Example of assertions from the Java Language Specification:smt Assertions and markup

    • A compile-time error occurs if the same modifier appears more than once in an interface declaration.
    • The binary name of a member type consists of the binary name of its immediately enclosing type, followed by $, followed by the simple name of the member.
    • A continue statement may occur only in a while, do, or for statement.

    There could be many statements that are non-testable or involve uncertainty. Sometimes such statements include words like “possible” or “maybe”. It is not true that if a sentence has a word “may” it is not testable, but usually it is so.

    Examples of non-testable statements:

    • We do not recommend such “mixed notation” for array declarations.
    • Situations where the class of an object is not statically known may lead to run-time type errors.
    • If, however, evaluation of an expression throws an exception, then the expression is said to complete abruptly.

    There are many discussions and disputes about assertions. Some say that examples should not be treated as assertions. Others say that each statement is an assertions and there are two kinds of them: testable and non-testable. My personal opinion is that an assertion is certainly something testable. And in most cases examples are assertions just because the test can be written checking the particular example.

    The process of identifying assertions in the specification is called markup. There are many approaches. But in any case the user must be  able to get information on whether the statement is an assertion and somehow distinguish one assertion from another. There could be a separate repository with mapping of assertions and their id’s to statements. I like the idea of integrating the markup into the specification. This approach was chosen for the language area of the Java SE test suite. The JLS was written in FrameMaker. With export mechanisms the PDF and HTML versions were created. The html version was used during the creation of the test suite.

    In JLS and JLS 2 some special anchors identified the beginning and the end of an assertion. Additional information was the assertionID and short summary of the statement. The end anchor was an image and a link to the test. The html view and the code view are shown on the corresponding illustrations. The assertion id’s are arr033, arr034, arr020, etc.

    JLC2 html1 Assertions and markup

    JLC2 html code1 Assertions and markup

    The general idea can be described as:

    <a name=assertionID><!– shord description as html comment –>
    assertion statement here
    <img src=”pics/assert.gif”><a href=”path to test”>test ID which is the same as assertion ID</a>

    If seperate statements in different parts of specification are tested by one test the first tag will be something like arr033_0, arr033_1, arr033_2.

    This kind of architecture was used for JLS and JLS 2. It was slightly modified for JLS3, but the main idea was kept. I know some examples of approaches with non-static assertion IDs kept in a separate repository, where ID is some hash-value calculated based on the content. For several reasons it showed up to be not a very good solution. There is always a hard process migrating to the new version of the specification. But in my opinion it is much easier with the static ID’s embedded into the specification.



    , , , , , , , , , , , , , , , , , ,
         

      Specification, conformance, compatibility, tests… What are they all about?

      Sometimes all this terms seem quite confusing. Or it’s better to say they usually or even always do. abstract Specification, conformance, compatibility, tests... What are they all about?Especially when different people and companies understand the situation around them in different ways. So let’s start from the very beginning.

      There are lots of specifications or standards around us. Webster dictionary describes standard as something established by authority, custom, or general consent as a model or example. So basically it is the list of rules, which others while using have to obey.

      Let’s create a computer language. First of all the specification is needed, that will describe the whole concept model, tell developers what can be written as a program, how it will behave, what will be compiled, executed and etc. After creating all these necessary documents – that’s it, one can stop right there. If the idea is good enough several other companies might want to create their implementations: f.e. compilers and executing environments. But they must obey the specification. Otherwise the same programs will run on one and run differently or even fail on the other implementation. Fulfillment by an implementation of all specified requirements is called conformance.

      Why is it so important? money coins Specification, conformance, compatibility, tests... What are they all about?Well, let’s say this new language was used to create a program for stock exchange. Imagine it was written in the US, well tested and used at NYSE. It was so good, that other countries all over the world bought a license and started using it on their implementation of this new language. If an implementation didn’t obey the spec, the same program might do different things with clients’ money. Basically this stock exchange program might sell when commercial agent pushed the “buy” button, or buy steel instead of fruit.

      The good question could be: “Why different implementations? Let’s create one and use it.”. There are different answers. Several companies might want to use this language on different platforms (Solaris, Linux, Windows) and devices (desktop, mobile phone, PDA, student calculator, etc). Others want to optimize algorithms for their needs, f.e. implement them so the big database program will be 10 times faster.

      The key point is that several different implementations must work exactly the same and according to the spec. If they do so they are called compatible. The bad part is that no one can be sure. That is why the verification mechanism is needed. Usually it is a test suite that verifies the conformance and compatibility. And in this case it is wrong to say that something is almost compatible or 99% compatible. There could be either yes or no.

      no bug2 Specification, conformance, compatibility, tests... What are they all about?Let’s move towards an example. Sun Microsystems invented Java language. To be more precise several versions of Java for different markets were created. The most famous are Java ME, Java SE and Java EE. And for each of them there is a separate specification. Sun Microsystems has its own implementation, which is most commonly used. However the language is so good, that there are quite a few other companies and their implementations. For compatibility and conformance purposes there are TCKs (Test Conformance Kit). TCK is a product that includes a number of tests, which check if an implementation is correct according to the rules from specification.

      My intend was to give an idea of what specification, conformance, compatibility and TCK are and why they are so important.



      , , , , , , , , , , , , , , , , , , ,