Assertion.ru

Conformance test development involves identifying assertions in a specification, writing conformance tests that check the identified assertions and linking the test to the assertion that it tests.
Lets start from the following points:
- assertion is marked
- actual assertion is difficult to view in the spec (currently there are only small assertion gif’s at the end of each assertion)
- complete assertions are only viewed by reading the html directly or looking at each individual test
- start of assertions are difficult to see in the html code
- providing a visual way to view the assertion easily is teh problem we are trying to solve.

The main point is to color the assertions (specification text itself) using html tags. The research was made which html tags to use. Div, span, table and font tags were looked at. The best solution is the font tag. So the text is surrounded with font tags. The class attribute of the font tag corresponds to the type of the assertion. F.e. if the asserion is new it is colored with red, to indicate, that tests need to be written, old assertions are colored with green to indicate that tests already exist. There should be a utility (script or java program) to scan marked up specification and automatically add the tags needed for coloring.  The  background color of the text will be determined by the title attribute color of the assertion. This method was implemented and works fine. For usability purposes, there should be a mechanism to hide coloring, f.e. a javascript.

A disadvantage of this solution is that the color is static since it is based on the title attribute.  A second solution would be that the tool would  check for an existence of a test (based on the assertion id or link in the assertion).  If a test exists, we would do something to set  the color of that assertion.  It could be as simple as setting a title attribute. A disadvantage to this solution would be that the assertion coloring would be still static, but based on when the user run the scripts.

A variation on the given solution is that we would dynamically generate the coverage data when the spec is viewed in a browser. We would determine if a test exists in the test directory for a given assertion and color the assertion accordingly. This could be done through a javascript/vbscript using objects, that allow accessing file system. This method would be dynamic and should always have the latest assertion coverage status.

Here are some examples from JLS3 chapters “Conversions and Promotions” and “Interfaces” :

Assertions conv063, conv047, conv065, conv48, conv66 and conv049 are from previous version of spec, they weren’t changed and tests update is not needed – color is aquamarine (neurtal green). Conv155 and conv156 are new, new tests should be developed, assertions are colored in glaring red. Conv064 was changed, test update is necessary – color orange. Annot019 is a new one, tests exist, but they are needed to be changed – salmon color. Annot020 is new, but кудумфте tests exist – color light green.

The main advantage of spec coloring is that the spec is visualized. User can see the whole assertion and its title. One can tell by looking at the spec, where there are areas with low coverage, where some or lots of  tests should be added or changed. There is basically the possibility to see how well a spec is marked up and how well it is tested.

Sometimes all this terms seem quite confusing. Or it’s better to say they usually or even always do. Especially when different people and companies understand the situation around them in different ways. So let’s start from the very beginning.

There are lots of specifications or standards around us. Webster dictionary describes standard as something established by authority, custom, or general consent as a model or example. So basically it is the list of rules, which others while using have to obey.

Let’s create a computer language. First of all the specification is needed, that will describe the whole concept model, tell developers what can be written as a program, how it will behave, what will be compiled, executed and etc. After creating all these necessary documents – that’s it, one can stop right there. If the idea is good enough several other companies might want to create their implementations: f.e. compilers and executing environments. But they must obey the specification. Otherwise the same programs will run on one and run differently or even fail on the other implementation. Fulfillment by an implementation of all specified requirements is called conformance.

Why is it so important? Well, let’s say this new language was used to create a program for stock exchange. Imagine it was written in the US, well tested and used at NYSE. It was so good, that other countries all over the world bought a license and started using it on their implementation of this new language. If an implementation didn’t obey the spec, the same program might do different things with clients’ money. Basically this stock exchange program might sell when commercial agent pushed the “buy” button, or buy steel instead of fruit.

The good question could be: “Why different implementations? Let’s create one and use it.”. There are different answers. Several companies might want to use this language on different platforms (Solaris, Linux, Windows) and devices (desktop, mobile phone, PDA, student calculator, etc). Others want to optimize algorithms for their needs, f.e. implement them so the big database program will be 10 times faster.

The key point is that several different implementations must work exactly the same and according to the spec. If they do so they are called compatible. The bad part is that no one can be sure. That is why the verification mechanism is needed. Usually it is a test suite that verifies the conformance and compatibility. And in this case it is wrong to say that something is almost compatible or 99% compatible. There could be either yes or no.

Let’s move towards an example. Sun Microsystems invented Java language. To be more precise several versions of Java for different markets were created. The most famous are Java ME, Java SE and Java EE. And for each of them there is a separate specification. Sun Microsystems has its own implementation, which is most commonly used. However the language is so good, that there are quite a few other companies and their implementations. For compatibility and conformance purposes there are TCKs (Test Conformance Kit). TCK is a product that includes a number of tests, which check if an implementation is correct according to the rules from specification.

My intend was to give an idea of what specification, conformance, compatibility and TCK are and why they are so important.