Specification, conformance, compatibility, tests… What are they all about?

Sometimes all this terms seem quite confusing. Or it’s better to say they usually or even always do. abstract Specification, conformance, compatibility, tests... What are they all about?Especially when different people and companies understand the situation around them in different ways. So let’s start from the very beginning.

There are lots of specifications or standards around us. Webster dictionary describes standard as something established by authority, custom, or general consent as a model or example. So basically it is the list of rules, which others while using have to obey.

Let’s create a computer language. First of all the specification is needed, that will describe the whole concept model, tell developers what can be written as a program, how it will behave, what will be compiled, executed and etc. After creating all these necessary documents – that’s it, one can stop right there. If the idea is good enough several other companies might want to create their implementations: f.e. compilers and executing environments. But they must obey the specification. Otherwise the same programs will run on one and run differently or even fail on the other implementation. Fulfillment by an implementation of all specified requirements is called conformance.

Why is it so important? money coins Specification, conformance, compatibility, tests... What are they all about?Well, let’s say this new language was used to create a program for stock exchange. Imagine it was written in the US, well tested and used at NYSE. It was so good, that other countries all over the world bought a license and started using it on their implementation of this new language. If an implementation didn’t obey the spec, the same program might do different things with clients’ money. Basically this stock exchange program might sell when commercial agent pushed the “buy” button, or buy steel instead of fruit.

The good question could be: “Why different implementations? Let’s create one and use it.”. There are different answers. Several companies might want to use this language on different platforms (Solaris, Linux, Windows) and devices (desktop, mobile phone, PDA, student calculator, etc). Others want to optimize algorithms for their needs, f.e. implement them so the big database program will be 10 times faster.

The key point is that several different implementations must work exactly the same and according to the spec. If they do so they are called compatible. The bad part is that no one can be sure. That is why the verification mechanism is needed. Usually it is a test suite that verifies the conformance and compatibility. And in this case it is wrong to say that something is almost compatible or 99% compatible. There could be either yes or no.

no bug2 Specification, conformance, compatibility, tests... What are they all about?Let’s move towards an example. Sun Microsystems invented Java language. To be more precise several versions of Java for different markets were created. The most famous are Java ME, Java SE and Java EE. And for each of them there is a separate specification. Sun Microsystems has its own implementation, which is most commonly used. However the language is so good, that there are quite a few other companies and their implementations. For compatibility and conformance purposes there are TCKs (Test Conformance Kit). TCK is a product that includes a number of tests, which check if an implementation is correct according to the rules from specification.

My intend was to give an idea of what specification, conformance, compatibility and TCK are and why they are so important.



, , , , , , , , , , , , , , , , , , ,
       

    99% – is it enough or not?

    99%Today is a great day. I will try to explain why. As I mentioned in my intro-post our team is creating several different TCK’s. The area that I work on is so-called LANG – I develop tests for Java Language. Long ago, more than 2 years from now, we started to work on JLS 3 specification. We had to solve many problems which often occur during spec change (I promise to write more about that). Our team is finishing JCK 6a, lang tests is part of this JCK. Today I run the coverage scripts and we can finally say that we have 99% assertion coverage for JLS 3. To be more precise we have 99.4%. It means that we wrote tests for 99% of sentances in JLS 3 that we had marked as potentially testable. Isn’t it cool? I bet it is!

    The work is certainly not over yet and will not be so - there are many reasons why more tests are needed :

    • depth coverage improvement – more tests for several assertions are needed;
    • there are sentances that are testable, but for several reasons we hadn’t marked them as potentially testable;
    • there will be JLS 4 soon, we should start working on it as soon as possible.

    Different people might have opposite answers for a question in a title.  Most would say "Yes, of course". Indeed 99% is almost 100%. And what is 100% – it is a perfection. 99% looks great, and it is great. But we must understand what this number stands for, and what can be improved. My opinion is "yes, it is great, colossal, tremendous; but no, it’s not enough, I want more, even more than 100%", that’s why I plan to create a script for depth coverage calculation.

    Thanks to all SUN developers who did JCK-Lang work, thanks to people who helped (especially to compiler team) and certainly great thanks to all developers who use Java :-)

    Java world became even more compatible and safer!



    , , , , , , , , ,
         

      Hello, world.

      I do read weblogs. I started reading them much more. Why am I doing this? Well.. I’m a believer that information sharing does make other people’s work more fun. And not only work, but people lives overall. Sharing knowledge and thoughts is a big step towards building communities. Weblog allows to publish thoughts, opinions, ideas, concerns to the readers. While the readers can open a dialog on a discussed topic using comments.
      I wanted to create my personal weblog long ago, but I guess I finally made it.
      I work as a software engineer in a group that creates a very important product. This product is not as well-known as Java or Solaris. I’d say most of software engineers in this huge world do not know about it. However without this product there will be no Java as we know it. At least Java will not be as good, as popular and as wide-spread as it is and as we want it to be. Without this product there will be no WORA (write once run anywhere). Yes, I’m talking about compatibility. And the product is a TCK (Technology Compatibility Kit) for Java. In two words TCK is a test suite, which makes sure that implementations of Java conform to the specification. My group works under several TCK’s. But the main one is for Java SE platform.
      There are several different areas in JCK (TCK for Java SE): API, Java Language and VM.
      My area that I work on for more than 2 years is "Java Language". I write compiler and runtime tests to make sure that Sun’s and other companies’ implementations behave exactly as it is said in the JLS (Java Language Specification).
      In this weblog I will try to share my knowledge, my opinions and thoughts regarding my work.
      PS: English is not my native language. That’s why I would like to apologize for possible mistakes (I’m sure there are many of them already) and weird sentances in my weblog. I’ll try to do my best.



      , , , , , , , , , , , , , , ,