Testing using TTCN-3

TTCN previously referred to Tree and Tabular Combined Notation. This was understandable because test cases were in tabular formats. They contained many levels of indentation that could be regarded a tree-like structure. With TTCN-3, the abbreviation refers to Testing and Test Control Notation. The focus is on testing and not really how those test cases are written. Yes, we can still write test cases in the old way of TTCN-2 but that’s not the only way.

Figure 1 gives an overview of TTCN-3 [1]. As we can see, test cases can be written directly in TTCN-3 core language (such a concept did not exist in TTCN-2), in tabular format or in graphical format. The standard also allows for newer presentations that could interface with the core language. For example, it’s perfectly valid for someone to write test cases in XML and have a conversion mechanism to the core language. Needless to say, an XML presentation format will remain proprietary with no tool support unless it gets standardized.

Figure 1: TTCN-3 Overview

The second fact that becomes obvious from Figure 1 is that the core language interfaces with different other languages. These interfaces facilitate the reuse of existing data types and definitions that might have been defined in those languages. For example, UMTS RRC signalling definitions are in ASN.1. For the test engineer, there is no need to convert such definitions into TTCN-3. Any respectable tool in the market must be able to interface directly to these definitions and handle them seamlessly as part of TTCN-3 core language implementation.

Language

At this point it is appropriate to see what’s the format of TTCN-3 core language. This is nothing more than simple text with well-defined syntax and semantics. The syntax is defined using Backus-Naur Form. What this means is that any text editor can be used to write TTCN-3 test cases. Such test cases are quite different in dynamic behaviour from C or Pascal. Still, it is quite easy for programmers well-versed with procedural languages to get used to TTCN-3 easily. There are many similarities - keywords, data types, variables, control statements, functions, operators, operator precedence, just to name a few.

Looking at the differences between TTCN-2 and TTCN-3, Table 1 illustrates an important point with regard to indentation. In TTCN-2, many levels of indentation lead to poor code readability and excessive scrolling in editors. With each alternative, there is code duplication (S4) which can be solved only if S4 is implemented in a reusable test step. Alternatives in TTCN-3 are more elegantly specified and control flow continues at the same level of indentation. Even the example in Table 1 can be simplied by defining default alternative behaviour earlier.

Table 1: TTCN-2 vs TTCN-3 Statements

Having the core language in text also makes it easier to look at differences in a version control system. At run time, it makes debugging at the level of TTCN source a lot easier. This is important for test case developers. I have never known anyone who did any similar debugging at TTCN-2 source. The best I have seen was engineers setting intermediate verdicts at lots of places to ascertain what went wrong and where.

The language is structured in a way that allows high level of flexibility. Test system definition is modular. In fact, an important unit of a test suite is a module which would contain one or more test cases or the control part of a test suite. Concurrency of operation is possible because components can execute in parallel. Of course, execution is serialized at the level of hardware unless the multi-processors are involved. Parameterization is possible just as it was possible in TTCN-2. Concepts of PICS and PIXIT still apply because they are fundamental to any conformance testing.

Test System

Figure 2 represents the test system based on TTCN-3 [2]. The modularity of the design is apparent. Adapters are distinct from the executable. Test management and codecs are distinct entities that interface to the executable. More importantly, interfaces TCI and TRI are standardized so that users have a choice of easily migrating from one tool vendor to another without needing to rewrite the test cases. TTCN-3 Control Interface (TCI) allows for interfacing to codec (TCI-CD) and to test management (TCI-TM). Likewise, TTCN-3 Runtime Interface (TRI) interfaces to the adapters. This interface does the translation between the abstraction in TTCN-3 and the behaviour in runtime.

Figure 2: TTCN-3 Test System

The adapters are implemented in ANSI C or Java, which have been included in the standard. TTCN-3 allows for dynamic mapping of communication channels between the TTCN-3 executable and the adapters. This is one more area in which TTCN-3 does it better than TTCN-2 where such mapping was static.

Typical Test Cycle

The following would be present in a typical test cycle:

• Implement the adapters in a chosen language (done only once per adapter per language of choice)
• Implement the encoders/decoders in a chosen language (done only once per language of choice)
• Implement the test cases in TTCN-3 (done only once)
• Compile the test case and test suite (done only once unless test cases change) - at this stage an executable is formed from the abstract definitions
• Link with adapters, codecs and test management (varies with tool implementation: may be a static link, runtime loading of library or inter-process communication)
• Execute the test suite (debug if necessary)
• Collate test results and correct the IUT (Implementation Under Test) if errors are seen

Tools

I have previously used tools from Telelogic but never really liked their GUI. Their tools have generally been the least user-friendly in my opinion. I hear from others who have evaluated their TTCN-3 support that they are now better. Telelogic is not doing just TTCN-3. They do a whole of things and I think their strength in TTCN-3 is not all that obvious.

Recently I evaluated TTWorkbench from Testing Technologies. It’s an excellent tool - easy to install and easy to use. It has good debugging support. It allows for test case writing in graphical format (GFT) and looking at logs in the same format. Naturally it also allows writing of test cases in core language format. The downside of this tool is that I found it to be slow in loading and building test suites. It uses Eclipse IDE.

Next I evaluated OpenTTCN. “Open” refers to openness of its interfaces which conform to open standards. This allows the tool to be integrated easily to other platforms using standardized TCI and TRI. With this focus, the tool claims to conform rigidly to all requirements of TTCN-3 standards. Execution is generally faster than other tools in the market. The company that makes this makes only this. Nearly 14 years of experience has gone into making this product and the execution environment is claimed to be the best. The downside is that the main user interface is primitive command line interface. There is no support for GFT although this is expected to arrive by the end of the year. Likewise, debugging capabilities are in development phase and are expected to be rolled out sometime this year. OpenTTCN also relies on certain free tools such as TRex that is the front-end editor with support for TTCN-3 syntax checking. This too is based on Eclipse.

This is just a sample. There are lots of other tools out there. Some are free with limited capability and others are downright expensive. Some are proprietory. One example in this regard is the General Test Runner (GTR), a tool used in Huawei Technologies.

Conclusion

TTCN-3 is set to become a major language for formal test methodology. WiMAX is using it. SIP tests have been specified in TTCN-3. LTE is starting to use it. Other telecommunications standards are using it as well and its use has split over to other sectors. The automotive sector is embracing it. AutoSAR is using it and these test cases may be available quite soon this year. The official website of TTCN-3 is full of success stories.

It is not just for conformance testing like its predecessor. Its beginning to be used for module testing, development testing, regression testing, reliability testing, performance testing and integration testing. TTCN-3 will work with TTCN-2 for some time to come but for all new test environments it will most likely replace TTCN-2 as the language of choice.

[get this widget]