TET-based LSB Test Suites

Introduction

This document is intended to be a primer on how to build or port testcases that are to be used in the TET/VSXgen framework utilised by the LSB test suites. It contains links to documents that are recommended reading and describes the structure which is specific to the LSB test suites.

Required Reading

If you are building TET based test suites I think there is probably no alternative but to first read the TET documentation. This document will make a lot more sense after you have done so.

The version of TET used by the LSB test suites is 3.3g. Further documentation on the use of TET and programmer APIs available for it are available from the TETworks website. The version of VSXgen used is 1.5 and the specification is available together with an overview of the current LSB test architecture .

Before attempting to write new TET based test suites for the LSB or convert existing test suites you should first familiarise yourself with the fundamentals of how to use TET. The TETware user guide is a good place to start and the source based LSB test suites can be used for experimentation of the various commands to build and run installed testcases. I recommend you do this concentrating on working out how to build and run individual testcases as well as getting the harness to output extra debug information. These skills are invaluable in debugging test failures.

If you are not familiar with the TET programming APIs you will also have to read the TET programmers guide. It describes in detail how to build TET based testcases and contains example code. TET supports C, C++, shell and Perl interfaces and it is possible to reuse some existing test programs by using the shell interface and wrapping existing test binaries.

Creating TET based test suites for the LSB

For testcases to be useful for certification testing they should meet several criteria:

The Test Specification
- The premise for all the tests is that they have to be based on a written, publicly available test specification.
- The test specification upon which the test suite is based shall conform to the POSIX.3 methodology (which means use of test assertions).
The Test Suite
- The test suite is aligned with the current draft/issue of the relevant specification. (Mandatory)
- The test suite must be able to establish conformance with all legitimate options and variables defined by the specification. (Mandatory).
- Each individually executable test must provide its own setup and clean up functionality (Mandatory)
- No individually executable test shall introduce side effects that will affect the results of any other tests(Mandatory)
- The unit of test execution should cover no more than a single interface function (Desirable)
- The test suite shall have a capability to check complete and correct configuration and installation. This may include a set of defined confidence checks (Mandatory)
- Configuration and installation checking facilities should be automated (Desirable)
- The test suite must include clear and comprehensive user and programmer documentation

The Open Group's test suite acceptance criteria from which the information above was taken is also useful reading.

Writing new test suites

The following procedure for developing test suites will help in creating tests which are most useful for the certification process:

Development of set of formal assertions
The aim here is to create a list of testable statements from the specification. This is an intermediate step between the language of the specification and actual test code.
Review of assertions
The review of assertions helps to identify bad assumptions made by the writer of the assertions as well as ambiguities or errors in the specification being tested.
Testcase development
This involves mapping the assertions to actual code which attempts to verify them. Testcase writers should attempt to follow the guidelines listed above such as only testing one thing at a time and avoiding having side effects between testcases. These two properties are especially valuable when attempting to debug test failures.
Testcase review
Having the testcases run by other people on a wide variety of implementations helps pick up problems in testcase code and the underlying specification.

The development of the FHS test suite is a good example of how to approach new test suite development. The document here describes the process used and this presentation contains more information including some details on assertion writing.

Converting existing test suites to use the TET/VSXgen framework

There are two ways of converting existing testcases to be usable in the TET/VSXgen framework used by the LSB. The first is to add the TET specific calls to the original testcase source code, the second is to wrap the test binaries in shell code which is TET aware.

Instrumenting testcase code
This approach can work very well in some cases. Where testcases are a series of macro'd assertions like:
```
ASSERT(foo==5, "Foo was not 5");
```
then the macro can be changed to call the appropriate TET calls making the test suite TET aware without a lot of intrusive code changes. If this is not possible then it is of course still possible to manually add the TET calls.
Wrap test binaries
Where testcases return with informative exit codes or output parseable error messages a shell script wrapper can be used to convert these to TET return codes. This approach will probably lead to test suites which are slightly harder to debug and results that may be harder to understand but will allow for the reuse of existing testcase code.

LSB TET/VSXgen Framework

Inner workings of LSB build/configuration layer on top of TET/VSXgen

The following is a description of how the TET based LSB test suites are installed, configured and built. Although all of these steps can be done manually and most of them are described in various parts of the TET and VSXgen document, in order to ease the development and testing process wrapper scripts were written which automates the procedure. Comments in italics highlight information which is of particular importance to those importing test suites into the framework

Installation of test tools. This first part of the installation is done as root. No files from the tarballs are installed outside of $TET_ROOT whose default value is /home/tet.
1. Creation of vsx0, vsx1 and vsx2 users and vsxg0, vsxg1 and vsxg2 groups. Also NGROUP_MAX supplementary groups are created. The test suites are nominally executed as the vsx0 user but the other users and groups are needed to test functionality such as permissions.
2. Install LSB TET/VSXgen framework
  The tet_vsxgen*.tgz tarball is unpacked into $TET_ROOT
3. Install the selected test suite tarballs
  The lts_* test suite tarballs are unpacked into $TET_ROOT/test_sets. Thus it is important that the test suite tarballs should be packed such that the root directory of the files to be unpacked is $TET_ROOT/test_sets.
4. Configures various skeleton shell scripts for the installation directory ($TET_ROOT).
5. Sets ownership of all installed files to vsx0 user and vsxg0 group.
6. Display licence information for the TET/VSXgen harness and all of the installed test suites. The installation script looks for a file named $TET_ROOT/test_sets/scripts/<test_suite_name>/display_licence_brief.sh and executes it. It is expected that this shell script handles the display of the licence information for that test suite. The directory $TET_ROOT/test_sets/script/<test_suite_name> is where the installation scripts look for test suite specific actions at various stages of the installation and building process.
7. The user is instructed to login as the vsx0 user and run the script $TET_ROOT/setup.sh which starts the next stage the configuration and build process.
The $TET_ROOT/setup.sh script is run as the vsx0 user but at various stages of configuration and building of the test suites super-user privileges are acquired (the user has to enter the root password).
1. TET whose source code lives in $TET_ROOT/src is built and then installed into $TET_ROOT/bin. The makefile in the TET/VSXgen tarball is configured so that a thread aware version of TET is built.
2. It is assumed that gcc is used to compile the test suites and the gcc includes directory is added to the list of directories that TET looks in for header files.
3. Removes files from $TET_ROOT/test_sets/scripts/locks . The files in here are generally used to ensure that setup code which common across multiple test suites is only run once per configuration or setup run.
4. Executes all of the scripts in $TET_ROOT/test_sets/scripts/*/ which are called pre-config.sh. This is the first hook that test suites can use to have test suite specific setup code invoked. For example the VSX-PCTS test suite sets up a loopback filesystem on which to do filesystem tests that cannot be run on the native filesystem such as testing that ENOSPC is returned correctly.
5. The VSXgen config.sh script which is described in Chapter 8 of the VSXgen Specification document is executed. During this configuration process various questions are asked and the default values have been set to the appropriate values for testing on Linux. The default test mode has been set to UNIX98 and the binary version of the test suites are compiled with this mode. All further test suites written must support this mode.
6. Executes all of the scripts in $TET_ROOT/test_sets/scripts/*/ which are called post-config.sh. An example of a test suite using this hook is the VSX-PCTS test suite which performs the installation of the pseudo languages needed by the test suite.
7. Build the VSXgen framework.
8. Executes all of the scripts in $TET_ROOT/test_sets/scripts/*/ which are called post-build.sh. An example of a test suite using this hook is the VSX-PCTS test suite which installs wrapper scripts for its tar format tests allowing for either tar and cpio or pax to be used.
9. Setting of test suite parameters for test suites which are passed to the testcases when they are executed. The setup script will run all of the scripts in $TET_ROOT/test_sets/scripts/*/ which are called parameterisations.sh. These scripts should write the parameterisation values which required for testing on a Linux platform. It is valid for a parameterisation script to ask for further input from the user. New parameter values should be written to $1.
10. The setup script executes the run_testsets.sh script which starts the process of building an executing the testcase binaries.

Summary of additional requirements needed for LSB TET/VSXgen test suite

The format of the name of the test suite tarball should be with lts_<test_suite_name>-A.B.X.Y.tgz where
- A.B refers to the version of the LSB specification that the test suite is targetted
- X refers to the major version of the test framework required
- Y refers to the version of the test suite itself
The root of the tarball should be $TET_ROOT/test_sets.
Scripts to be run during the build, configuration and just prior to the execution of the test suites should be placed into scripts/<test_suite_name>. Valid names for scripts are:
- display_licence_brief.sh (Required)
- pre-config.sh
- post-config.sh
- post-build.sh
- parameterisations.sh
Where there is common setup code between some test suites they can be included with the TET/VSXgen tarball and placed in the scripts/common directory to ensure consistency.
Where there is common helper library code (from SRC/common) between test suites then they should be placed in the main TET/VSXgen tarball to avoid consistency problems. Otherwise they should be supplied by the test suite tarball.

Directory hierarchy description

All of the directory paths described are relative to $TET_ROOT, which is commonly /home/tet. This is not an exhaustive description of all the directories, but should give you an idea of how things fit together.

/src, /include - TET framework source and include files.
/bin - TET framework binaries are installed here.
/test_sets/tset - Source code for individual testcases reside under here. The directory hierarchy under this point is reflected in the scen.bld and scen.exec files which tell TET/VSXgen which testcases to build and run.
/test_sets/TESTROOT/tset - This hierarchy is a copy of the one under /test_sets/tset. It is automatically constructed and populated when the test suites are built.
/test_sets/MAN/tset - This is also of identical structure to /test_sets/tset and contains the manual pages for individual testcases. These correspond to the test assertions. Very useful information can be found here when attempting to debug testcase failures.
/test_sets/scripts - scripts supplied in the test suite tarballs which are used during the installation and configuration of the test suite. Scripts here are also supplied with the binary package to setup the correct environment prior to the testcases being executed.
/test_sets/results - journal files recording the results of building and executing the testcases are placed here. Directories ending in a 'b' contain build results, those ending in an 'e' contain execution results.
/test_sets/SRC/common - directories in here contain helper library code. Some of these are supplied by VSXgen others are test suite specific.

Examples

For an example on how to write a shell based test suite (say you are wanting to write a test suite that tests the behaviour of a set of commands) you should look at the LSB-FHS test suite.

The VSX-PCTS and LSB-OS test suites contain many examples of how to test C library interfaces. There are many scenarios which involve the test setting up complex situations to test certain aspects of an interface's functionality.

Last Updated 2002/07/18 Christopher Yeoh
Please send any feedback/comments to lsb-test@lists.linuxbase.org