Test Failure analysis of TET based LSB Test Suites

Introduction

This document is an introduction on how to analyse test failures in the TET based LSB Test Suites. There is much TET documentation available here and much of what is here is just a summary of sections in those manuals.

Environment Setup

To be able to perform the debugging procedures described you must login as the vsx0 and dot in the profile file in the vsx0 home directory. Eg:

vsx0@localhost:~$ . profile

This will set the environment up correctly.

Interpreting test results

Executing testcases will result in the creation of a journal file in /home/tet/test_sets/results/xxxxe/journal where xxxx is a four digit number starting at 1 that is increased with each test suite run. Results of testcase compilations are placed in journal files in a similarly named directory except ending in 'b' instead of 'e'.

vrpt

vrpt will generate a report detailing the all of the testcases executed and the result. There are several result types which can be separated into two main groups, pass or fail. For more details see this document document on test result classification. For testcases which did not pass the test assertion and test strategy will also be output.

tjreport

Also supplied with the binary test suites is the tjreport tool. This will produce a summary of all of the testcases run and the results, or optionally only display those that failed. It can also be used to intelligently diff between two journal files and display those testcases for which the results differ between the two test runs. See the tjreport man page for detailed information.

Raw journal files

If you're interested in interpreting the journal files manually have a look at Appendix C of the TETware User Guide which describes the format of the file.

Using binary versions of the test suites

There is a limited amount of analysis that can be done with the binary versions of the test suites alone as they are not compiled with debug symbols.

Enabling debug output

You can enable some debugging information which is output during the execution of the tests by editing the file /home/tet/test_sets/TESTROOT/tetexec.cfg. Edit the setting for VSX_DBUG_FLAGS as follows:

VSX_DBUG_FLAGS=t:d,trace:l,2

The setting VSX_DBUG_FILE in the same file specifies where the debug information is placed. With reference to the source code for the test suite its often possible to work out the code path executed. The source tarballs are available from the freestandards ftp site, or if you prefer unpacked from the test_sets module in CVS on Gforge.

Building the test suites from source

Download source tarballs
The source tarballs are available from ftp://ftp.freestandards.org/pub/lsb/test_suites. It may be useful to look at the versions in the beta directory as if the problem you are seeing is a test suite bug it may already have been fixed.
The install.sh and tet_vsxgen_*.tgz files must be downloaded. Only the individual test suite tarballs (prefixed with lts_) that you wish to install need to be retrieved. See the README file for details on how test suite version numbers are chosen.
Install source
With all of the downloaded files in the current directory execute the install.sh script. In addition to the TET/VSXgen harness you must install at least one of the test suite tarballs. The license information for all of the packages you have installed will be displayed followed by instructions how to proceed further.
Configure test suites and build test harness
Follow the instructions given after installing the source packages. Note that if you "su" to the vsx0 user instead of logging in then some tests will fail (such as those that verify that the tty that the user is logged in is owned by the user).
The default values for all the configuration questions are what are used during the build of the binary versions of the test suites. If you believe that a setting is wrong for the platform you are testing please let us know (send an email to lsb-test@lists.linuxbase.org).
Build and execute testcases The script will automatically progress to the build and execution of the testcases. If you are only wanting to debug a couple of the testcases you may prefer to save a quite a lot of time and not get the script to do this stage for you. Instead follow the procedure for building testcases shown below.

The script automates the standard build and setup procedure for TET and VSXgen. For details on what is done see the document on how to Build TET based LSB Test Suites.

Rebuild and execute testcases

Once the TET/VSXgen framework has been configured and built you can (re)build and (re)execute any or all of the testcases. The file /home/tet/test_sets/scen.bld specifies the testcases which will be built and /home/tet/test_sets/scen.exec specifies those that will be executed.

(Re)Building testcases

You can build all of the testcases using:

vsx0@rockhopper:~$ tccb -p

The names of the testcases will be displayed as they are being built. Note that build failures will not be shown and you need to inspect the journal file especially if you have changed the testcase code.

You can build a subset of the available testcases like this:

vsx0@rockhopper:~$ tccb -p -y T.fgetpos

which from the lsb-os test suite would build /tset/LSB.os/streamio/fgetpos/T.fgetpos, or like this:

vsx0@rockhopper:~$ tccb -p -y string

which from the lsb-os test suite would build:


	/tset/LSB.os/string/mblen/T.mblen
	/tset/LSB.os/string/mblen_L/T.mblen_L
	/tset/LSB.os/string/mbstowcs/T.mbstowcs
	/tset/LSB.os/string/memmove/T.memmove
	/tset/LSB.os/string/wcstombs/T.wcstombs
	/tset/LSB.os/string/wctomb/T.wctomb

You can specify the -y argument multiple times which will do an inclusive OR of the specified testcases.

(Re)Executing testcases

The tcce command works in a similar manner to the tccb command but in executing testcases. Eg.

vsx0@rockhopper:~$ tcce -p

which will execute all of the testcases, or

vsx0@rockhopper:~$ tcce -y T.fgetpos

which will execute only the T.fgetpos test.

tccb and tcce are shell functions which call tcc. You can find out more details on the uses of tcc in Chapter 6 of the TETware User Guide.

Executing test binaries directly

It can sometimes be useful to execute test binaries directly (eg you want an easier way to attach a debugger). Although the source for the testcases lives under /home/tet/test_sets/tset when they are built the binaries are placed under an identical directory hierarchy under /home/tet/test_sets/TESTROOT/tset.

To run test binaries directly you must first set the following environment variables:

export TET_CONFIG=$TET_EXECUTE/tetexec.cfg
export TET_CODE=$HOME/tet_code

Then cd to the directory containing the testcase and run the executable. A journal file will not be created, but the contents of what would have been placed in the journal file for that test will be left in tet_xres in the current directory.

You can also specify only specific tests within that binary are to be run. Eg.

./T.makecontext 1,3

will run only the first and third tests in T.makecontext and

./T.wctomb 2-4

will run only the second to fourth tests.

General Debugging Hints

The in_rpt call works in an equivalent manner to printf and the output is put into the journal file.
Edit the Makefile for the testcase (it is in the same directory as the code for the testcase and add debugging flags to the compile line.
Many of the tests fork and execute the test code in the child, or have some form of communication with the parent. It can be useful to add a sleep call in places to make it easier to attach with a debugger (especially when gdb is resisting following forks).
It is often useful to write a small standalone program which attempts to do the same things as the test. It helps to isolate implementation problems from testcase and test harness problems and can be easier to debug. Also upstream maintainers will appreciate (if not insist on) having an easily compilable and executable test program when fixing bugs.

Last Updated 2002/08/2 Christopher Yeoh
Please send any feedback/comments to lsb-test@lists.linuxbase.org