LSB Runtime Certification Test Suite Frequently Asked Questions
Last Updated : Dec 28 2005: lsb-runtime-faq.html
This is the Frequently Asked Questions file for the LSB Runtime
Certification Test Suite.
Its maintainer is Andrew Josey.
Suggestions and contributions are always welcome, post them to
the lsb-test mailing list.
This document can be found on the world wide web at http://www.linuxbase.org/test/lsb-runtime-test-faq.html.
This article includes answers to the following.
General Questions section
Q1.0 What is the LSB Runtime test suite? How long does it take to run?
Q1.1 Why is it a binary rpm?
Q1.2 Where can I download the test suite from? Where do I get
information on how to run it?
Q1.3 What is the License?
Q1.4 How do I become a participant in LSB testing?
Q1.5 Can I run the tests with NIS or LDAP enabled?
Q1.6 Can I run the tests by logging in via su ?
Q1.7 I installed the package providing the lsb dependency after the test suite, is this ok?
Q1.8 The sigconcept and sigsuspend tests appear to have hung. Is this a
known problem?
Q1.9 Do shadow passwords need to be enabled?
Q1.10 Some tests results say FIPS?
Q1.11 What do the various tests results mean, which equate to PASS/FAIL?
Q1.12 What does "F" and "M" from the test report mean?
Q1.13 Is theruntime test intelligent enough to determine which library version
they are running against?
Code fixes section
Q2.1. Where can a conforming version of Pax be obtained from?
Q2.2. Are there any known fixes for miscellaneous test failures?
Q2.3. I cannot build the pseudolanguages from source
Troubleshooting section
Q3.1. I get execution errors referencing a missing ls-lsb.so.1 interpreter.
Q3.2 Is there a user guide for using the lsb-runtime test suite?
Q3.3 How can I run just a single test case?
Q3.4 How can I just run tests that failed?
Q1.0 What is the LSB Runtime test suite? How long does it take to run?
This is now a set of binary test suites (supplied in RPM format) for testing
LSB runtime environments for conformance to the LSB Specifications.
It comprise a large test suite for core interfaces known as the
lsb-runtime test , and a number of additional test suites
packaged as separate rpms.
Typical time for a single run of the main test suite is approximately 7 hours
on a uniprocessor machine. The other tests take minutes rather than
hours to run.
For LSB 3.0 the tests include lsb-runtime for the core library
and OS tests,
lsb-c++ for the C++ functionality, and
lsb-vsw4 for the X Window System functionality.
Note that the lsb-pam test suite for Pluggable Authentication
Modules is included in lsb-runtime for LSB 3.0 testing.
Q1.1 Why is it a binary rpm?
LSB conformance is about binary systems and application binary
interfaces, hence formal testing uses a binary rpm. Installation of
Rpm package formats are required to be supported on LSB runtime
environments.
A source version of the test suites are also available for
development and general quality assurance testing.
Q1.2 Where can I download the testsuites from? And where can I find information on how to run the tests?
The tests (both the binary rpm and the source versions) can be obtained from:
http://www.linuxbase.org/download/#test_suites.
Information on how to run the runtime tests, and other test tools in the
LSB test program are at:
http://www.linuxbase.org/test/lsb_test_1.1.html.
Q1.3 What is the License?
See the copyright notice on the test suites and the licenses
displayed during installation of the source version. These are
primarily under a modified Artistic License .
Q1.4 How do I become a participant in LSB testing?
See the information about the LSB pilot certification program.
URL:
http://www.linuxbase.org/test/pilot/.
Q1.5 Can I run the tests with NIS or LDAP enabled?
No, NIS and LDAP need to be disabled, since some tests for
the LSB.usergroups test suite and/or the tests for
"unused userids" fail.
Q1.6 Can I run the tests by logging in via su ?
If you login using "su - vsx0; run_tests", you will find some
tests fail. You need to login to the machine either using
telnet localhost, or slogin localhost.
Q1.7 I installed the package providing the lsb dependency after the test suite, is this ok?
No.
If you first install lsb-runtime-test and then the package providing
the "lsb" dependency (as required by the spec (see below for LSB2.x and on)) then
the postinstall script from lsb-runtime-test will have failed .
Ensure you install the "lsb" dependency package first, or reinstall
lsb-runtime-test again.
Note for LSB 2.x and on the dependencies required for a runtime environment are
"lsb-core" and "lsb-graphics".
Q1.8 The sigconcept and sigsuspend tests appear to have hung. Is this a
known problem?
These test cases take a long time to run, they are checking
for behavior when signals and alarms occur or timeout. The sigconcept
test case typically can take over 100 minutes, and the sigsuspend test
over 60 minutes. Note that in LSB 2.0 we have attempted to reduce the time
needed.
Q1.9 Do shadow passwords need to be enabled?
Yes, shadow passwords need to be enabled to successfully pass some of
the tests in the LSB-USERSGROUPS test suite.
If you have the shadow package installed on your system, you
might need just to run the pwconv command.
Q1.10 Some tests results say FIP (futher information provided)?
These FIP results occur when by default the test suite is unable to determine
whether a result is a PASS or FAIL. For LSB Certification you
have to sign these off as resolving to pass when you submit
your test results into the web certification system.
Q1.11 What do the various tests results mean, which equate to PASS/FAIL?
An explanation of the result codes follows below:
-
PASS - a test result belonging to this group is considered to be a pass
for compliance testing purposes:
-
Pass - the test has been executed correctly and to completion without
any kind of problem
-
Warning - the functionality is acceptable, but you should be aware
that later revisions of the relevant standards or specification may
change the requirements in this area.
-
FIP - additional information is provided which needs to be checked
manually.
-
Unsupported - an optional feature is not available or not supported
in the implementation under test.
-
Not in Use - some tests may not be required in certain test modes or
when an interface can be implemented by a macro or
function and there are two versions of the test only one is used.
-
Untested - no test written to check a particular feature or an optional
facility needed to perform a test is not available on the
system.
-
FAIL - a test result belonging to this group is considered to be a fail
for compliance testing purposes (unless the failure has been waived
by an agreed Problem Report in the Certification Problem Reporting database):
-
Fail - the interface did not behave as expected.
-
Uninitiated - the particular test in question did not start to execute.
-
Unresolved - the test started but did not reach the point where the
test was able to report success or failure.
-
Unreported - a major error occurred during the testset execution.
Q1.12 What does "F" and "M" from the test report mean?
F is for function, M is for macro. The OS testset driver automatically
generates tests for the macro versions of an interface in addition
to the underlying Function . In general not many function interfaces are
implemented as macros hence the high NOTINUSE count in that column.
Q1.13 Is the runtime test intelligent enough to determine which library version they are running against?
- the tests are not library-specific that way
- for libraries where there's a change in so-version,
the tests won't even run if the correct library isn't present
- the rest of this job is handled by lsblibchk
Section 2 - Code fixes for LSB conformance
Please note that these fixes are unofficial, provided by
the community, these were primarily for LSB 1.2 certification, they
are not known to be required these days, your mileage may vary...
Q2.1 Where can a conforming version of Pax be obtained from?
A conforming version of pax can be obtained from
ftp://ftp.suse.com/pub/people/kukuk/pax/
Q2.2. Are there any known fixes for miscellaneous test failures?
-
/tset/LSB.os/mfiles/msync/T.msync 1 FAIL
Reason: Kernel bug.
Chris Yeoh posted a fix for this to linux-kernel:
--- linux-2.4.18/mm/filemap.c~ Thu Mar 21 16:04:48 2002
+++ linux-2.4.18/mm/filemap.c Fri Mar 22 15:48:40 2002
@@ -2205,6 +2205,9 @@
int ret = 0;
struct file * file = vma->vm_file;
+ if ( (flags & MS_INVALIDATE) && (vma->vm_flags & VM_LOCKED) )
+ return -EBUSY;
+
if (file && (vma->vm_flags & VM_SHARED)) {
ret = filemap_sync(vma, start, end-start, flags);
-
/tset/LSB.os/procenv/nice/T.nice.7,8
2002-03-02 Thorsten Kukuk
* sysdeps/unix/nice.c: Use getpriority() for the return value
--- sysdeps/unix/nice.c
+++ sysdeps/unix/nice.c 2002/03/02 09:40:14
@@ -43,7 +43,7 @@
result = setpriority (PRIO_PROCESS, 0, prio + incr);
if (result != -1)
- return prio + incr;
+ return getpriority (PRIO_PROCESS, 0);
else
return -1;
-
/tset/POSIX.os/ioprim/fcntl/T.fcntl 37 FAIL
This is a kernel bug. Fix below submitted to linux-kernel:
--- linux-2.4.18/fs/fcntl.c~ Fri Mar 22 11:54:35 2002
+++ linux-2.4.18/fs/fcntl.c Fri Mar 22 12:31:18 2002
@@ -66,6 +66,10 @@
write_lock(&files->file_lock);
+ error = -EINVAL;
+ if (orig_start >= current->rlim[RLIMIT_NOFILE].rlim_cur)
+ goto out;
+
repeat:
/*
* Someone might have closed fd's in the range
-
/tset/POSIX.os/procprim/sigconcept/T.sigconcept-10:
--- linux-2.4.18/arch/i386/kernel/signal.c~ Thu Mar 21 16:04:30 2002
+++ linux-2.4.18/arch/i386/kernel/signal.c Thu Apr 18 12:19:37 2002
@@ -658,7 +658,7 @@
continue;
switch (signr) {
- case SIGCONT: case SIGCHLD: case SIGWINCH:
+ case SIGCONT: case SIGCHLD: case SIGWINCH: case SIGURG:
continue;
case SIGTSTP: case SIGTTIN: case SIGTTOU:
-
/tset/LSB.os/mfiles/mmap_P/T.mmap_P-34:
--- mm/mmap.c~ Tue Nov 6 11:12:25 2001
+++ mm/mmap.c Mon Feb 11 15:49:58 2002
@@ -620,7 +620,7 @@
{
if (flags & MAP_FIXED) {
if (addr > TASK_SIZE - len)
- return -EINVAL;
+ return -ENOMEM;
if (addr & ~PAGE_MASK)
return -EINVAL;
return addr;
-
Miscellaneous LSB.os/read and LSB.os/write tests
--- linux-2.4.17/fs/read_write.c Sun Aug 5 22:12:41 2001
+++ linux-2.4.17.SuSE/fs/read_write.c Sun Feb 3 15:05:39 2002
@@ -237,15 +237,14 @@
/* BSD readv/writev returns EINVAL if one of the iov_len
values < 0 or tot_len overflowed a 32-bit integer. -ink */
+ /* According to POSIX.1-2001, readv may only fail if tot_len
+ overflowed a ssize_t. */
tot_len = 0;
ret = -EINVAL;
for (i = 0 ; i < count ; i++) {
- size_t tmp = tot_len;
- int len = iov[i].iov_len;
- if (len < 0)
- goto out;
- (u32)tot_len += len;
- if (tot_len < tmp || tot_len < (u32)len)
+ ssize_t tmp = tot_len;
+ tot_len += iov[i].iov_len;
+ if ((ssize_t)tot_len < tmp)
goto out;
}
-
/tset/LSB.os/mfiles/msync_P/T.msync_P 7,8
/tset/LSB.os/mprotect/mprotect_P/T.mprotect_P 8,9
--- linux/mm/filemap.c
+++ linux/mm/filemap.c 2002/03/03 17:47:15
@@ -2197,23 +2197,26 @@
goto out;
if (flags & ~(MS_ASYNC | MS_INVALIDATE | MS_SYNC))
goto out;
+ if ((flags & MS_ASYNC) && (flags & MS_SYNC))
+ goto out;
+
error = 0;
if (end == start)
goto out;
/*
* If the interval [start,end) covers some unmapped address ranges,
- * just ignore them, but return -EFAULT at the end.
+ * just ignore them, but return -ENOMEM at the end.
*/
vma = find_vma(current->mm, start);
unmapped_error = 0;
for (;;) {
/* Still start < end. */
- error = -EFAULT;
+ error = -ENOMEM;
if (!vma)
goto out;
/* Here start < vma->vm_end. */
if (start < vma->vm_start) {
- unmapped_error = -EFAULT;
+ unmapped_error = -ENOMEM;
start = vma->vm_start;
}
/* Here vma->vm_start <= start < vma->vm_end. */
--- linux/mm/mprotect.c
+++ linux/mm/mprotect.c 2002/03/03 18:00:38
@@ -286,7 +286,7 @@
down_write(¤t->mm->mmap_sem);
vma = find_vma_prev(current->mm, start, &prev);
- error = -EFAULT;
+ error = -ENOMEM;
if (!vma || vma->vm_start > start)
goto out;
@@ -333,7 +333,7 @@
nstart = tmp;
vma = next;
if (!vma || vma->vm_start != nstart) {
- error = -EFAULT;
+ error = -ENOMEM;
goto out;
}
}
-
/tset/POSIX.os/devclass/tcsetattr/T.tcsetattr 3
--- linux-2.4.18.SuSE/drivers/char/tty_io.c.orig Tue Mar 5
15:35:55 2002
+++ linux-2.4.18.SuSE/drivers/char/tty_io.c Tue Mar 5 15:39:30 2002
@@ -1696,8 +1696,18 @@
/* the driver doesn't support them. */
case TCSBRK:
case TCSBRKP:
- if (!tty->driver.ioctl)
+ if (!tty->driver.ioctl) {
+ if (cmd == TCSBRK && arg) {
+ /* tcdrain case */
+ retval =
tty_check_change(tty);
+ if (retval)
+ return retval;
+ tty_wait_until_sent(tty,
0);
+ if
(signal_pending(current))
+ return -EINTR;
+ }
return 0;
+ }
retval = tty->driver.ioctl(tty, file, cmd,
arg);
if (retval == -ENOIOCTLCMD)
retval = 0;
--- linux-2.4.18.SuSE/drivers/char/tty_ioctl.c.orig Tue Mar 5
14:31:37 2002
+++ linux-2.4.18.SuSE/drivers/char/tty_ioctl.c Tue Mar 5 14:32:16 2002
@@ -394,7 +394,7 @@
return -EFAULT;
return 0;
case TCSETSF:
- return set_termios(real_tty, arg,
TERMIOS_FLUSH);
+ return set_termios(real_tty, arg,
TERMIOS_FLUSH | TERMIOS_WAIT);
case TCSETSW:
return set_termios(real_tty, arg,
TERMIOS_WAIT);
case TCSETS:
@@ -402,7 +402,7 @@
case TCGETA:
return get_termio(real_tty,(struct termio *)
arg);
case TCSETAF:
- return set_termios(real_tty, arg,
TERMIOS_FLUSH | TERMIOS_TERMIO);
+ return set_termios(real_tty, arg,
TERMIOS_FLUSH | TERMIOS_WAIT | TERMIOS_TERMIO);
case TCSETAW:
return set_termios(real_tty, arg, TERMIOS_WAIT
| TERMIOS_TERMIO);
case TCSETA:
-
/tset/LSB.os/ipc/semctl/T.semctl 16
--- ipc/sem.c.orig Fri May 24 15:32:44 2002
+++ ipc/sem.c Fri May 24 15:22:44 2002
@@ -643,6 +643,7 @@
for (un = sma->undo; un; un = un->id_next)
un->semadj[semnum] = 0;
curr->semval = val;
+ curr->sempid = current->pid;
sma->sem_ctime = CURRENT_TIME;
/* maybe some queued-up processes were waiting for this */
update_queue(sma);
-
/tset/POSIX.os/files/readdir/T.readdir.9
===================================================================
RCS file: /cvs/glibc/libc/sysdeps/unix/sysv/linux/getdents.c,v
retrieving revision 1.17
retrieving revision 1.17.2.1
diff -u -r1.17 -r1.17.2.1
--- libc/sysdeps/unix/sysv/linux/getdents.c 2001/07/06 04:56:12 1.17
+++ libc/sysdeps/unix/sysv/linux/getdents.c 2002/06/22 05:51:25 1.17.2.1
@@ -1,4 +1,4 @@
-/* Copyright (C) 1993,95,96,97,98,99,2000,2001 Free Software Foundation, Inc.
+/* Copyright (C) 1993, 1995-2002 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
@@ -128,6 +128,10 @@
struct kernel_dirent64 *kdp;
const size_t size_diff = (offsetof (struct kernel_dirent64, d_name)
- offsetof (DIRENT_TYPE, d_name));
+
+ /* Return the error if encountered. */
+ if (retval == -1)
+ return -1;
/* If the structure returned by the kernel is identical to what we
need, don't do any conversions. */
Q2.3. I cannot build the pseudolanguages from source
-
Description: Failures to build the test pseudolanguages from source.
This is a problem with:
glibc/locale/iso-4217.def version 1.13
This file is a misinterpretation of the requirements of POSIX for the
int_currency_symbol.
The Austin group spec says:
"The international currency symbol. The operand shall be a four-character
string, with the first three characters containing the alphabetic
international currency symbol in accordance with those specified in the
ISO 4217 standard. The fourth character shall be the character used to
separate the international currency symbol from the monetary quantity."
This can take the value of any currency symbol
defined in ISO 4217 -- there is no requirement that this be "valid
for current cash transactions" . We suspect the data in this file
has been drawn from Table 1 of ISO 4217 and has neglected Tables 2 and
Table 3 which are important tables.
ISO 4217:2001 includes three tables, table A-1 is the Currency and funds code
list, table A-2 is for Funds code registered with a maintenance agency,
and table A-3 lists codes for historic demoninations of currencies,
Table A-1 contains the european countries who have adopted
the Euro, together with the Euro and their other pre-Euro currency
and a note pointing to a timetable when the currency will be withdrawn.
This is not a withdrawal of the currency symbol in the ISO standard.
I would anticipate that a future revision of 4217 will
migrate the DEM, FRF etc to Table A-3.
Implementations that are disallowing currency symbols now obsoleted
by the Euro are doing so in error . The DEM and FRF symbols
are listed in 4217:2001.
We expect that the current European currencies presently
in Table 1 in ISO 4217:2001 will migrate to table 3.
Limiting this to "valid" currencies as in what is valid for cash
transactions does not allow for historical processing applications nor the
fact that the central banks plan to continue accepting certain currencies
even though they are not the valid cash for a particular currency.
How-To-Repeat:
Try to use a currency symbol defined by ISO 4217 that
was removed from the glibc upstream source from
locale/iso-4217.def in the MAIN branch as of
version 1.13 and the glibc-2.2 branch as of 1.10.2.2.
Fix:
Reinstate the locales obsoleted by the Euro phaseover
DEFINE_INT_CURR("ADP ") /* Andorran Peseta obsoleted by EURO 28.2.2002 */
DEFINE_INT_CURR("ATS ") /* Austrian Schilling */
DEFINE_INT_CURR("BEF ") /* Belgian Franc obsoleted by EURO 28.2.2002 */
DEFINE_INT_CURR("DEM ") /* German Mark obsoleted by EURO 28.2.2002 *
DEFINE_INT_CURR("ESP ") /* Spanish Peseta obsoleted by EURO 28.2.2002 */
DEFINE_INT_CURR("FIM ") /* Finnish Markka obsoleted by EURO 28.2.2002 */
DEFINE_INT_CURR("FRF ") /* French Franc obsoleted by EURO 28.2.2002 */
DEFINE_INT_CURR("GRD ") /* Greek Drachma obsoleted by EURO 28.2.2002 */
DEFINE_INT_CURR("IEP ") /* Irish Pound obsoleted by EURO 28.2.2002 */
DEFINE_INT_CURR("ITL ") /* Italian Lira obsoleted by EURO 28.2.2002 */
DEFINE_INT_CURR("LUF ") /* Luxembourg Franc obsoleted by EURO 28.2.2002 */
DEFINE_INT_CURR("PTE ") /* Portugese Escudo obsoleted by EURO 28.02.2002 */
Section 3 - Troubleshooting
Q3.1 I get execution errors referencing a missing ls-lsb.so.x interpreter.
After installing the test suite I get the following error when trying
to execute them (note for LSB 2.x the interpreter is ld-lsb.so.2).
/home/tet/bin/tcc: /lib/ld-lsb.so.1: bad ELF interpreter: No such file
or directory.
Failed to execute the test suites correctly.
Either they are not installed properly or you may be
missing the lsb linker /lib/ld-lsb.so.1 on your system.
You need to ensure that the system is configured to support a conforming
LSB Runtime Environment. In some cases, this will require installation
of the appropriate rpm package, for example on the initial version
of Fedora Core 1 this is the package redhat-lsb-1.3-1.
If you want to test a system that does not provide the LSB runtime linker,
you can
normally just create a symbolic link to the standard linux linker. eg:
ln -s /lib/ld-linux.so.2 /lib/ld-lsb.so.1
Or for LSB 2.x
ln -s /lib/ld-linux.so.2 /lib/ld-lsb.so.2
Note that for LSB 1.3 and later for certified runtime
environments you can check the associated LSB Conformance Statement
since this has a question on what is needed to configure the
system to provide a conforming runtime environment. See here for the
register of LSB certified products (the final column has a link to the
conformance statement for each registered product) .
Another potential cause of this error message is if you have installed
the rpm but later moved the files into another area (say for disk
space reasons) and symlinked the hierarchy back in (eg any part of
/home/tet/test_sets is a symlink). This will result in TET being
unable to find a default testset to execute and generate the error
message above.
Q3.2 Is there a user guide for using the lsb-runtime test suite?
A pdf guide which includes the core test sets is provided by
The Open Group at
http://www.opengroup.org/infosrv/lsb/ogdeliverables/docs/.
It should be noted that this explains in detail how to build
and configure the source versions of the test suites and thus
is most suitable if you wish to do detailed debugging of problems.
Since the test suites are provided as binary rpms the configurations
are preset.
Q3.3 How can I run a single test case?
The lowest level of granularity in the lsb-runtime test allows you to
execute individual tests. Most tests can be executed in this way, but some
are dependent upon execution of earlier tests in the testset, in which
case only groups of dependent tests may be executed as a single unit.
A way to perform a one-off execution of selected testcases from a single
testset can be performed using the -l option of tcc. For example:
tcc -e -l /tset/POSIX.os/ioprim/write/T.write{3,7,8}
Q3.4 How can I just run tests that failed?
In order to execute testsets which failed during a previous run,
use
tcc -e -r code-list other-options old-journal-file
where code-list is a comma-separated list of result codes to
be re-executed, other-options are the other tcc options (e.g., -y or -n)
and old-journal-file is the journal file from which the codes are extracted. For example, to re-execute all the tests that failed with
FAIL, UNRESOLVED and UNINITIATED codes from journal file
results/0002e/journal, use the following command:
cd results
tcc -e -r FAIL,UNRESOLVED,UNINITIATED -s ../scen.exec 0002e/journal
Q3.5 How can I execute a test directly without tcc?
When debugging tests it is sometimes useful to execute them directly
instead of under the control of tcc. When tests are executed in this way
the current directory must be the location of the testset executable
file. Also the variables TET_CONFIG and TET_CODE must be set in the
environment. Once the tests have been executed the results are found
in a file called tet_xres. For example to execute the tests for write()
directly you would use the commands:
TET_CONFIG=$TET_EXECUTE/tetexec.cfg
TET_CODE=$HOME/tet_code
export TET_CONFIG TET_CODE
cd $TET_EXECUTE/tset/POSIX.os/ioprim/write
./T.write
more tet_xres
This will execute all the tests in the testset. If you want to execute only specified testcases, give the testcase list as an argument:
./T.write 1-3,7
QXX. How do I add a question to this FAQ?
Send the question (preferably with a proposed answer) to Andrew Josey
or the lsb-test mailing list.