Attendees:

Andrew Josey, The Open Group
Andrew Twigger, UniSoft
Chris Yeoh, IBM
Kevin Caunt, IBM
George Kraft, IBM

Summary

The gating items are as follows:

• Specification
  • Processor-specific definitions for constants and structures needs to be defined, so that a conforming development environment can be used for development of the various test tools for all three proposed programs.
  • We need a status update on FHS2.2 in order to complete the FHS tests
• Test Set
  • We need feedback on the LSB-OS testset
  • We need resources for the VSW / VSC testsets, or we need to take the decision to delay those to a later phase.
• We could do with further input on the requirements for application testing.
• We need input on the certification policies.

LSB Certification Strawman

Firstly we need to define the policy for certification: This needs to define

• What does Certification mean; are we saying passed a test or met a specification? What does it mean for applications, implementations and development environments.
• We will need to define very clearly the conformance requirements, defining precisely what a conforming product must implement in order to be certified. An action on the specification review should be to check that the requirements are clear.
• Interpretations policy
• Process or processes for certification

It is proposed that there be three levels of certification:

1. Platform certification :
   • What:

     A "brand" mark for platforms to signify they support an environment to run LSB conforming applications

   • Uses:

     Requires a binary test suite for the runtime environment

     This will require repackaging of what we have -- includes removal of parts of the existing tests that are more suited for testing the development environment rather than the runtime environment, for example the header tests

     Are there some of Stuarts tools that should be used as part of the platform certification ? libcheck?

     Also requires running a certain number of key LSB conforming applications (we need to define which applications). This is known to be a good approach if a key set of applications exercising a wide set of APIs is chosen.

     Opportunity for Scott: we need resource for the repackaging of the suites into a binary runtime test suite , although Chris does seem to have a handle on it. A prerequisite for this is that we have a conforming development environment otherwise we will have skew in the certification program due to platform dependencies.

2. Development Environment Certification:
   • What:

     A "brand" mark for compiler environments to signify they provide an environment to build an LSB conforming application

   • Uses:

     We should have the capability to test the development environment using devcheck, elfcheck, libcheck?

     Note that we would not use the main platform tests for these, the header tests are not thought to provide that valuable output for the development environment, better to use the binary value tests generated by the database.

3. Application certification:
   • What:

     A "brand" mark for applications to signify they work with an LSB conforming platform

   • Uses:

     lsbappchk elfchk ?

Platform Testing

• The major development stages required for each TET based test suite are:
  1. Develop/Acquire test cases

  2. Source Release
     The tests packaged such that they can be used in the test framework. Source packages are released initially to make it easy for initial testing of the test suites to be done.

  3. Review of test results.
     The test results reported by users of the test suite need to be analysed. Decisions need to be made as what to do with test cases that fail - (remove the test, change the test, develop new test to check observed behaviour).

     Resources needed
     We should probably produce a matrix showing common test failures across a set of platforms, and hold a meeting to classify the failures:
     (a) bug in systems (b) bug in tests (c) known difference between the LSB and the standards
     For those tests in category (c) we will need either to fork a version of the tests under LSB.OS, or remove it from the default execution scenario (a scenario is a list of tests).

  4. Re-release of source packages taking into account review.

  5. Binary Release
     Compiled version generated and need to decide on:
     • Architecture targets (i386, IA64, PPC, ...)
     • Strip out compile time tests
     Resources needed The binary versions of the test suites should just contain the TESTROOT directory and the execution scenarios in the TET_ROOT directory and not contain the tset or most of the SRC directories. We will probably need to have something that allows parameterization of the tetexec.cfg file and any other support scripts but only if they require functionality outside of the LSB Spec.

     We will need to start experimenting with this sort of tree to find out the exact dependencies. I suspect it is something like this.

     1. Install and build test suite using conforming build and development environment
     2. Strip out all runtime compilation level tests (mainly .hdr sections) (need to develop a script to do this)
     3. Remove all build and install scripts (need to develop a script to do this)
     4. Running the binary version should be just a package install, some setup questions and then a tcc -e (no -b or -c).

• The current further development requirements of the test suites are:
  • VSX-PCTS (Stages 3, 4, 5)

    Current estimate: ~1 month

  • LSB-FHS (Modify for FHS 2.2, then Stages 3, 4, 5)
    Need initially to update the test assertions document for FHS2.2 specification, then approximately one week for the test modifications and then we'll need to have a feedback cycle. Estimate of effort: Two weeks engineering, two weeks review.

  • LSB-OS (Integration into framework, 3, 4, 5)

    Current estimate: 2 months

  • VSClite (3, 4, 5)
    We need to check that VSC is a candidate, it may be that the Linux shell and utilities are sufficiently different as not to require passing this test suite, similar to the Threads issue

    Current estimate: 2 months

  • VSW4 (2, 3, 4, 5)
    We need to identify a resource to pick this test suite up and work with it.

    Current estimate: Unknown

  • VSTHlite: We explicitly will not require use of the Threads test suite.

• Work out how to (generally) do a binary release of test suites. Implement common infrastructure needed.

• Automate building of source and binary releases from CVS

• libchk. Available as source, need to build official binary package.

• Determine/Document current test coverage (Source/Binary coverage will differ)

Application Testing

• appchk. Available as source need to build official binary release.

• Need for more application testing programs
  • package format correct
  • installed package conforming to FHS
  • runtime FHS conforming (hard to test!?)

Development Environment Testing

• devchk. Needs further development - TET based?

• What else do we need to do here?
  We could package up the various header tests from the VSX-PCTS if we felt it was worth the effort.
  Another idea would be to use elfcheck to check that binaries generated by the development environment are conforming?