LSB Test & Certification 4Q2001

• Past Meetings Minutes & Notes
• Participants to Date
• Test & Certification Program Milestones & Status 4Q2001
• Test & Certification Program 4Q2001
• Test & Certification Tasks 4Q2001
• Test Suite Technical Milestones & Status 4Q2001
• Test Suite Deliverables 4Q2001
• Test Suite Technical Tasks 4Q2001

Past Meetings Minutes & Notes

• July 25, 2001 LSB Test Team meeting minutes by Andrew Josey
• July 20, 2001 LSB Test Team meeting minutes by Andrew Josey
• July 12, 2001 LST Test Status notes  by George Kraft
• May 11, 2001 LSB Test Team meeting minutes by Andrew Josey

Participants to Date

• Andrew Josey, Open Group, Austin Group, LSB Test Team Leader
• Nick Stoughton, Austin Group
• Mark Brown, IBM, Austin Group
• Andrew Twigger, UniSoft
• Scott McNeil, Free Standards Group, FSG Executive Director
• Kevin Cuant, IBM, LSB Test Developer
• Chris Yeoh, IBM, LSB Test Developer
• George Kraft, IBM, LSB Chairman
• Marvin Heffler, IBM
• Matt Wilson, Red Hat
• Doug Beattie, Caldera
• Ralf Flaxa, Caldera,
• Johannes Poehlmann, Caldera
• Mary Esther Middleton, Compaq
• Jill Jones, Compaq
• Raja Srinivasan, Oracle

Test & Certification Program Milestones & Status 4Q2001

• Test & certification program plan: platform, application, development
• Test & certification pilot program
• Test & certification pilot program data sheet for LinuxWorld [due 8/22 but not assigned]

Test & Certification Program 4Q2001

• What does Certification mean; are we saying passed a test or met a specification? What does it mean for applications, implementations and development environments.
• We will need to define very clearly the conformance requirements, defining precisely what a conforming product must implement in order to be certified.
• Interpretations policy
• Process or processes for certification
• It is proposed that there be three levels of certification:

The Vision

A "LSB certified application" should run on any "LSB certified platform" (for a particular processor architecture)

Rationale: This means a single binary application image being able to  execute correctly on any LSB conforming platform (operating system) for a particular processor architecture. Some form of ``Application Certification' is required in order to bound the problem space to well formed applications

What does it mean to be LSB certified?

For the platform this means: The platform meets the definitions of implementation conformance and provides a conforming runtime system. Platforms would be expected to maintain compliance once they have registered a system. There would need to be a policy regarding recertification for updates to the distribution, addressing when a new set of results would need to be audited.

Platform Certification

• What: A "brand" mark for platforms to signify they support an environment to run LSB conforming applications
• Uses: Requires a binary test suite for the runtime environment
• This will require repackaging of what we have -- includes removal of parts of the existing tests that are more suited for testing the development environment rather than the runtime environment, for example the header tests
• Are there some of Stuarts tools that should be used as part of the platform certification ? libcheck?
• Also requires running a certain number of key LSB conforming applications (we need to define which applications)
• Opportunity for Scott: we need resource for the repackaging of the suites into a binary runtime test suite

Development Environment Certification

• What: A "brand" mark for compiler environments to signify they provide an environment to build an LSB conforming application
• Uses: We should have the capability to test the development environment using devcheck, , libcheck?
• Note that a development environment does not need to be compliant itself. It is required to produce conforming output.

Application Certification

• What: A "brand" mark for applications to signify they work with an LSB conforming platform
• Uses: lsbappchk?
• The application has used the tools provided for checking it, appchk, and elfchk?
• The application has been tested on the sample implementation and two others. (There needs to be a policy determining what constitutes testing and how this is verified)
• The application has been checked for FHS compliance?

Test & Certification Program Tasks 4Q2001

• Input on certification policies and procedures
• Requirements for Linux distribution testing with a set of applications
• What does Certification mean; are we saying passed a test or met a specification? What does it mean for applications, implementations and development environments.
• We will need to define very clearly the conformance requirements, defining precisely what a conforming product must implement in order to be certified. An action on the specification review should be to check that the requirements are clear.
• Interpretations policy
• Process or processes for certification
• Brand marks (trademarks) for platform, development environment, and application certification to signify they are in compliance
• Platform certification:
1. What: A "brand" mark for platforms to signify they support an environment to run LSB conforming applications
2. Test suites/tools:
1. Requires a binary test suite for the runtime environment
2. This will require repackaging of what we have -- includes removal of parts of the existing tests that are more suited for testing the development environment rather than the runtime environment, for example the header tests
3. Binary versions of vsx-pcts, lsb-os, lsb-fhs
4. Also requires running a certain number of key LSB conforming applications (we need to define which applications). This is known to be a good approach if a key set of applications exercising a wide set of APIs is chosen.
• Development Environment/SDK Certification:
1. What: A "brand" mark for compiler environments to signify they provide an environment to build an LSB conforming application
2. Test suite/tools:
• We should have the capability to test the development environment using devcheck, elfcheck, libcheck
• Note that we would not use the main platform tests for these, the header tests are not thought to provide that valuable output for the development environment, better to use the binary value tests generated by the database.
• Application certification:
1. What: A "brand" mark for applications to signify they work with an LSB conforming platform
2. Test Tools:
• lsbappchk

• What are the steps needed to get to the vision?
• We currently have a circular conundrum: we need conforming development environments to produce conforming apps and test suites, which in turn we need to determine whether a platform is conforming.
• How do we break out from this and move forward? We need to start using the sample ``xdev' development environment provided with the Sample Implementation and start to build applications and the binary test suites with this.
• We will need to roll out a pilot program in good time to ensure that we achieve the vision. This would be at least three months in advance of the target launch.

• What other things can we do in the certification processes to ensure that we achieve the vision
• To strengthen the testing procedures for the platforms we need either to build, or select some applications that would be required to be demonstrated as running on a platform. These should be selected so as to maximize the coverage
• We should  organize an interoperability workshop (under non disclosure to all parties) where we bring distributions , application developers, and the development tools teams, and spend several days building LSB applications , and then testing interoperability across multiple LSB trial platforms.

• Who are the players and how do we get them involved?
We need to have certified products on the day of the launch. Many of the distributions vendors are actively involved. We need to issue an invitation to others to get involved. We also need to get more ISVs involved. We need to issue an invitation to lsb-discuss or elsewhere looking for candidate applications for trialing, these should be drawn from  a cross section of application categories: database, desktop, system administration etc.

Test Suite Technical Milestones & Status 4Q2001

• LSB v1.0 Distribution Test Suite
• RPM packaged binary runtime test suites
• LSB v1.0 Development Environment Test Suite
• RPM packaged binary runtime test suites
• LSB c1.0 Application Test Suite
• RPM packaged binary runtime test suites

Test Suite Deliverables 4Q2001

Distribution Test Suite,  LSB 1.0

• LSB-VSX-PCTS (binary)
• Contact: Andrew Josey - OpenGroup
• Test purpose for core POSIX.1
• Coverage:
• Status: Nearly complete.   Waiting on final LSB v1.0 header file information.

• LSB- FHS  (binary)
• Contact: Andrew Josey - OpenGroup; Chris - IBM
• Test coverage for the filesystem hierarchy standard
• Coverage:
• Development Status: FHS v2.2 rework, testing, & beta completed by Andrew Josey
• Packaging Status:

• Contact: Andrew Josey - OpenGroup
• Additional test  coverage for core libc behavior above LSB-VSX
• Coverage:
• Status: Beta code complete  [Target 8/31]

•  LSB - UsersGroups (binary)
•  Contact: Kevin Caunt- IBM
• Testset for User/Groups  specification
• Coverage: 100% [some users & groups testing done in lsb-vsx-pcts]
• Status: Beta code complete  [Target 8/31]

• LSB - LIBCHK
• Contact: Stuart Anderson - Metrolink
• Test for existence of API's
• Coverage: 100%
• Status: Complete

Development Test Suite,  LSB 1.0

• LSB_HDR
• Contact: Stuart Anderson - MetroLink; Jill Jones - Compaq
• Test the binary values of the headers
• Coverage: ??
•  Status:  Complete

Application Test Suite,  LSB 1.0

• LSB-APPCHK
• Contact: Stuart Anderson - MetroLink
• Test for existence of API's
• Coverage: 100%
• Status: Complete

Post LSB v1.0 Candidates

• LSB-COMMANDS
• Commands and Utilities test coverage
• LSB-PTHREADS
• Test coverage for core pthreads
• LSB-NCURSES
• Test coverage for ncurses
• hard: never been done before
• LSB-VSW(X11)
• Test coverage for X11

hard: current X11 tests are outdated

Test Suite Technical Tasks 4Q2001

• Task 39620: Test suite build environment on OSDL.
• Task 27942: Processor-specific definitions for constants and structures needs to be defined, so that a conforming development environment can be used for development of the various test tools for all three proposed programs.
• COMPLETE [Andrew Josey]: LSB-FHS needs to be updated to reflect FHS v2.2.
• Versioning needs to be updated
• Task 19677: Test Case Coverage needs to be measured, calculated, and database updated

• Binary test suites:
• Merge of LSB-VSX/LSB-OS
• Task 35904 (LSB-OS GA): Need to set a target date for moving LSB-OS to GA state, still presently in beta. Suggest 8/31
• Task 36438: Source level test suites:
• LSB-FHS, done but needs new versioning
• LSB-USRGROUPS
• COMPLETE [Kevin Caunt] Task 2001-07-02 (LSB-USRGROUPS BETA): Need to commence a beta for LSB-USRGROUPS test suite. Action on Kevin to create the test bundle and announce the beta test period ASAP.
• Real LSB compliant applications must run
• Task 35905 (SAMPLE APPLICATIONS): Need to select applications for use as part of the platform test package
• COMPLETE [Chris Yeoh] Task 2001-07-04 (LIBCHK): Need to evaluate suitability of libchk as a conformance test tool. For example does it put out identification and version information on the tool itself and the system it is checking?
• Uses for libchk?  distro and/or build environment?
• needs to flag extra symbols found in the libraries

• Applications:
• The application has used the tools provided for checking it, appchk, and elfchk?
• The application has been tested on the sample implementation and two others. (There needs to be a policy determining what constitutes testing and how this is verified) The application has been checked for FHS compliance?
• Task 35907 (REQTS FOR APPLICATION CERTIFICATION): Need to define the policy detailing the requirements for application certification, how the certification body will pick a number of systems from a pool of certified  systems, and require to see the application running on them. This needs to be done is a round robin manner so it can achieve coverage fairly across  the set of certified systems and not just favour one or two systems.
• Task 2001-07-06 (FHS QUESTIONNAIRE): We need a Questionnaire for FHS compliance, asking a set of questions, that the certification body can then assess the FHS compliance of the application.   George posted to fhs-discuss & lsb-test for a volunteer

• Development environment:
• Note that a development environment does not need to be compliant itself. It is required to produce conforming output.
• Task 35914 (DEVCHK COMPLETENESS): Need to evaluate whether devchk is complete, or whether we still the processor specific constants added to it
• Depends on header file information
• Task 35064 (EXAMPLE APPLICATION CODE): Need to provide two or three realistic sample applications that can be used to test the development environment, and also be used as examples for ISVs. The examples should show different cases for an application, for example:
• The LSB provides all facilities for the application
• The application provides its own libraries , or other tools which themselves are LSB compliant
• The application links statically, or dynamically
• How do we break out from this and move forward?
• We need to start using the sample ``xdev' development environment provided with the Sample Implementation and start to build applications and the binary test suites with this.
• Task 35913  (XDEV): Need to get xdev into a useable state so we can start using the xdev development environment to build sample applications and binary test suites
• We will need to roll out a pilot program in good time to ensure that we achieve the vision. This would be at least three months in advance of the target launch.
• Task 35912 (PILOT CERTIFICATION PROGRAM): Need to put a datasheet, high level description of the Pilot Certification program that can be announced at LinuxWorld. This will detail the test suites to be run, and outline the procedures     and detail how folks can enrol in the program.
• Online Questionaire:
• pilot participation?
• application developer?

Platform Testing

• The major development stages required for each TET based test suite are:

 
1. Develop/Acquire test cases

 
2. Source Release

The tests packaged such that they can be used in the test framework. Source packages are released initially to make it easy for initial testing of the test suites to be done.
 
3. Review of test results.

The test results reported by users of the test suite need to be analysed. Decisions need to be made as what to do with test cases that fail - (remove the test, change the test, develop new test to check observed behaviour).

 Resources needed
We should probably produce a matrix showing common test failures across a set of platforms, and hold a meeting to classify the failures:
(a) bug in systems (b) bug in tests (c) known difference between the LSB and the standards
For those tests in category (c) we will need either to fork a version of the tests under LSB.OS, or remove it from the default execution scenario (a scenario is a list of tests).

4. Re-release of source packages taking into account review.
5. Binary Release

Compiled version generated and need to decide on:
• Architecture targets (i386, IA64, PPC, ...)
• Strip out compile time tests
Resources neededThe binary versions of the test suites should just contain the TESTROOT directory and the execution scenarios in the TET_ROOT directory and not contain the tset or most of the SRC directories. We will probably need to have something that allows parameterization of the tetexec.cfg file and any other support scripts but only if they require functionality outside of the LSB Spec.

We will need to start experimenting with this sort of tree to find out the exact dependencies. I suspect it is something like this.

1. Install and build test suite using conforming build and development environment
2. Strip out all runtime compilation level tests (mainly .hdr sections) (need to develop a script to do this)
3. Remove all build and install scripts (need to develop a script to do this)
4. Running the binary version should be just a package install, some setup questions and then a tcc -e (no -b or -c).
• The current further development requirements of the test suites are:
• VSX-PCTS (Stages 3, 4, 5)

Current estimate: ~1 month
• LSB-FHS (Modify for FHS 2.2, then Stages 3, 4, 5)

Need initially to update the test assertions document for FHS2.2 specification, then approximately one week for the test modifications and then we'll need to have a feedback cycle. Estimate of effort: Two weeks engineering, two weeks review.
• LSB-OS (Integration into framework, 3, 4, 5)

Current estimate: 2 months
• VSClite (3, 4, 5)

We need to check that VSC is a candidate, it may be that the Linux shell and utilities are sufficiently different as not to require passing this test suite, similar to the Threads issue
Current estimate: 2 months
• VSW4 (2, 3, 4, 5)

We need to identify a resource to pick this test suite up and work with it.
Current estimate: Unknown
• VSTHlite: We explicitly will not require use of the Threads test suite.
• Work out how to (generally) do a binary release of test suites. Implement common infrastructure needed.
• Automate building of source and binary releases from CVS

Application Testing

• appchk. Available as source need to build official binary release.
• Need for more application testing programs
• package format correct
• installed package conforming to FHS
• runtime FHS conforming (hard to test!?)

Development Environment Testing

• devchk. Needs further development - TET based?
• What else do we need to do here?

We could package up the various header tests from the VSX-PCTS if we felt it was worth the effort.
Another idea would be to use elfcheck to check that binaries generated by the development environment are conforming?