Linux Binary Deployment Effort Collaboration

Introduction

The Linux Standard Base workgroup is regularly approached by developers that are concerned with the large number of Linux runtime environments and the binary portability of their applications across those implementations. It is understandable why so many people would have this same concern, these developers have a need to tets and deploy binary packages in as easy a way as possible. This is not just commercial and/or propritary applications, but non-commercial and/or open as well.

Past efforts to address (at least partially) this problem include LDPS, lsb-platform, UnitedLinux, and UserLinux.

The majority of these developers are used to the software deployment model of legacy Microsoft and UNIX based systems that involves applications "certified" for a particular version of a runtime environment. They have suggestions for solving this "problem", and they usually revolve around somehow standardizing on one Linux runtime. They will point to Microsoft Windows and say, "Windows developers only have to target one runtime, and on Linux there are many. Wouldn't it be great if there was just one version of Linux?" In addition to being wrong (there are many versions of Microsoft Windows as well) this statement indicates several misunderstandings about Open Source,

• Having a diverse heterogeneous set of runtime environments has benefits not present in a homogenous single implementation
  • promotes competion at the distribution level, ie Who can assemble the best runtime environment?
  • promotes competion at the interface level, ie Who can write the fastest implementation of library foo?
  • competition allows for implementation evolution, when a superior distribution or implementation of an interface is created, people can freely move to it with few of the blockers found in the propriatary "lock-in" model.
  • promotes problem or application specific implementations that better meet users needs, ie you can have different distributions for french speaking school teachers, teenage gamers, and blind hackers.
• You can't force everyone to use the same binaries.
  • If if you wanted to, in FOSS there are no "control points" to make people use one implementation over another if they don't want to. If you label incentives as "carrots" and "sticks" there are no sticks, and most carrots can be freely adopted by other runtimes when it's clear that users want them.
  • You may be able to get a particular group of users to rally around a common implementation, but the benefits of diversity will always mean that the majority of users and developers will be using something else.

We need ways to solve the problems of Linux binary deployment that don't eliminate the reasons for using FOSS in the first place. In addition, if they are to be trusted, these must be open and documented processes.

There are several groups working on this problem. This document attempts to record the details of these efforts and how they relate so that they might work better together.

Linux Standard Base (LSB)

Website: http://www.linuxbase.org

Mailing lists: http://www.linuxbase.org/modules.php?name=Content2&pa=showpage&pid=27

Purpose: To develop and promote a set of standards that will increase compatibility among Linux distributions and enable software applications to run on any compliant system.

Benefit: Defines a base set of application development interfaces that meet the criteria at http://www.linuxbase.org/futures/criteria/ . By meeting this criteria these interfaces should be widely accepted as appropriate for convergence by the LCC, addition by the Working Set, and for use by applications used in the BRT. The LSB Sample Implementation effort produces an implementation of these interfaces. The LSB Application Battery produces a set of open source application binaries expected to exercise the interfaces in the LSB.

Notes: Just as the LSB is a subset of the LCC and WS, the LSB-SI is a subset of what the LCC is converging on and the Working Set adding. The SI could serve FIXME The LSB AppBatt is currently build using one implementation and then run on the SI as well as Runtimes applying for LSB certification.

• A core set of interfaces that meet the LSB criteria,
• For those interfaces,
  • Written Specification
  • Components needed to develop applications that use the interface (stub libraries and header files)
  • Test Suite
  • Sample Implementation
  • Applications that exercise the interface
• A set of software required in addition the the LSB in order to create a "Minimal Working System" (produced as a result of the Sample Implementation efforts).

Linux Core Consortium (LCC)

Website: http://componentizedlinux.org/lsb

Mailing list: lsb-workers@lists.progeny.com

List info: http://lists.progeny.com/listinfo/lsb-workers/

Purpose: The Linux Core Consortium (LCC) is a group of Linux companies (initially Conectiva, Mandrakesoft, Progeny, and Turbolinux, with membership open to additional companies) collaborating to build a reference implementation of the LSB 2.0 standard. This implementation will be used as the basis of a platform on which to build Linux Distributions.

Benefit: Working to get 5+ implementations to converge on a common implementation of the existing LSB interfaces and in the future converge on additional interfaces. This will greatly help additional interfaces meet the "Phase 2" LSB criteria accelerating their inclusion in the LSB via the LSB futures process. These efforts may also identify deficiencies in existing LSB interfaces that can be directed to the LSB Specification Authority team for fixing.

In addition the LCC will attempt to bridge the gap between the "standard interfaces" approach of the LSB and the legacy "standard implemention" model that many developers are requesting.

Deliverables: Single binary platform, a set of packages both DEB and RPM, that is an implementation of of the LSB and can be used as the foundation for a Linux Distribution. Rather than attempt to implement the LSB interfaces yourself, you can use this platform to build a distribution that will be LSB compliant. This will be a superset of the LSB and what's required to create a working system.

• build processes
• package sets and rationale for selection

Getting the existing LCC partner distributions to sync up on interface version and patch levels.

OSDL Working Set SIG (WS-SIG)

Website: http://developer.osdl.org/dev/ws/

Mailing list: ws_sig@lists.osdl.org

List info: http://lists.osdl.org/mailman/listinfo/ws_sig

Purpose: The purpose of the Working Set SIG is to identify a collection of open source packages that are needed to create complete, functioning computing systems for the enterprise and to publish and maintain the list packages included in the collection.

The working set is intended to be used for the following purposes:

• As a baseline for binary regression testing

• As a guideline for allocating resources to intellectual property challenges to open source software
• As a guideline for granting licensing rights to intellectual property controlled by the open source community

Benefit: The difference between the ongoing definition of the working set and the current LSB will provide a list of most appropriate interfaces for the LCC effort to converge and eventually for standardization (either by the LSB application development standard, or some additional standard like a Distribution Best Practices or a System Administration Standard). The ongoing working set will also serve as a base for the binary regression testing project.

Deliverables: Set of packages beyond what the LSB and LCC provide that are required for the applications that the group is analysing.

OSDL Binary Regression Test SIG (BRT-SIG)

Website: http://developer.osdl.org/dev/brt/

Mailing list: binary_sig@lists.osdl.org

List info: http://lists.osdl.org/mailman/listinfo/binary_sig

Purpose: The purpose of the Binary Regression Test (BRT) project is to be able to execute regression test suites focused on specific proprietary binaries. Successes and failures will be mapped to the specific versions of software packages tested. A subset of these packages will be monitored and the test will be executed whenever they have changes. When failures do occur, information to be able to regenerate the system and regarding the cause of the failure will be captured. The source of the failure will need to be determined. This could be a problem in the application being tested, the test being run, or any of the underlying packages. The major goal is to shorten the passage of time in the development cycle between when a code change happens and when it is noticed to have an adverse effect on the execution of the binary.

Benefit: When the BRT discovers a test failure it is one of,

• application bug
• deficiency in the Working Set base
• deficiency in the LSB
• ABI divergence in build vs runtime implementation

This should result in a fix in either the application, working set, LSB, or distribution. In addition the group can publish warnings to alert developers about changing/deprecated interfaces that are coming in the future. Unlike the assertion based tests of the LSB, the BRT method tests actual applications using the working set as a base. This is similar to the LSB's application battery, but represents a more heterogeneous array of build environments. This testing will provide additional coverage for the promise of binary portability and an alarm for when things break. The BRT also provides a way to determine if the WS/LCC/LSB contain the interfaces that application developers are using. This is similar to what the lsbappchk and lsbdynchk tools do today.

Deliverables: Bug reports when problems are found to

• Application Developers
• Working Set group
• LSB
• runtime implementors

Notes: While the BRT it going to focus on testing on the working set for now, the ultimate would be to run, and build where possible, on a matrix of different build and runtime implementations.

Interface sets diagram

As described above, all groups produce sets of interfaces, each building on the next. As releases happen the LSB set expands and the other sets add addition interfaces and expand the overall scope.

The above is a simplified view of the sets that makes things easier to understand. Reality is probably more like this,