Package Management: System in a Package

After a week of working on the package management support as per my contract with Haiku, Inc., I have reached my first milestone: The base system is packaged... and it boots again. Here's a short account of what I have been doing exactly and what I'm up to next.

First of all, anyone watching Haiku's main source repository for anything I have done so far or will do in the next weeks will be disappointed. In anticipation of our switch from subversion to git as our source control management system and to ease joint work on the package management feature, Oliver has migrated the package management branch to git and made it available on GitHub. I'm currently working in my pm-flat branch ("flat" due to the fact that Oliver squashed all subversion changesets into a single git changeset to simplify the migration).

As the first step I started to get the package building tool, "package", working on the build platform (so far only tested on Linux). I originally wrote it as Haiku program and later Oliver refactored and extended the code into a proper Haiku package kit. After getting the package kit and the "package" program building and actually working on the build platform, I butchered the build system to produce a Haiku package (HPKG) file, "haiku.hpkg", and put it into "/boot/system/packages" in the Haiku image.

This package -- weighing about 52 MiB -- contains the whole core system, including the Haiku kernel, boot loader, runtime loader, the libraries, servers, add-ons, drivers, lots of CLI programs, applications (Tracker, Deskbar, ShowImage,...). That's most of the previous contents of the "/boot/system" directory. For the remaining "/boot/system" software separate packages are built. E.g. one for the "Welcome" and one for the User Guide documentation. Furthermore there's a basic development package, containing the header files, a few static libraries, the glue code, and a bit of adminstrative infrastructure for development (symlinks, scripts). That leaves the ICU libraries and the BeBook, which are downloadable packages that I simply repackaged from the pre-existing zip files.

When looking into the image file the "system" folder is extremely spartan now. It only contains "haiku_loader" (the boot loader) and a "packages" subdirectory with the six packages I mentioned. "haiku_loader" itself is also in the "haiku.hpkg" package, but since it would prove very tough to implement package file support in the stage one boot loader (those eight hundred something bytes in the first two blocks of the partition that load the stage two (i.e. main) boot loader), we keep the unpacked version for the time being.

Since the core system is now packaged, the boot sequence needs to work somewhat differently. The stage one boot loader is unchanged (yet). It still loads the stage two boot loader, "haiku_loader", and hands over control to it. The stage two boot loader, however, has to understand package files, now. It looks for the "haiku.hpkg" package and extracts anything that's needed to get the system up and running -- the kernel and a bunch of fundamental kernel modules and drivers. The kernel required only one change: After mounting the boot volume, it does now mount the packagefs in "/boot/system". Packagefs virtually extracts all packages it finds in the "packages" subdirectory and lays their unified contents over "/boot/system". Thereafter "/boot/system" looks as we knew it and the boot process continues as usual.

Well, there are a few differences. First of all "/boot/system" is now read-only. Conceptually it should have been read-only before as well, but it wasn't really. E.g. the MIME update task that runs on the first boot is no longer able to assign the correct MIME types to the documentation files. I.e. we will have to do that already in the build system when building the packages. ATM packagefs doesn't support queries. Besides "/boot/system" being a blind spot for Tracker "Find" now, there are also other consequences, like the Deskbar replicants no longer showing up. Obviously that needs to be rectified eventually, though it might not present an immediate obstacle for continuing work on the package management infrastructure.

The next two features I will be working on are, however, vital for that infrastructure. Both are extensions of the packagefs itself. The first is support for "shine-through" directories. ATM when packagefs is mounted at "/boot/system" the "packages" subdirectory will be hidden by the package contents tree that is laid over the "/boot/system" directory on the disk. Hence one cannot add anymore packages there. While that might be less of a problem for "/boot/system" it is major one for "/boot/common" and "/boot/home/config" which will soon work exactly the same way. There are subdirectories -- not only "packages" but also "settings" and others -- that have to be writable and whose contents have to live on the disk. The plan is to let those directories "shine through" from the normally hidden directory on the disk. That's something that might best be implemented with the help of the kernel's VFS subsystem. I'll have to investigate that.

The other feature to be implemented soon are automatically created (virtual) symbolic links for all active (aka installed) packages and their dependencies. This aims at solving two problems both caused by the fact that we have three different installation locations for packages (system, common, and home). The first problem is that some software requires absolute paths to other software (e.g. to an executable). Since at build time it is not known where the other software will be installed, packagefs will provide a fixed location that doesn't depend on where the package is installed. That fixed location will probably be in "/boot/system/package-links" and, for shorter paths, with a symlink "/package-links pointing" to it.

Say for example the "autoconf-2.68-1" package, which desires an absolute path to the "m4" program, is installed in "/boot/common", packagefs will provide a (virtual) directory "/package-links/autoconf-2.68-1". That directory will contain a symlink "cmd:m4" which points to the installation base directory of another package that provides the "m4" command, e.g. the "m4-1.4.16-1" package also installed in "/boot/common". The absolute "m4" path that can be built into "autoconf" is therefore "/package-links/autoconf-2.68-1/cmd:m4/bin/m4". If instead of the m4 package in "/boot/common" a different one is installed in "/boot/home/config", packagefs will simply let the "/package-links/autoconf-2.68-1/cmd:m4" symlink point there and the same absolute path still refers to a valid "m4" program.

That second issue the package links will solve is somewhat related. If several different versions of a package, e.g. for m4, are installed, and another package needs a particular (or minimum) version, the package link will point to the installation location of the matching one. This use case is not quite as urgent as the other one -- it can be put off until the whole package building infrastructure is set. This first one is more pressing though, since it must already work when we build packages.