Package Management: Queries, Solutions, and the Status Quo

Alas the time of my contract with Haiku, Inc. is up. So besides giving some details about the query support and libsolv port which I've been working on since my last post, I'll also summarize where Haiku's package management stands now and what still needs to be done.

Implementing the query support in packagefs turned out to be a bit more work than anticipated, since the ramfs code I recycled wasn't quite in as a good shape as I hoped, respectively needed more adjusting to fit in packagefs than expected. Now packagefs has the three standard built-in indices for file name, size, and last modification time. Furthermore indexing arbitrary other attributes is implemented. It is currently only used for a fourth index for the "BEOS:APP_SIG" attribute. That's the one needed to find applications by their MIME signature. I couldn't come up with any other attributes that would make sense to be indexed in packagefs.

While query support in itself is nice to have -- e.g. when searching files with Tracker's "Find" dialog -- my hope that certain applications that use queries internally would now work nicely again was only partially fulfilled. "lpe", the command line tool synchronously invoking the Pe editor for a file to edit (commonly used by developers e.g. for svn/git commit messages), works fine now (it queries for Pe by application signature). Deskbar replicants still don't show up, though. Apparently the mechanism it uses to track the replicants is to tag their program/add-on files with a special attribute. A live query for files with that attribute provides an up-to-date state of activated replicants. The query part would work now (if we'd index the attribute in question), but since packagefs is read-only, adding that attribute to files doesn't work. Since the current method isn't going to fly with multi-user anyway, it needs to be replaced by a different one, e.g. a simple settings file in the user's settings directory.

I also worked on porting libsolv -- until recently known as libsatsolver -- to Haiku. It will be the engine of our package manager's dependency solver. Often the question comes up what's so complicated about whole the dependency solving. After all it's only about also (recursively) installing the packages a package depends on, right? Well, there are a few additional conditions: Packages can also conflict with other packages. Different packages might require different versions of another package (e.g. a library). There can be abstract dependencies (e.g. something like "web browser") which can be satisfied by several packages. Multiple repositories need to be supported, as well as package and whole system updates. And worst of all, given all the repository and package information, there must be some useful result in case of problems, i.e. user comprehensible sets of options for how to proceed instead of just a "sorry, there's some conflict" message. That's the service libsolv provides.

The current state of the port can be found in my Gitorious repository. The libsolv library, which contains the data structure management for package and repository data and the solver, didn't need much porting. It's companion library libsolvext, which loads platform specific package and repository formats (RPM, DEB, and others), however, needed support for Haiku's formats. That's basically finished. Repositories, information about installed packages, and individual package files can be imported to libsolv's internal format. There's a test program, "solv", which is a simple but, save for repository management, complete command line package manager. It needed a bit of porting as well, which is mostly done. The package installation and deinstallation part is still missing. And lacking a set of properly built packages I haven't done much testing yet, so there's likely still some bug fixing to be done.

So, what's the current state of package management? Haiku is "package aware", being able to boot off of a volume that contains the core system as well as all software in "/boot/common" in packages. The basic tools for building packages ("package") and repositories ("package_repo") are working. There's the beginnings of a command line package manager ("pkgman") which is able to manage repositories.

Missing on the system side of things are the dependency solver (to be based on libsolv) and the whole package installation/activation process, including a package management daemon which deals with package changes done by the user via the file system (i.e. packages manually added to/removed from one of the "packages" directory). After that is done "pkgman" can be completed.

Missing on the package building side are the higher level tools for package building, most notably one that takes some kind of build recipe, prepare a clean room (chroot) environment with only the dependencies of the software installed, and build the package from its sources. A tool on top of that would build a set of packages or even all packages of a repository in an order implied by the dependencies. To solve the chicken and egg problem that certain packages are always required for building (like the compiler and other basic tools), there must also be a tool to cross-build certain packages on another OS (respectively another Haiku version). And, of course, the tools alone don't suffice. We'll also have to create and verify the build recipes for the dozens of packages we rely on.

On the repository side there's a bit of work left, too. We'll have to think about how we want to organize repositories for different Haiku versions and the Haiku trunk. We'll also need a strategy for how repositories are updated and maybe a bit of tool support to make that more convenient.

Finally, there're plenty of odds and ends. Like a GUI companion for "pkgman", fixes for various problems (like the Deskbar replicants), support for application bundles, performance optimizations, if necessary, etc.

So there's still lots left to do for Haiku's package management solution to be ready for prime time. The next big milestone will be reached when the package installation/activation and package building are working. At that point the core package management system will be what one could call self-hosting. I'll try to continue working towards that goal in my spare time.