Package Management: The New Season Starts

Blog post by bonefish on Mon, 2013-03-25 19:05

After quite some delay Oliver and I have finally started our contracts with Haiku, Inc. to continue our work on package management. Each of us will work 320 hours in total, i.e. the equivalent of 2 months of continuous full-time work.

We probably won't quite work full-time, so it will take a little longer. One reason for that is that, due to delays on our customers side, we still haven't finished our previous contract. There're still a few things to do which depend on their feedback (bug fixing, documentation updates, etc.). But instead of further postponing the Haiku contracts we've decided it would be in everyone's best interest for us to start now and just pause (or reduce work hours) as needed.

Last Friday Oliver and I met up to discuss the state of things and how we intend to proceed. The run-time support for package management in Haiku (in the package management branch, of course) is in pretty good shape already. With the system itself and all the third-party software living in packages the system boots and is fully functional.

So first of all we're going to focus on the package building side of things. ATM all third-party packages are just repackaged versions of the ZIP files Haiku used before. Back in 2011 I started to extend haikuporter to produce actual packages instead of ZIP files and update the BEP files (build recipes) for some fundamental tools accordingly. My first task, now, is to continue updating BEP files until at least all the packages needed for building packages can be built this way. Then I will have enough packages to complete the work on libsolv and the dependency solving.

Oliver will start working on haikuporter itself and adjusting/rethinking the BEP file format. Since Haiku doesn't have (and probably won't have any time soon) a horde of developers continuously maintaining and testing all third-party software and their dependencies, we'll greatly benefit from borrowing important package meta information -- what dependencies packages have, what build features should be enabled/disabled, what bugs and quirks are to be considered, etc. -- from other, larger OSS projects. We (well, mostly Oliver) have checked out MacPorts, pkgsrc, Gentoo-Portage, and others for that reason and came to the conclusion that we can't copy any of those verbatim. E.g. because they lack resolvable declarations (instead only declaring dependencies between packages), because they include too many specifics for the respective OS/distribution, or both. Also, most of them provide a lot more features than we need for Haiku, resulting in more complicated build recipes. We also came to the conclusion that the Gentoo-Portage EBuild files allow the easiest extraction of the information we're interested in, since they are just shell scripts declaring variables and functions.

One thing we might change is to turn the BEP files into proper sourceable shell scripts -- we were surprised to notice they almost are, already -- and switch haikuporter from Python to shell and/or C++. ATM the tasks haikuporter performs -- parsing the bep files and executing commands -- benefit very little from being done in Python (turning BEP files into shell scripts, we even save the parsing completely). Python, however, is a complex dependency and removing it from the package building process might save us some trouble. Oliver will do some more research and work out a good solution.

Our first important common goal is a haikuporter that, given a package name, can automatically build this package and all its dependencies (the dependencies first, of course). Or consequently, given a list of package names, it can build a complete repository. After that we can work on the repository infrastructure, improve the run-time support (i.e. installing and removing packages), migrate the Haiku build system fully to package management (e.g. get needed packages from the correct repositories),...

Finally a few pointers for those who want to keep themselves informed:

  • Oliver's and my GitHub repositories. The "Haiku" repository has a "pm-flat" branch (will probably be renamed eventually) which contains the latest PM version of Haiku. There are also the "buildtools" with a "package-management" branch matching the PM Haiku. The "libsolv" repository is a Haiku port of libsolv, the dependency solving library.

  • The HaikuPorts repository has modules "haikuporter" (the tool) and "haikuports" (the build recipes and patches), both featuring a "package-management" branch which reflects the current state of our work on those.

  • The Haiku Trac Wiki sports several PM related pages which we'll try to keep up-to-date and extend.

Comments

Re: Package Management: The New Season Starts

This is great news! I'll be following the progress very closely, and I'm sure I won't be the only one! I can't do much on the programming side of things yet, but I'd like to help in whatever way I can.

2013 is going to be a great year for Haiku. =]

Re: Package Management: The New Season Starts

Just wondering how optional dependencies will be expressed with the package files. With optional I mean additional packages that add new features, but are otherwise not required for the package's function.

There might be optional dependencies at compile time as well as runtime, although probably only the former would be of interest for haikuporter.

Re: Package Management: The New Season Starts

Optional dependencies can be expressed via a supplements attribute, which tells the package manager that the package in question supplements another. This only applies to runtime dependencies.

Compile time dependencies aren't part of a package, they are only declared in the build recipe. The current idea is to avoid optional build time dependencies as much as possible, and instead try and pick a reasonable set of supported features.

Inspecting other existing build trees has shown that at least some of them are supporting quite a lot of optional build features (e.g. for something curl, you can decide if you want ssl support and if so, which of the 5 or so available implementations it should use). The ssl options will most likely be dealt with by picking openssl as our default ssl implementation and activate that for all packages where it is an option.

On the other hand, curl has optional support for using kerberos as authentication mechanism. If we disable this option (which I think is rather likely), any user that requires kerberos support has to manually adjust the build recipe.

Re: Package Management: The New Season Starts

Just wondering - where will the official package repository be and how will devs get their apps into it? Something like how HaikuWare does it?

Re: Package Management: The New Season Starts

There will be a small set of official repositories, one of which will contain the core packages (everything that comes with a Haiku release) and probably another one that will be filled from the haikuports tree.

Haiku applications from other vendors will most likely be provided by something like Haikuware, albeit accessible as a package repository.

Re: Package Management: The New Season Starts

We will need a tool for third partys to create haiku packages even if they only keep them on their web site and the user must download the package file to install it.

Re: Package Management: The New Season Starts

bbjimmy wrote:

We will need a tool for third partys to create haiku packages even if they only keep them on their web site and the user must download the package file to install it.

Isn't that the package manager fork of Haikuporter?

On a side note, I remember the package manager fork being called HpkgBuilder (or something like that) back in 2011, is it now going to keep the HaikuPorter name?

Re: Package Management: The New Season Starts

The tool for creating a single package from a built application is called package. This tool reads a package-info file (a textfile containing the description of the package and the files it provides) and then builds a package from it.

Concerning the name of haikuporter, I'd rather stick with that name, as IMHO the name HpkgBuilder does not convey the purpose of the tool as clearly. Haikuporter is meant for porting OSS software to Haiku and I believe the name conveys that well.

Re: Package Management: The New Season Starts

Very nice! Thanks for providing this overview at the time the work starts. I am very excited about the progress that these contracts bring to Haiku! Is Oliver's repository also streamed to the haiku-commits list, or are you both working in Ingo's repository?

Re: Package Management: The New Season Starts

Oliver's GitHub Haiku repository is registered as well, so his commits will appear on the haiku-commits list. ATM he is working on haikuporter and the build recipes for ports, though, so you'll have to subscribe to the HaikuPorts commit list to get mail notifications for his work.

Re: Package Management: The New Season Starts

Awesome! Sooo looking forward to Haiku package management, also the implementation as described seems very nice and 'Haiku-ish'.

Re: Package Management: The New Season Starts

Hi,

As a potential future contributor, I wanted to chime in and encourage you to reconsider moving away from Python for haikuporter. I'm not active in Haiku development (though I've been keeping an eye on it for a few years), so I'm not speaking from a position of knowledge or experience with particulars with regards to Haikuporter, but I do have a non-trivial amount of software development experience, with a heavy emphasis on Python in the past 5 years. (http://www.ohloh.net/accounts/ivanov, https://github.com/ivanov)

You wrote "removing [Python] from the package building process might save us some trouble" -- which might be true with respect to Python being a non-trivial dependency on Haiku, but what you would be doing is substituting that trouble for a lot of trouble for potential contributors. Python, from a developer's stand point, is a dream for being able to jump in and make contributions. Module organization, doc strings, nosetests, batteries-included standard library, etc. all contribute toward that, and tools like IPython make spelunking in an unfamiliar codebase seamless and fun.

Moving the package to bash or C++ would be a lose in terms of all of those. C++ would be even worse by adding a compile cycle between changesets. By sticking to Python, you will be standing on ground that swelling up under you, as the packages you leverage improve, and new ones get developed.

Re: Package Management: The New Season Starts

Don't worry, we will stick with Python for haikuporter.

What made us think about possible alternatives was the fact that haikuporter needs to be extended to use chroot()-ed environments for doing actual builds. From experience, chroots tend to become complicated with respect to dynamic loading of modules (i.e. loading a module isn't possible from within the chroot()).

In this particular case, however, all the Python modules we use are part of the standard library (so they do not add more dependencies) and neither of them seems to use dynamic loading internally, such that hopefully there won't be any problems once the chroot environment has been entered.

Re: Package Management: The New Season Starts

zooey wrote:

Don't worry, we will stick with Python for haikuporter.

That's too bad. I had hoped you were moving away from python. It has crossed my mind (more than once) to rewrite haikuporter in another language.

Re: Package Management: The New Season Starts

augiedoggie wrote:
zooey wrote:

Don't worry, we will stick with Python for haikuporter.

That's too bad. I had hoped you were moving away from python. It has crossed my mind (more than once) to rewrite haikuporter in another language.

I'm still very new to programming, but is the only reason some want to move away from Python speed? Is the increased speed/performance certain other languages (like C++) have significant enough to warrant a complete rewrite of HaikuPorter?

Re: Package Management: The New Season Starts

Is there any particular reason why you think we should move haikuporter away from Python?

Re: Package Management: The New Season Starts

zooey wrote:

Is there any particular reason why you think we should move haikuporter away from Python?

Just personal preference. I'm not a fan of python and I didn't like having to deal with it when I needed to work on haikuporter.

Re: Package Management: The New Season Starts

Just popping in to tell you that it's just fantastic to see those massive commits rolling into the PM branch almost daily! I keep wondering how damn long it would take Haiku to get package management in place if there were not this contract. Outstanding work, guys!

As we're now about 2 weeks in, how about a quick update? While I see hundreds of lines of code hitting the branch, I understand very little of the commit messages...

Thanks!
Humdinger

Re: Package Management: The New Season Starts

I second this, it would be nice to have some sort of update on the progress of this project, no matter how small it is. But if the update is being held back because you guys are nearing a milestone, then I am definitely