End of package management contract

Blog post by zooey on Sun, 2013-11-10 16:47

From mid-June to early November, I have spent 320 more hours working on package management. After having worked on the bootstrap support in HaikuPorter (which Ingo has already mentioned in his blog entry), I have spent some time reworking the way how Perl and Pythong organize their modules: Scripting languages with module support usually expect to be able to build modules from source and install them somewhere into the hierarchy of the scripting language. In our case, they can no longer just put the built (site-specific) modules into the "standard" folder, as that one is read-only. So the configuration of Perl and Python had to be changed, such that modules built from source are being installed correctly into a writable folder (in the non-packaged hierarchy). Perl already supports the notion of a vendor-modules folder, where packaged Perl modules will be installed (an example of such a beast are the perl modules contained in the git package). That idea has been extended to Python (such that it will automatically pick up the python modules provided by the mercurial package). Separating packaged modules from local (built from source) modules is an idea that should be followed by all (not only scripting) languages ported to Haiku.
The updated perl, python, git, mercurial and scons packages that are the result from that work have not yet been published, but I will do that during the next couple of days.

The package management branches of both haikuporter and haikuports have been merged into the respective master, i.e. all porting work done in there will produce packages. Several new 'testing-' branches will be added to the haikuports repository soon in order to be able to separate work on the bleeding edge from the work that tries to stabilize a given set of packages for a release.

Finally, I have started with work on the infrastructure required for automatic building of repositories and (at a later stage) packages. The first effect of this is that whenever a developer changes the version of a package used by Haiku's build system, the respective repository will be created automatically. Those repositories are being published to http://packages.haiku-os.org/haikuports/master, from where pkgman will download packages.

Summarizing those months of work: I think we have managed to put most of the work behind us, but we haven't reached the goal just yet. There's still more work to do and I personally will return to continuing that work in my spare time.

Comments

Re: End of package management contract

Awesome work Oliver!

Danke vielmals!

Now that you and Ingo have prepared the grounds, many others can follow.

Is there any documentation on how to bring about the inclusion of something not already packaged for Haiku?
"From nowhere to HaikuDepot and back again."

Re: Danke vielmals!

Re: End of package management contract

do you have an estimation how long it could take till package manager can update/upgrade the system?

Re: End of package management contract

Thanks for all the hard work!

Re: End of package management contract

It sad after all money spent:

"we haven't reached the goal just yet"

so many times we hear this, why can't user usable results be acheived for money paid?

Re: End of package management contract

I think the topic of poor estimation of labour required for coding tasks has been well covered in a book called "The Mythical Man-Month" - or something like it - I remember seeing this in a discussion about 5 years ago.

One positive here is that Oliver has indicated that he will continue advancing the task in his spare time. This is great commitment to Haiku and I hope he will get more spare time than anticipate.

Re: End of package management contract

There never was any estimate of how long package management was going to take to be fully implemented -- at least not by Oliver or me -- nor were there any promises made.

The purpose of the contracts was to advance PM, since it is part of the planned R1 feature set. The hope was that the contracts would advance the PM implementation far enough to allow it to be merged back into the main development line (master). That has happened.

While there certainly still are plenty of rough edges, in many aspects (particularly wrt. to package building and installing (recall installoptionalpackage?)) the current state already far surpasses pre-PM Haiku.

Understanding the scope and goals of Haikus form of PM

It's easy to fall into the bloat/failure/money wasted sentiment if one sees package management as merely a software find/install method. It's more than that. The scope of the PM project was expanded to meet some very elaborate goals and the complexity of this can be understood by simply looking at all the areas of Haiku which have seen work in relation to package management.

Rather than have a clear-cut base system and a base-dependent set of packages on top of that base system, it was decided to make the base system itself packetized. This adds complexity in bootstrap and in the build system, but the idea here is to have system upgrades as part of the overall package management lifecycle rather than be two separate software management domains, so to speak. It might fall under the terms of rolling release or rolling updates; a continually developing software system.

Re: End of package management contract

I think the fact that Haiku has moved from having no sense of package management in May to being fully package-aware a in November is a huge achievement and my congratulations go out to all those that achieved this huge goal. Sure, it will take a while longer for package management to be "feature-complete", but that's pretty understandable given the scale of the task