Package Management: Bits and Pieces

Blog post by bonefish on Mon, 2011-07-04 13:28

Time does fly. It has already been two weeks since my previous blog post, so I guess an update is in order. The support for shine-through directories and the package links has been implemented as planned and works nicely as far as I have tested them. Other than that I've worked on a few odds and ends.

The most notable one is probably that, with a few exceptions, I've repackaged all gcc 2 built optional packages that are are installed as part of the alpha build profiles. Their dependency declarations (and probably other package attributes) aren't correct yet, but other than that they are all proper HPKG files and placed in "common/packages" now. Consequently the "common" tree is also mounted via packagefs when booting.

Another item I've tackled is the build system update feature ("jam -q @myprofile update ..."), which broke with the introduction of package support. The feature allows to just build one or several files and replace them in the image, as opposed to rebuilding the whole image. Since almost everything lives in the system package now, that meant building the whole system package and replacing it in the image. After adding add/replace functionality to the "package" command and kneading the build system a bit, individual files will now be updated in a package and the package will be updated in the image. Probably not that fascinating for an end user, but my time for updating only a few components dropped from about 30 seconds back to significantly less painful 4 seconds.

I also had a first look at libsatsolver, which will likely become the library that will power our package dependency solver. API documentation for it is unfortunately non-existent. Luckily the sources come with an example program that basically seems to be a full-blown command line package manager with back-ends for RPM and DEB package and repository formats. I'll have to see how well the library will fit with our formats and how much it needs to be adjusted to satisfy our needs. That's definitely something I want to look into before my contract ends.

The other thing I'd like to look into -- and I think I'll even attack that first -- is the query support in packagefs. Initially I intended to postpone that until more of the infrastructure was finished, but I miss it a bit more than I thought. The implementation shouldn't be that complicated. Most of it can be recycled from the BFS sources. Merely the indices need to be written from the scratch and some glue code needs to be massaged into place.

I've added a bit more documentation to the Wiki, including a general overview of the infrastructure and some details on building packages.

Comments

Re: Package Management: Bits and Pieces

Thanks for the write up and the nice wiki entry.

There is some talk that package management could be used to solve the GCC2/GCC4 problem. Any thoughts on this?

Re: Package Management: Bits and Pieces

Thanks for the update.

Re: Package Management: Bits and Pieces

That depends a bit on what problem you're referring to. What I consider the main issue of the status quo is that gcc 2 and gcc 4 optional packages aren't as cleanly separated as they should be. First of all there should be no mixing between gcc 2 and gcc 4 compiled libraries and programs at all (i.e.a gcc 2 program must not use a gcc 4 library and vice versa). This is already solved to some degree in that for most optional packages we have both a gcc 2 and a gcc 4 version. That is not sufficient, though. On a gcc 2 hybrid Haiku one cannot cleanly install a gcc 4 optional package, since some files need to be put into different directories (e.g. libraries into "common/lib/gcc4" instead of "common/lib"). So basically one has to build a gcc 4 package specially for a gcc 2 hybrid Haiku (and vice versa).

That would already be possible and package management won't change anything in this respect. What will change with the introduction of package management, however, is that we will create tools to automatically and consistently build packages. Hence building four versions of the same package won't be as much of a pain as it is now. In fact the tools will even be able to automatically build a complete package repository from the scratch.

ATM we're still quite a bit away from that point, though. Currently only a plain gcc 2 Haiku builds at all. I haven't repackaged any of the gcc 4 optional packages as hpkg yet and the hybrid support in the build system is broken. We'll probably start looking into gcc 4 when the (high level) package building tools have been created.

Re: Package Management: Bits and Pieces

OK, that's a nice description of the roadmap.

Do I understand correctly that any and all classic BeOS apps would need to be manually packaged? Or could this process be automated?

It seems that once all BeOS apps are correctly packaged to use gcc 2 libs, this could solve the BeOS compatibility issue going forward, yes?

Re: Package Management: Bits and Pieces

AndrewZ wrote:

Do I understand correctly that any and all classic BeOS apps would need to be manually packaged? Or could this process be automated?

Those apps that are just self-contained zip files don't really need to be repackaged. Unzipping anyway them will continue to work. If one prefers a bit or organization, one would unzip them in "/boot/common/non-packaged/apps". Repackaging them has hpkg files would be fairly simple. Automating that is not so simple though, since package files require a bit of meta info (usually provided in the form of a .PackageInfo file) that has to written. If the required info is already available in machine-readable form, generating a .PackageInfo from it automatically would be possible, of course.

BeOS apps that come as a pkg (SoftwareValet/PackageInstaller) file can be a bit tricky though. At least if they have some kind of installation script. Those scripts often hard-code the installation path and might do things that don't work anymore. Those apps might have to be repackaged manually with great care.

Quote:

It seems that once all BeOS apps are correctly packaged to use gcc 2 libs, this could solve the BeOS compatibility issue going forward, yes?

That depends a bit how an application was written. E.g. if it hard-codes a path that has changed in Haiku, there will likely be some trouble. Maybe it is possible to run such an application in some kind of "compatibility jail". I guess the best approach is to move forward and see what applications misbehave and then decide how to solve the issues.

Re: Package Management: Bits and Pieces

Is anyone working on the website and repository where packages will be uploaded, stored, community reviewed and retrieved?

Re: Package Management: Bits and Pieces

Not to my knowledge.

Re: Package Management: Bits and Pieces

If we have package-links we don't need three "packages" directories, and if we have the "packages" directories we don't want package-links - developers will ignore them. We need to make a clear choice!

Besides that makes too many magical directories, please keep the filesystem structure simple.