Package Management: The Return of the Hybrid

Blog post by bonefish on Mon, 2013-08-12 16:37

Although I have been a lazy blogger lately we haven't been lazy working on our remaining tasks at all. So, unsurprisingly, since my previous post we have reached and passed a few nice milestones. The latest one is that we're finally able to build the gcc2/gcc4 hybrid Haiku images again, including all the software needed for the official release.

While that in itself isn't a particularly impressive feat -- after all we were already able to build the complete gcc 2 part before -- the interesting aspect is how we are doing it. First of all the build system no longer "cheats" when building a hybrid image. For building a hybrid image in the current Haiku master branch separate gcc 2 and gcc 4 builds have to be configured and the build system then "borrows" files from the other configuration. Also the needed optional packages of the other configuration are simply reused.

With package management we separate the hybrid's primary and secondary architecture cleanly ("architecture" in anticipation of things like x86-64 with x86 support; currently it's just two incompatible compilers for the same hardware architecture). The build is configured with the compilers for both architectures and the build system knows what to build with which. Furthermore there are dedicated packages for the secondary architecture. E.g. the ncurses package built with gcc 4 for a gcc2/gcc4 hybrid differs from the ncurses package built with gcc 4 for a gcc4/gcc2 hybrid. This is necessary not only because the files in the primary and secondary architecture package would otherwise clash, there may also be things like data files that we don't want to duplicate and therefore leave out in the secondary architecture package.

Commands, libraries, and add-ons for the secondary architecture live in an architecture subdirectory of their usual location. For commands it's just to avoid clashes, for libraries and add-ons the directories are also significant to the runtime loader. It will only search libraries and add-ons in the directories matching the architecture of the program.

haikuporter has grown support for building packages for the secondary architecture and the build recipes for the required software have been adjusted accordingly. There are still some open questions with respect to organizing certain types of files in secondary architecture packages (documentation, certain development files), but those will be answered when we have gained more practical experience.

But back to the build system: Besides having support for hybrid builds it also retrieves packages from the HaikuPorts repository. Yes, there is one now! In fact there are already multiple versions of it. Whenever packages are added to it, removed, or updated a new version is generated (currently it has to be done manually). This way it will be possible to check out older revisions of Haiku and still have the matching repository for it. Those repositories are fixed. For official Haiku releases we'll have mutable repositories in which packages get updated over time. The freshly built Haiku is pre-configured with the matching repository, so that the package manager is immediately usable.

And finally we've also added support for bootstrapping Haiku to the build system. If the build has been configured respectively, a jam -q @bootstrap-raw builds a Haiku bootstrap image. The process will not download any pre-built packages. Instead haikuporter is used to cross-compile bootstrap versions of the essential packages -- all that's needed to run Haiku (ICU, zlib,...) as well as the basic build tools. It also builds source packages for all software needed for a regular Haiku image and adds them to the bootstrap Haiku image. This way the bootstrap Haiku can be used to build the regular packages. We still have to do exactly that for the architectures for which we don't have any packages yet (x86 gcc 4 and x86-64).

Once that's done I think we should very soon merge the package management branch back into the master branch. Package management isn't "complete" yet, but it should be reasonably usable and there are only a few smaller known issues/regressions it introduces. It would definitely benefit from wider exposure at this point. So, everyone please feel encouraged to test it already and shout (via Trac), if you find your favorite feature or use case broken. I've started a wiki page that lists the changes users can expect by switching to a package management Haiku.

On a related note, Stephan Aßmus has started working on HaikuDepot, a GUI package manager application which will also provide functionality like screenshots and user ratings. He has posted a first screenshot of the GUI.

Comments

Re: Package Management: The Return of the Hybrid

That is awesome news! Looks like we will have Package Management to show off at BeGeistert! Will this help with porting Haiku to other architectures like ARM and PPC? I am bubbing with excitement over this. Perhaps Beta is only a few months away!

So at this point, can packages be installed/upgraded/removed? Can the whole system be updated/upgraded to the newest revision?

ddavid123

Re: Package Management: The Return of the Hybrid

The bootstrapping support will help with porting to new architectures in that there's now a well-defined, automated method to create the initial packages. That had to be done manually before and was probably rather tedious. The biggest hold-off with the ports, however, is that there's no one actively working on those. There's only some sporadic activity from time to time.

Yes, packages can already be installed, updated, and removed via the command line package management tool (pkgman). There's more work to be done to complete the functionality and make the experience more user friendly.

System upgrades aren't supported yet. Implementing the functionality in Haiku shouldn't be that much work. We also need to build the repository providing the updated system packages. I think it makes most sense to work on that after we've merged the package management branch back to the master.

Re: Package Management: The Return of the Hybrid

For those curious, a new set of images ("r20130812_4c6b3ef") have been uploaded to http://www.haiku-files.org/unsupported-builds/package-management-preview/

They were built with configure --use-gcc-pipe --use-xattr --build-cross-tools x86_gcc2 ../BuildtoolsPM --build-cross-tools x86

Re: Package Management: The Return of the Hybrid

Awesome! Thanks Matt!

Re: Package Management: The Return of the Hybrid

Looks great! Amazing how fast the installation is, now that mostly big package files are copied!
I was thrown at first a bit at the blueish folders in the Deskbar. Opening one I thought it was a query because of its grey background. But those are some sort of virtual folders. Can you expand on that?

Oh, and the read-onlyness of apps already bit me: I like to check the "Background app" setting of LaunchBox to exclude it from Deskbar. Saving the setting fails now of course (silently)... :)

Regrads,
Humdinger

Re: Package Management: The Return of the Hybrid

Yes, the Deskbar uses virtual directories now. The different window background and the blueish icon serve the purpose to avoid confusing them with regular directories. If anyone wants to improve the visuals they are very welcome to do so.

Similarly to stored queries the virtual directories are actually just files with a special MIME type. For a query file the query string and other information are stored in attributes. The specification for what is displayed in the virtual directory is, however, stored in the file itself -- its just a plain text file. Here's how the definition file for the virtual directory used by default for the Deskbar menu looks like. It's just a list of "directory ..." entries listing the paths of the directories that shall be merged virtually. Should we switch to (deeper) categories for the Deskbar menu, people who prefer a flat list could just define a virtual directory that flattens the categories.

That you don't even get an alert that changing the app info failed is definitely a bug. That it can't be done anymore cannot be helped, though. I guess Deskbar would have to grow an additional preferences GUI to allow excluding applications, if that feature was considered desirable.

Re: Package Management: The Return of the Hybrid

That it can't be done anymore cannot be helped, though.

This means that adding the package manager is removing some features and power from the user. There must be a way to extract the package, change the executable attributes, and re-package it. or at least extract it and use it as a "normal" app. I too use the background app setting for LaunchBox. This is how Haiku is supposed to work, place the power in the hands of the user.

Re: Package Management: The Return of the Hybrid

Yes, the package could be extracted, changed, and rebuilt. The issue in this case isn't that package management takes away a feature, but that this either wasn't a feature in the first place (just a mechanism that could be abused for this purpose) or it wasn't designed properly. User settings should not be associated with system/globally shared files, because 1. this obviously doesn't work in a multi-user environment and 2. wrt. system safety and security having system files being writable by users by default is an anti-feature.

As written before, such a feature should be implemented where it belongs: in Deskbar, with the settings being stored in the user's settings directory.

Re: Package Management: The Return of the Hybrid

I absolutely second Ingo's take on this. The Deskbar needs to be changed, so that entries have a context menu. This is where I would place options such as "Pin to Deskbar" (making LaunchBox superfluous) and also "Hide from Deskbar". Having to change some obscure flag in the application resources is completely non-intuitive and just the wrong place for a user-setting.

Re: Package Management: The Return of the Hybrid

It seems perfact to me as it is, right click, Add-ons, FileType, click the setting. It has nothing to do with the deskbar, but changes the way the application presents itself to the system.

Re: Package Management: The Return of the Hybrid

In case you havent noticed, Haiku is not multi-user. and in most use cases, neither are Windows, Linux, or Mac. Why wory about this arcaine mainframe idea of multi-user on a personal computer? If one really needs to share his computer and is concerned with his files or settings, he can make a "guest boot partition" and boot it up when needed. This takes only seconds in haiku. Multi-user is far more complex than any benefit it may bring to a personal computer.

just my 2 cents.

Re: Package Management: The Return of the Hybrid

Actually, you would be completely incorrect there, every single one of the systems you mentioned except Haiku are in fact multi-user and have been since many years. That's exactly how e.g. UAC on Windows works, or the password prompts on OSX/Linux when you go to do something like install an application or something else that affects the system. It's not necessarily so much about several people using the system concurrently as it is privilege separation.

Re: Package Management: The Return of the Hybrid

Did you miss "use cases?" All those password prompts are just the user experiance I don't miss when using HAIKU. It is my computer, if I want to do something, even if it might be the wrong thing, at least I can do it without having to be pestered to death by the operating system.

That might not be what you like, but that is the way I like it. Let me make my own mistakes.

Personal compuiters should just work, not make me type a password, use su or sudo or some such nonsence just to get done what I need to do. Let me decide what is the right thing to do with my computer, and let me take responsibility for my own actions.

Re: Package Management: The Return of the Hybrid

+1

I've been using computers since the Timex Sinclair. If I fubar my computer I can handle that. If the OS annoys me like a nagging wife, I will format and move on!

Re: Package Management: The Return of the Hybrid

These discussions are so frustrating when they get so easily out of hand and lose focus. We were talking about changing an obscure setting that defines application behavior just to hide something from Deskbar. That is no longer possible with read-only apps. I remember one time I loaded up a BeOS app in a hex editor and changed a string directly in the binary. I guess that is no longer possible either for apps installed via packages.

Let's try to keep the discussion on subject! Yes, package management forces a cleaner design on the system (as in better layering and separation). It has advantages and drawbacks such as the ones outlined above. The fact that package management forces the same cleaner design as true multi-user would force, has nothing to do with UAC prompts or having to use sudo at the command line. Nobody said that package management would lead to that. You are still root in Haiku.

I am the author of LaunchBox for crying out loud. I made the decision that it's not a backgroud system service. Blame me! It has windows on screen, they might be different per workspace, there might be multiple pad windows on the screen, they might be hidden below other windows. So I absolutely stand by my decision to let LaunchBox have a Deskbar entry so you can get to those windows. Stop bashing package management for nonsense reasons, please!

Re: Package Management: The Return of the Hybrid

humdinger wrote:

Oh, and the read-onlyness of apps already bit me: I like to check the "Background app" setting of LaunchBox to exclude it from Deskbar. Saving the setting fails now of course (silently)...

I'm confused. Isn't that setting saved to "/boot/home/config"? Is that directory read-only now?

Re: Package Management: The Return of the Hybrid

No, it's part of an attribute/resource in the executable itself.

Re: Package Management: The Return of the Hybrid

Oh. Eww. :(

Thank you.

Re: Package Management: The Return of the Hybrid

Awesome work Ingo and Oliver !

Thanks Matt !

I can finally start to understand what is going on.

Just one problem : I was unable to boot the ISO in Virtual Box (r20130812_4c6b3ef) : no partition found. Is it related to the stage 1 boot loader problem ?

The VMWare image boot without problems though.

There is only one remaining problem : i will have to learn how to make my own recipe to package my software ;-) I will start tomorrow !

Re: Package Management: The Return of the Hybrid

We haven't looked into the creation of ISO and anyboot images yet. They have separate sections in the build system, so it is entirely possible that something needs to be adjusted there. Off the top of my head I can't think of any special handling needed in the boot loader, but I may as well be missing something.