Package Management, First Draft

Blog post by zooey on Sat, 2011-01-08 19:38

The following is a first draft of how package management on Haiku could look like. I'm more than sure that there are aspects missing here, but we need to start the discussion somewhere ...

Several people have already added their ideas on package management to the wiki article. Additionally, many opinions have been stated in comments to my first blog entry.

I have tried to incorporate most ideas mentioned into this draft, but due to the sheer amount of contradicting views, some sacrifices have to be made.

HPKG and Package-FS

Haiku's package format (hpkg) has been designed explictly to support swift access to the contained file-hierarchy and -contents in order to facilitate mounting one or more folders containing such packages via Haiku's package-fs.

What happens if you mount a folder full of *.hpkg files is that the package-fs will combine all those packages to construct a (read-only) hierarchy of files and directories at the mountpoint. Moving packages into/out of the package-folder will adjust the mounted hierarchy accordingly (live).

In order to see how well this concept scales, I have experimented with two sets of hpkgs, which were converted from pacman packages: the small set had 168 packages amounting to 260 MB and the large one had 1052 packages amounting to 2.4 GB (all sizes refering to the compressed package files). On my testing machine, package-fs managed to populate the small tree in 2 seconds with an empty file cache and in 0.6s with a full cache. The large set took 16 seconds to populate after boot and 3.2s on second try. With keeping in mind that package-fs currently doesn't do any caching by itself whatsoever, these are very promising numbers. I'm pretty positive that with added caching, we could bring down the delay for mounting a large package-fs to levels which are acceptable as part of Haiku's boot process.

A modified folder hierarchy

In order to be able to support user-specific as well as system-wide activation of packages, and in order to clearly separate the cumbersome, dependency-ridden ports hierarchy from the rest of Haiku, I've found it necessary to remodel Haiku's basic folder hierarchy to some extent.

Here's my filesystem layout suggestion (please excuse the ascii-art):

/boot
|                     [/boot/apps is gone - system-wide applications now appear
|                      in /boot/common/tree/apps]
|-common
|---develop           [moved here from /boot/develop]
|---packages          [dropping a package here will activate it system-wide]
|---settings          [common settings]
|-----backup          [room for setting backups - used by package manager to backup settins during an upgrade]
|---tree              [mountpoint of system package-fs, contents of the packages in /boot/common/packages appear here, the 
|                      folders follow "Haiku filesystem hierarchy" (to be defined, an example given below)]
|-----add-ons
|-----apps
|-----boot
|-----data
|-----documentation
|-home
|---packages          [dropping a package here will activate it for user]
|---settings          [user-specific settings (N.B.: may come from apps that live in /boot/common/tree)]
|-----backup          [room for setting backups ...]
|---tree              [mountpoint of user package-fs, contents of the packages in /boot/home/packages appear here, the 
|                      folders follow "Haiku filesystem hierarchy"]
|-----add-ons
|-----apps
|-----data
|-----documentation
|-ports
|---packages          [dropping a (port) package here will activate it]
|---tree              [mountpoint of ports package-fs, contents of the packages in /boot/ports/packages appear here, the 
|                      folders follow whatever facilitates easy porting, i.e. the hierarchy basically is unixy - what's shown below
|                      is just an example]
|-----bin
|-----boot
|-----etc
|-----include
|-----lib
|-----share
|-----tmp
|-----var
|-system              [unchanged, contents can be updated via package manager by actual installing (unpacking) of either a complete
|                      "system" package or sequential "system-fix" packages]
|---optional          [moved here from /boot/optional, as the contents depend on the system]

The most obvious change is the ripping out of the ports-related stuff from /boot/common into a separate hierarchy. Basically, this means that /boot/common no longer has to contain Unixy paths like 'etc', 'share', 'var' and the like. After all, Haiku isn't Unix and we can chose our own folder layout. The ports subhierarchy can be modeled such that doing the actual porting of Unixy software is as easy at it can be.

Early during the boot process, Haiku would populate the three package-fs mountpoints and could then use all the software contained in any of them. Inter-dependencies between these the four sub-systems (/boot/sytem and the three package-fs mountpoints) are as follows:

- /boot/system does not depend on anything else.
- /boot/common/tree contains Haiku-specific applications/frameworks and may depend on anything in /boot/system, but not on anything in the /boot/ports. If any software requires Unixy components (libraries), it must bring them along as part of its own bundle.
- /boot/home/tree is the same as /boot/common/tree only for a single user
- /boot/ports/tree will most probably never depend on anything else but the POSIX-/libc-specific parts of /boot/system, i.e. libroot.so & libbsd.so. Unixy software doesn't know about Haiku kits, so it won't use them.

So how would this be used then?

Judging by the comments to my initial blog entry, it seems clear that many people do not want to have anything to do with the package management system at all. They'd like to be able to download Haiku-apps from any odd website and then start them by double-clicking the downloaded file.
Fair enough, when you invoke a package, Haiku would temporarily mount this package by means of package-fs and then start the application.

Installation of a downloaded bundle would mean to drag'n'drop it into /boot/common/packages or /boot/home/packages, depending on whether it shall be available system-wide or just for per-user.

Unixy software with dependencies on other packages demands for support by a package management tool, so if you'd like to install such software, you start that tool (either the console- or the GUI-version), search and select the software and then tell it to install it. Taking Subversion as an example, the package management tool would automatically download APR-util, Neon, LibIconv, LibXML2, OpenSSL and SQLite and place all of these packages into /boot/ports/packages. Unpacking would be done by package-fs and the 'svn' command would be available immediately (in /boot/ports/tree/bin).

Since the package-trees are immutable, you can no longer simply fiddle with any of the files. In order to change something manually, you'd have to find the hpkg containing the specific file you'd like to change, move that hpkg to some place else, unpack it, change the contents, repack it and then move it back to the .../packages folder.

System Upgrades

System upgrades would be done by the package management tool (maybe with a separate UI just for that task). A system upgrade would be executed by downloading a single package that contains the whole /boot/system folder. This package would be unpacked to /boot/system_next, /boot/system renamed to /boot/system_last and /boot/system_next to /boot/system. These renames aren't atomic, but the bootloader could surely be adjusted to cope.

For bug- & security-fixes, small "system-fix" packages could be created, which contain changed files that would be unpacked on top of the existing /boot/system folder.

The package management tool

The package management tool, as I imagine it currently, would know about all the available and installed hpkgs (by means of queries), but clearly separate Haiku applications from ports.
When updating any software, the pm-tool could zip an eventual settings folder of that software and store that in a backup folder, such that these settings can be restored when a downgrade should ever be done.

The implementation details of the actual tool are still vague (to say the least), but it is clear that it will only ever be relying on the actual hpkg files that live on the system. Any special markups like locking of specific packages (in order to keep them from being upgraded) will be done as attributes on those hpkgs. For performance reasons, it may be necessary to use some kind of cache for the info contained in all those packages and probably a service that manages the cached info and answers requests for that data to any interested party.

Many other aspects (handling of drivers, support for application-mimetypes via package-fs, writable folders in /system/ports) are explicitly not mentioned here.
First we need to settle on the general idea, so please attach your flames as comments or send them my way ;-)

Comments

Re: Package Management, First Draft

Well I like that we are considering the bundle as viable... we like call hpkg but it is OK!

One thing it is unclear if I not install it, but double click on it from the download dir it works, right (I imagine Haiku mounts it in the application tree in background, so in some sense is not so different from installation...)?
So application are portable too :-)

I'm right with the thing to hide the ugly unix structure as WE CAN... I always hated to see that links on /.

But I'm of the idea that ported apps should adapts to Haiku, not the contrary -:)

Unixy software with dependencies on other packages demands for support by a package management tool, so if you'd like to install such software, you start that tool (either the console- or the GUI-version), search and select the software and then tell it to install it. Taking Subversion as an example, the package management tool would automatically download APR-util, Neon, LibIconv, LibXML2, OpenSSL and SQLite and place all of these packages into /boot/ports/packages. Unpacking would be done by package-fs and the 'svn' command would be available immediately (in /boot/ports/tree/bin).

OK but in my mind this too can be realized as bundle a particular one maybe (a simple zip file at the end as it use the unixy tree) and dependency set as attributes...

I know I can be annoying but what's happens if application is not in package manager?

If it is a bundle (or hpkg) I download it from the developer site and "install it" haiku see it depend for say on "QT" (as we like to abuse this example) package manager is started it installs QT then the application is mounted in the Haiku and then it start!

In this way we can have Unixy application portable, too!

Obviously I not say Package Manager not has to have a search tool and so it is a commodity...but it should not the only way to install an application...

Re: Package Management, First Draft

You've definitely done your homework. Thank you for even making the first draft look quite usable, even if it's a little rough around the edges. I'm confident you'll make this better than any other package management system currently available. :D

Re: Package Management, First Draft

A good first draft in my opinion, my favourite section was "A modified folder hierarchy", I'm really glad you've tackled refining the file system structure now as it's the right time to do it (better now than when we've got tons of packages).

I'm still working on a wiki to collect my own package manager thoughts together, so I'll try to save repeating myself on this blog post.

One question, with regards to this:
"For performance reasons, it may be necessary to use some kind of cache for the info contained in all those packages and probably a service that manages the cached info and answers requests for that data to any interested party."

Could we not use saved filesystem queries for this, rather than a cache? A service would update the query results. For example, the first time a HPKG is run, saved queries are updated, scanning the folder containing the new HPKG file, that would help keep the data up to date without relying on a central cache. Would mean refreshing the data (if/when required) would be as easy as re-running the filesystem queries.

Re: Package Management, First Draft

Zeno Arrow wrote:

One question, with regards to this:
"For performance reasons, it may be necessary to use some kind of cache for the info contained in all those packages and probably a service that manages the cached info and answers requests for that data to any interested party."

Could we not use saved filesystem queries for this, rather than a cache? A service would update the query results. For example, the first time a HPKG is run, saved queries are updated, scanning the folder containing the new HPKG file, that would help keep the data up to date without relying on a central cache. Would mean refreshing the data (if/when required) would be as easy as re-running the filesystem queries.

I'm not sure what you mean by saved filesystem queries - the only thing that can be saved about a query is the actual query string, the results will have to be collected everytime the query is run. We could of course use a query to collect packages, but we'd still have to collect the attributes from all the entry_refs (package files) that the query returns. This would be a rather slow process, especially when compared to reading the required structures directly from a cache file (where the package service dumped it last time it ran).

My original idea was that the package-fs could notify the package service of any changes (activation/deactivation of packages) and that service could then update the cache file. Thinking about that again, now, I can see that caching this info in a file wouldn't make much sense at all, since the package service can always request that info from all the mounted package-fs instances, which should be able to provide that info swiftly. So no need for caching the installed packages' metadata separately.

Re: Package Management, First Draft

zooey wrote:

I'm not sure what you mean by saved filesystem queries - the only thing that can be saved about a query is the actual query string, the results will have to be collected everytime the query is run. We could of course use a query to collect packages, but we'd still have to collect the attributes from all the entry_refs (package files) that the query returns. This would be a rather slow process, especially when compared to reading the required structures directly from a cache file (where the package service dumped it last time it ran).

My impression was that Haiku could save both the query string and the query results from the last time it was run. For example, if you have a query that pulls together all of your MP3s from across your file system, does it build fresh every time? I thought the results from such a query were quicker than a new system search could provide. I take it I'm wrong about this then?

zooey wrote:

My original idea was that the package-fs could notify the package service of any changes (activation/deactivation of packages) and that service could then update the cache file. Thinking about that again, now, I can see that caching this info in a file wouldn't make much sense at all, since the package service can always request that info from all the mounted package-fs instances, which should be able to provide that info swiftly. So no need for caching the installed packages' metadata separately.

Yes, that's true, it doesn't make sense to duplicate the package data. As long as you know where the packages are to be found then obtaining package meta data should be quick.

Re: Package Management, First Draft

That's a very good brainstorming.

BTW I have some ideas that might be helpful for the task.

- I think that there's a need to a system-wide updater tied with the package manager which has to be some sort of Windork updater but not only for the OS but every package can register with it to take care of the updates instead of having the application bundle an intrusive updater and having hundreds of per-application updater living in the taskbar and doing nasty things without our knowledge, that can eliminate security risks because we can deny network access to an application and being sure that the updater (the one with internet access grants) still updates the application. Applications can register with the updater with a special named text file with a simple format detailing the URLs of the "volatile", "security", "optional", "major", "minor" updates and checkout intervals etc ... So the user is only prompted for example when major update occurs and if he wishes he can know at anytime which application is downloading an update. The updates can be in the form of a compressed diff patch (for binary of text files) detailing the old hash and the new hash (and hashing algorythm too) of the concerned files or a complete files replacement or anything that you may think of.

- Another application that connects to 'haikuware', some sort of front-end to the "bundles-store" (Following the hype of "software-stores") which is great by itself but everyone likes to download applications directly from his editor of choice.

- Also some thoughts about dependency hell : I think that we should overcome this by providing to every app its required shared libraries but not copying the same library to each application folder. I think that we can go the M$ way it did with .Net, many versions of the same library are available in the global assembly cache, we should find a way to have the library loader pick the right library bases on the version+hash for example. Another question my rise, when application foo requires library libbar1.1.so or higher (the higher and the lower version number of a required library can be in the bundle attributes for example) and after installing libbar2.0 which breaks compatibility with our applicatin foo, we can go back and recover libbar1.1 (because it had to be replaced by libbar2.0) and modify the attribute of application foo and change to dependency info to have the higher version number of libbar to be 1.1 and for example issue a compatibility bulltin that can be seen by the system updater so the system updater to every haiku installation in the world receives the information and updates the dependency information accordingly.

- Finally, if we install a new bundle from a CD for example and some dependencies are not existing in the system AND the internet connexion is not working (because in that case the package manager would prompt you to accept installing the missing stuff from internet), the package manager should give the useful information (about the already installed software)in a way we can download all the required software bundles (thinking about QT) and putting them in the same folder with the wanted application so the package manager takes care of installing everything when launching the installation the second time. This can avoid the "fat bundle" syndrom.

my 2 cents

Re: Package Management, First Draft

How are you planning to handle upgrades? In particular, I'm thinking of nasty edge cases such as a third-party application having a file handle open to a file in the application being upgraded. My Haiku fu isn't great, but certainly on Unices this would cause problems when unmounting/remounting the bundle.

Have you had any thoughts as to the other side of handling packages, which is how to build them, and what the metadata inside them is (install scripts, uninstall scripts, upgrade scripts, dependency management, etc)? Linux has discovered the hard way that it is not possible to be too anally retentive about package management: Debian's success is almost entirely due to being incredibly prescriptive about how packages are made, and having very high standards as to what packages are accepted. In the early days Red Hat didn't do this, with appalling results. rpms still have a terrible reputation because of this. (I invite interested parties to read the 20000 word Debian New Maintainer's Guide at http://www.debian.org/doc/devel-manuals#maint-guide.)

Re: Package Management, First Draft

@david.given
I'm not an expert when it comes to package build systems, but I do know that PKGBUILD (used in Arch Linux) has a reputation of being one of the best build systems to use, in that's simple to build a good package using it. This has led to a huge amount of packages for Arch Linux, many contributed by its user community (another sign of its relatively simple to follow nature). Probably not as many as packages as Debian, but Arch Linux certainly punches above it's weight in terms of a packages/user ratio. Also worth bearing in mind that Arch Linux is a rolling release distro, which means its packages have to be sturdy enough to deal with a fair amount of change under the hood.

So in my opinion, PKGBUILD could be a good model to get inspiration from. If you're interested to know more, you can read about PKGBUILD here:
https://wiki.archlinux.org/index.php/PKGBUILD

What do you think?

Re: Package Management, First Draft

david.given wrote:

How are you planning to handle upgrades? In particular, I'm thinking of nasty edge cases such as a third-party application having a file handle open to a file in the application being upgraded. My Haiku fu isn't great, but certainly on Unices this would cause problems when unmounting/remounting the bundle.

Unmounting a bundle could indeed cause problems, but I suppose (though I'm not sure) that on-the-fly mounting of applications could be implemented such that it flags the package-fs to automatically unmount when no FDs are open anymore.

Concerning the general upgrading process: I'd only ever upgrade application by the package manager that are actually installed (i.e. live in one of the 'packages' folders). In that case FDs for open old files of the application would have to be kept around (which is standard behaviour).
Applications that aren't installed would have to be upgraded manually by the user downloading the new release. During that, the application may actually run, too, since the browser would just download a different file.
In principle even starting of two versions of the same application would be supported by package-fs (it would just have to find a unique mountpoint for each), but that could still cause havoc unless the application is prepared to deal with two (differently versioned!) instances of it running. Such a setup is pretty likely to wreck the settings files, I suppose.

david.given wrote:

Have you had any thoughts as to the other side of handling packages, which is how to build them, and what the metadata inside them is (install scripts, uninstall scripts, upgrade scripts, dependency management, etc)? Linux has discovered the hard way that it is not possible to be too anally retentive about package management: Debian's success is almost entirely due to being incredibly prescriptive about how packages are made, and having very high standards as to what packages are accepted. In the early days Red Hat didn't do this, with appalling results. rpms still have a terrible reputation because of this. (I invite interested parties to read the 20000 word Debian New Maintainer's Guide at http://www.debian.org/doc/devel-manuals#maint-guide.)

Yeah, I did read that (well, most of it) a couple of days ago, actually ;-)
The thing is that the details of building the ports tree depends a good deal on what the people responsible for those ports (in our case: the Haikuports team) are willing to accept. I agree that we need to create some pretty tight rules for package creation, but it doesn't make sense to force that. All I can promise is that I will initiate a discussion on the haikuports list in due time.

Considering the exact package metadata, I haven't spend much thought on those yet. One thing that did cross my mind, though, is the question about on what terms we declare dependencies between packages: do we mimic openSUSE's zypper with full gory declaration of each and every library provided by a package, or do we take the simpler approach of apt & pacman (mostly just making packages depend on other packages, not explicit files)? The latter is simpler, no doubt, but is it good enough? Currently, I suppose it is, but please speak up if you think otherwise.

Why the separation?

Hi zooey,

thanks for your reseach work and preparing the draft! If I may say so, I think you are trying to please they naysayers a bit too much. In my opinion, a self-contained package (i.e. one without dependencies) is the same as a bundle or a traditional selfcontained .zip Haiku/BeOS application. As such, I see no reason at all to make a distinction between native applications and ports. From your posts, it looks this was only motivated because so many people expressed they want selfcontained downloads of apps. It doesn't really matter whether the reasons for this are considered uninformed, the fact is that once package management is introduced at all, distributing an application as selfcontained package is already implicitely possible and does not need to be a special Haiku native applications restriction. The second reason seems to be to keep the unixy file system layout out of /boot/common, but I personally don't care so much. I'd be ok with a separate /boot/ports folder, as long as package management is not reserved for ports only.

Re: Why the separation?

I agree with you stippi, it makes sense to me to have a unified package manager across applications (ports and native) and system. App bundles are a subset of the main package management functionality, as long as the system can extract/mount packages in a background process then people who want to avoid other package management features should be happy.

Re: Why the separation?

Hi stippi,

as relating to the package manager, the separation between bundles and ports indeed is only superficial. Basically they are just the same thing, maybe going to end up on different tabs in the package manager GUI. That kind of separation could be done along some other lines, too, like type-of-software (application, desktop applet, library, ...). From the package manager's point of view, all these are just hpkgs, some of which happen to have dependencies, while others do not.
Of course applications that were downloaded from some odd website are a bit more difficult to support in the package manager, as there's no repository info for it. But the general idea was to provide support for many different repositories, such that the package manager could fetch and manage software offered by Haikuware and similar websites.

My motivation to separate ports from Haiku software in the filesystem is that this would make it much clearer for developers of Haiku applications where exactly in the filesystem they should put the data of their application. When I look at the contents of /boot/common as it is currently, I find it a bit messy: there's etc and settings, so where does an application programmer put common configuration files? Then there's documentation and share, so where does documentation for Haiku applications go and which format is preferred (currently it's a wild mixture of info,man and some html)?
Splitting off the ports tree would give Haiku more breathing room in /boot/common and OTOH the Haikuports team would be free to define /boot/ports as it fits their needs best. The Unixy filesystem layout is being forced on us, but we may as well chose to contain it in a little cage instead of allowing it to spread throughout the system.

Furthermore, anyone working on a Haiku application should not be easily tempted to use ported libraries. Instead, they should look at our kits and try to use them (ideally helping to improve them where needed). If a developer thinks he can't do without a ported lib, he's free to use it, but will have to make sure to bundle that lib with his application (or declare its dependencies and rely on the package manager getting things right).

Haiku PackageFS and intentionally writable files?

Hi again,

forgot another question that popped up while reading your draft: Do you know if/how PackageFS handles config files that come as part of a package and are supposed to be installed in the real file system and be editable by the user?

Re: Haiku PackageFS and intentionally writable files?

stippi wrote:

Hi again,

forgot another question that popped up while reading your draft: Do you know if/how PackageFS handles config files that come as part of a package and are supposed to be installed in the real file system and be editable by the user?

I don't think package-FS currently has support for that, so since the mounted packages are immutable, no package could provide config files directly. One possibility would be to let them provide config-file templates in a specific folder, from where the user would have to copy the file into /etc (I think Debian does this at least to some extent).
More convenient would be to implement special support in package-fs, such that it would handle config files in the hpkg file by copying them to the respective folder (/etc in most if not all cases). Automatic backups during upgrades could be handled this way, too, if the config file/folder(s) carried an attribute describing the exact package to which they belong. Parts of this could perhaps be moved from package-fs to the package service running in user space.

Re: Package Management, First Draft

Thanks for sharing Oliver. I think best to post this to General Mailing List to get some developer feedback to the idea. You've already heard enough I think from average users & making other developers happy should come first or they'll quit.

I'm really not clear which way you're going. I'm tired today and maybe not following you well.
On the one hand you say:
"If any software requires Unixy components (libraries), it must bring them along as part of its own bundle."

Which sounds like self-contained application bundles. On the other hand you say:

"Taking Subversion as an example, the package management tool would automatically download APR-util, Neon, LibIconv, LibXML2, OpenSSL and SQLite and place all of these packages into /boot/ports/packages. Unpacking would be done by package-fs and the 'svn' command would be available immediately (in /boot/ports/tree/bin)."

I prefer the Linux way myself and think you are saying the same with above quote?

I understand you are trying to make the folder structure easier but not liking it right now. I'm used to /boot/common with unix subfolders but then maybe I'm having trouble picturing something different right now. I not thrilled about putting Unix binaries into another folder (/boot/ports/tree/bin) because I believe will confuse me. Right now I see "system" as OS programs & libraries, "common" as group programs & libraries, "home" as user programs & libraries and makes logical sense to me. I think your way may make Haiku feel more like Linux but you should get other opinions on that to confirm.

System upgrading sounds very good. You might want to look or check binary diffing using rsync. You copy /boot/system to /boot/system_next & then binary diff /boot/system_next to latest. Something for you to look into and consider as another way to update the system. Don't forget about /boot/common, /boot/develop, etc. because files may change in there & you'll have to merge to the latest versions.

Re: Package Management, First Draft

Technically, any package, no matter if Haiku application or some port, could declare dependencies to other packages and the package manager would try to resolve those. But I suggest that it would be better if 3rd-party Haiku applications are full application bundles, bringing all their libraries with them (not the Haiku kits, of course).

My reasoning is that updating of a library through the package manager puts trust into this library's ABI not having changed from one minor release to another. If it *has* changed, however (which happens now and then) the application could no longer be started. While the same is of course true for all the ports depending on the same library, those ports are being controlled by us (the Haikuports team, to be precise). Hopefully, we are going to have some continuous build tool which builds and executes the tests of the whole tree in regular intervals. So if a library has violated its declared ABI, chances are that we'd notice and could do something about it.
For Haiku applications that have come from some other source (i.e. which are under our control), we could not do any of these tests, and neither could the developer of that app, as he'd probably not know about us having changed that library in an incompatible way. In effect, the users would have to notice and tell the app developer about it, who'd have to analyse the situation and then ask the Haikuports team to fix the lib. Adjusting his application and providing a new release would not be enough, as that version of the app would then be incompatible with older versions of the library - bummer! All he could do would be to put a compatible version of the library into his app bundle.

I believe mismatches between declared and real compatibilities are the real dependency hell, everything else can be solved by package managers. The large Linux distros try to solve it by simply providing all the applications in their own repositories. But since Haiku explicitly encourages 3rd-party developers to create and publish Haiku applications, we should try as hard as possible to avoid them getting broken by package upgrades. Updating those applications themselves with our package management tool is no problem, but managing the inter-dependencies of 3rd-party- and "our" software is a no-go.

There is a more general solution to this: let every package carry a "PKG:Vendor" and then let the package manager disallow any cross-vendor dependencies.

Re: Package Management, First Draft

About the mentioned dependency hell, mind you read my suggestions above ?

Re: Package Management, First Draft

Ok, things are more clearer now.

There is a more general solution to this: let every package carry a "PKG:Vendor" and then let the package manager disallow any cross-vendor dependencies.
I think you are saying to avoid exact library version dependencies between vendors which I agree with.

"better if 3rd-party Haiku applications are full application bundles"
I disagree. Open-source libraries should always be done as dependency packages. Only closed-source libraries should be bundled with the application. That should be enforced. First I will list why and then provide examples using libpng.so.

Issues with full-application/fat bundles
1) Same open-source library & version might be used by multiple fat bundles. So, you are downloading it multiple times. I don't want to install 15 applications that have libpng.so.1.4 or Qt-4.7, etc. bundled with them.
2) Can really waste disk space. The more applications that use exact same libraries the more space wasted
3) Not future-proof to protect against dependency hell.
4) Create larger download size for packages. That affects user's download & server load & download limit.

The simple solution is to package open-source library dependencies separately and have the program link to that particular dependency & use it. Two ways to use it, either when program compiled it knows to look for specific library version (like libpng.so.1.2) or if compiled to use only library name, create symlink in program library folder pointing to specific version (libpng.so->/xxxxx/libpng.so.1.2). That solves the application always using a specific version of the library to avoid incompatibilities in the future.

When developer (or porter) updates his program he should have it work with newer library version (to eventually remove old library versions from repo) and use library dependency package. Doing it this way removes 1) having multiple copies of exact same open-source library name and version on the system. 2) reduces disk space, package size, lower download time and size, etc.

If developers want to use newer open-source library then that is ok too. They package it separately which is included in package manager as program dependency. ie: PNG Library 1.5.0 (TESTING). Then other developers can choose to use default version (PNG-1.4.4) or newer (testing) library and you do not punish or force developer to use default library version.

But now you lose application portability between systems? Not really. Package manager can copy program folder to temp folder, find the (non-system) dependent libraries and combine both to create a portable (compressed) package version.

But a fat bundle will make sure that program always works! No, not really. If I compile a program today and it links to an included library in Haiku (libpng.so.1.4) and I provide a fat bundle without including libpng.so.1.4 because already on Haiku system. Then at future point in time when Haiku switches to libpng.so.1.6 as default (and assuming incompatible with PNG-1.4) then that application will not run and require re-packaging to add PNG-1.4. Same library situation we are facing today.

With library dependency you can have future-proof working applications and not run into issue above if properly coded which I can provide simple design solution for without any future application re-packaging.

Very Important:
Dependency hell happens because developer or porter has not updated program to work with newer version of incompatible library (normally with major but sometimes minor revisions). Can be because dev dropped support (don't maintain), or change happened too quickly or unexpectedly for update to happen, etc.

Solution is simply providing exact (major+minor) version of library linked during compile time to that program until program can be updated to use newer library version. If program was compiled with libpng.so.1.2 you give it that version and not force it to use default libpng.so (libpng.so.1.4). That can be done with library dependency packages which install library only once on system. Fat bundles do same thing but very inefficiently.

You can have have it all with very little trade-off. You can have efficiency, simplicity (for user) and working applications which don't break because of library changes.

PS Don't BeOS applications use libpng.so.1.2? Haiku switched to libpng.so.1.4 awhile back and broke compatibility with BeOS programs using older png library.

Re: Package Management, First Draft

The problem with undeclared ABI breakage may certainly come up from time to time. What may be a much more often encountered problem, though, at least later on, is security issues needing to be fixed, sometimes in applications, but more often in libraries. Encouraging application developers to publish complete bundles makes the situation very likely that the same library is security fixed on the system level, but several apps come with their own version which remains unfixed.

Shared libraries are shared for a reason. Yes, there are dangers (undeclared ABI breakage, anything else?), but the dangers of not using shared libs properly are far greater IMHO. Besides the dangers of not using them properly, there are the disadvantages like increased resource consumtion when what would ideally be the same code is loaded more than once in RAM, and increased application startup time when the application could re-use a lib already in RAM but loads its own copy.

All in all, I think encouraging native app developers to publish complete bundles is the wrong path.

Re: Package Management, First Draft

Great point Stippi, I posted those in the other forum post but completely forgot about them posting here.

Issues with full-application/fat bundles (continued)
5) difficult to upgrade for security fixes (libraries)
6) difficult to upgrade for bug fixes (libraries)
7) smaller issues - like greater memory use, slower load-up for applications that could have shared libraries

ABI breakage is very important to users and I solve this problem (simplest way possible) by providing older library versions to older programs. I cannot say this is the most ideal solution because most minor revision changes are backwards compatible (no breakage) but maybe better safe than sorry. A better solution is to provide older version of library only for those that have confirmed ABI breakage, just until program is updated to work again, but have not figured out how package manager could do this for single library for multiple packages.

Re: Package Management, First Draft

stippi wrote:

The problem with undeclared ABI breakage may certainly come up from time to time. What may be a much more often encountered problem, though, at least later on, is security issues needing to be fixed, sometimes in applications, but more often in libraries. Encouraging application developers to publish complete bundles makes the situation very likely that the same library is security fixed on the system level, but several apps come with their own version which remains unfixed.

Shared libraries are shared for a reason. Yes, there are dangers (undeclared ABI breakage, anything else?), but the dangers of not using shared libs properly are far greater IMHO. Besides the dangers of not using them properly, there are the disadvantages like increased resource consumtion when what would ideally be the same code is loaded more than once in RAM, and increased application startup time when the application could re-use a lib already in RAM but loads its own copy.

All in all, I think encouraging native app developers to publish complete bundles is the wrong path.

This thread on ycombinator raises (and squashes) this very point: http://news.ycombinator.com/item?id=2075714

If Haiku's kits are sufficiently featureful, then this is a non-issue.

Re: Package Management, First Draft

Moopark wrote:

This thread on ycombinator raises (and squashes) this very point: http://news.ycombinator.com/item?id=2075714

If Haiku's kits are sufficiently featureful, then this is a non-issue.

Quoting yourself to back up your own argument, how low can you get?

So your argument is make an API for everything and there isn't an issue. Well, Haiku does provide a pretty rich set of APIs (or certainly plans to), but it's not feasible to provide a native API for all possibly needed functionality, for example should Haiku have a Asterisk-like VoIP API just because some developers might eventually want to write a Haiku-native VoIP client/server?

On top of that, Haiku is a young OS, many of the gaps in its application library are likely to be filled by ports from other systems at first, it's not a matter of idealism but pragmatism. You have to allow for the fact that 3rd party libraries are needed, and have a good way to deal with those 3rd party libraries. Sticking your head in the sand and pretending it won't happen is not a solution.

@Moopark
On reflection, our discussions have gotten out of hand, I'm stepping away from this topic, probably the whole OS too. Apologies for being rude, I'm passionate about technology so its easy to overstep the mark when discussing it.

@all
Carry on discussing, but please bear in mind just because someone disagrees with you it doesn't necessarily make them wrong, nor does it necessarily make you wrong. Please consider if your remarks are helping move the discussion on or whether you're stuck repeating yourself.

P.S. FWIW, here's the wiki I was working on, I'm unlikely to finish uploading my package manager ideas to it now, but just as proof it exists:
http://projecttanka.wikia.com/wiki/Project_Tanka_Wiki

Re: Package Management, First Draft

Post deleted by author.

Re: Package Management, First Draft

I would like to mention the biggest 'danger': added complexity.

Completely ignore technical details for a moment. Think about how it's used by the average user.

Download an application, double click, get prompted with a list of dependencies that you don't understand/care about, wait while they download, and then run the application. To uninstall, drag to trash (with a prompt about uninstalling settings), at which point the same list of dependencies appears that you did't understand/care about before, and wait while they uninstall. Not sure how you'd figure out how big the application and the depenedencies are - a right click menu that loads the package manager? That's the most optimistic scenario, most package managers are much more intrusive, and quite slow.

Download an application, double click to run it, and it runs. To remove, you drag it to the trash (with a prompt about uninstalling settings). The size of the application is the size of the bundle. Conceptually, really simple. Everyone who has ever used a computer can understand that. Lightning fast too - no having to wait to install or uninstall. I can imagine showing people that and them going "wow" at how easy and obvious it all is.

No installation. That's a killer USP for Haiku. A clear differentiator with obvious benefits to the user. No other OS has that.

Re: Package Management, First Draft

System Upgrades

"System upgrades would be done by the package management tool (maybe with a separate UI just for that task). A system upgrade would be executed by downloading a single package that contains the whole /boot/system folder. This package would be unpacked to /boot/system_next, /boot/system renamed to /boot/system_last and /boot/system_next to /boot/system. These renames aren't atomic, but the bootloader could surely be adjusted to cope"

Beautiful - This gives control to the dev's and allows for depreciation and removal over time of systems lib's and api's to keep everything in there nice and clean. I tend to find updating an OS only add files and does not get rid what you don't need anymore.

Re: Package Management, First Draft

A thought occurred to me. Is it possible to create 'immutable' file attributes? Or attributes that can be locked? I'm thinking that system and package files should all have a package and version attribute so users and the system can easily tell which version of a package a file came from.

Consider: all of Haiku 1.0's files would have package attribute 'Haiku' and version attribute '1.0.0.0', including its settings files. 1.0.0.1's updates would have '1.0.0.1' version. When 1.0.0.1 is installed, it upgrades settings files as needed and creates new versions of those files with version 1.0.0.1 attribute. Uninstalling the 1.0.0.1 update would involve renaming the 1.0 settings files to, perhaps, *.ORIG, then downgrading the 1.0.0.1 settings files to 1.0.0.0, then deleting all files with attributes package='Haiku' and version='1.0.0.1'. Other packages and OSS imports would employ a very similar method.

Running any program would be as 'simple' as running the one with the greatest version. Programs would use the library versions they were programmed with.

Such a scheme could make a PMS easier to implement.

Food for thought.

Re: Package Management, First Draft

You will remove the /boot/apps folder? Wow, thanks, I lost my hope in Haiku now. I might as well use Linux where everything is system-wide.

Re: Package Management, First Draft

sparklewind wrote:

You will remove the /boot/apps folder? Wow, thanks, I lost my hope in Haiku now. I might as well use Linux where everything is system-wide.

It is so a big deal?

In the end it is only moved to /boot/common/apps... are application shared by all user so "common" seems the right path to me!

For the 3rd party thing... I don't understand if it is suggested the best way is a fat bundle (and sorry I continue to think it is!)... well then Package Manager what has to do?

I don't think we have to left too chooses: bundles with dependency, fat bundles, packages at Linux way (I have to change an /etc file by hand in Haiku? It is not so haikuish for me...) I fear to choice make confusion in the end :-)
I have the impression we are talking in circle, too...

For the configuration of "unixy" application it is not possible to solve the problem using link in writable dirs? An Haiku application I think must put its configuration files in the user home folder for example /boot/config/webpositive. Suppose I install a ported app (let's call bubu) it, as of its unix legacy, wants to put its configuration in /etc and in our equivalent /boot/home/ports/tree/etc we find bubu.cfg if we make it a symlink to /boot/home/config/bubu/bubu.cfg? This is obviously writable and the application configuration is in the user directory as an Haiku native application...

Re: Package Management, First Draft

It's not about whether the "app" folder is /boot/apps, /boot/common/ or /boot/home/apps. The issue is that we're removing the concept of having self-contained apps in one folder, and instead polluting the whole system. Apps that don't need to change anything in the system should stay self-contained. This works fine for most emulators on Windows for example. Check this one out for example: http://bsnes.googlecode.com/files/bsnes_v073.7z

It's all elegant and portable. Just extract to a folder and run. This I think is ideal. This doesn't mean we have to scrap everything that has to do with package management. We can still let the packages have repositories that let you update them easily. But I want to be able to put my apps wherever I want them, I want to be able to edit any files they have manually, and I want to be able to run them immediately without installing anything extra onto the system, and they should be unaffected if I do a clean reinstall of the system. If the app needs an external library, put it in APPFOLDER/lib and be done with it. Why do we need a bunch of black magic like dependency resolution, system-wide installs, etc, that make everything a hassle?

Re: Package Management, First Draft

If I understood well the Haiku Native application remains self contained in their directory as possible... in a multiuser future Haiku if an application is for all users is in /boot/common/apps and is obviously (for me) that configuration file(s) in the /home folder... it seems more right to me... we can think of put in the application directory if we want that is /boot/common/apps/myApp/configuration/fano for the fano user and /boot/common/apps/myApp/configuration/xyz for the xyz user.
Put configuration in home folder is MORE in favor of the portable thing... if I put application in a USB key I not want someone see my app configuration... I can give to a friend it installs (or play directly form the key) application create the relative configuration folder in /boot/home...

The thing is that folder content should not be visible to users at all! The user has no means to see this content a real Haiku application has a GUI you not have to mess with editing files by hand...but using the settings in the File menu :-)

So we can use the pure BUNDLE that is a simple directory with a particular structure and a mime/type so the system know it is a bundle and not a normal directory and shows it as a file... I think they are writable, too so after the configuration is modified it can be saved in the bundle itself... (it makes sense for the update side of things... you update the bin or the lib only and leave the rest in peace :-) )

The problem is with ported application we have 2 type:

- GUI applications: they use a not-native framework QT, JAVA, Wine... if they want me to edit a file for configure them (in the messy /etc folder) they are not well designed application in my opinion: they should use the GUI (and I add the shuld be seem native as they can!). They MUST forced to be self contained... do it wants to write configuration in /etc? OK... this is NOT Linux... the configuration is in your same folder (or in /boot/home/myApp/conf)... it needs QT and want to install it in /lib? ehhr NO, it can't modify the OS... put it in your application bundle...

The other possibility (more risky) is to have some certified and tested frameworks following anyway the Apple example: Java is a big beast... Apple release a certified version of JAVA (yes is Apple to do it NOT SUN/Oracle!) and application not put it in the application bundle as they know there is always a JVM in OSX, tested and guaranteed that it works.
The same reasoning can be done with other frameworks so a "Certified" QT, VXWidgets, Wine and so on... the updating of them will be done only when sure, for not broke anything...

The OS is not modified by them... can be put in the port tree a Frameworks folder to put this giant beasts and not native application depends on them...

As I said is more risky... you can be careful as you want but the risk to broke an application is always possible: s\it happens...

- CMDLine application the majority of them are simple one file utility as perl, awk... they not need configuration... real application as for example apache (that for same reasons is called httpd) is the ugliest thing for file pollution in the file system... I'm not sure we need this sort of application on a Desktop OS... we need to install Apache in your windows PC? I never have to do this... so we neded to do this in Haiku?
These are server things leave it to Linux.

Re: Package Management, First Draft

Well, my main concern is not configuration files, but everything else, like external libraries. I don't want anything to get installed system-wide unless it's really necessary. If you open the archive in the link, you should get an idea of what I mean. If configuration files are the only thing that gets installed in another folder, then that shouldn't be too big of a deal.

When it comes to ported apps, the same thing should apply, really. The lib subdirectory inside the app's directory will get a little bigger if an app uses a foreign GUI toolkit for example, but that's still better than polluting the system by installing it system-wide. If you find the extra space it needs to be too much, then you should use an app that is more native instead.

I don't think we need to install Qt or WxWidgets system-wide. The few .so files that you will need to include with an app are small enough. Wine is different though as it's a huge compatibility library that needs to run in the background. But we could still isolate Wine somehow and not put its files all over the system, I think.

Re: Package Management, First Draft

sparklewind wrote:

Well, my main concern is not configuration files, but everything else, like external libraries. I don't want anything to get installed system-wide unless it's really necessary. If you open the archive in the link, you should get an idea of what I mean.

Yes, your bsnes example is my favorite type of application: no install, I decompress where I want and it works... it need a supergameboy.dll? It not put in C:, its useful for it only and then is its directory... simple!
(Well in this case in reality Windows says "bsnes.exe is not a Windows application" :-) archive corrupted maybe?)

sparklewind wrote:

If configuration files are the only thing that gets installed in another folder, then that shouldn't be too big of a deal.

Yes, I think configuration is the only thing can be out of the bundle... but in a clear path /boot/home/myApp/myApp.cfg not in a strange /etc path.

sparklewind wrote:

When it comes to ported apps, the same thing should apply, really. The lib subdirectory inside the app's directory will get a little bigger if an app uses a foreign GUI toolkit for example, but that's still better than polluting the system by installing it system-wide. If you find the extra space it needs to be too much, then you should use an app that is more native instead.
I don't think we need to install Qt or WxWidgets system-wide. The few .so files that you will need to include with an app are small enough.

In reality I have to say I've noted to have in my download directory decompressed (what a coincidence!) bsnes 0.70 and it is a QT application... in its direcoty I find 2 dll relating QT:

QtCore4.dll    657 Kb
QtGui4.dll    2574 Kb 

The entire bsnes folder is very huge... 10.2 MB! Wow is this we are speaking for days? For add QT dependency I have to add 3 MB? Well I can survive to this... in the worst case I install 100 QT applications I occupy 300 MB :-)

sparklewind wrote:

Wine is different though as it's a huge compatibility library that needs to run in the background. But we could still isolate Wine somehow and not put its files all over the system, I think.

Yes as Java is but they can self contained, yes... instead of the port tree, if the fake unixy directory is in the bundle itself?

MyApp.bundle
     ----- bin: myApp
     ----- lib: test.so qt_core.so
     ----- usr/share/pixmap: myApp.icon
     ----- etc --> /boot/home/MyApp/config/: myApp.cfg
     ----- var/log --> /boot/system/var/log

In this way the unixy compatibility links are gone... if application need them write it in the bundle!

@thatguy that imagine is a fantastic description of what we are speaking...

Re: Package Management, First Draft

fano wrote:

Yes, your bsnes example is my favorite type of application: no install, I decompress where I want and it works... it need a supergameboy.dll? It not put in C:, its useful for it only and then is its directory... simple!
(Well in this case in reality Windows says "bsnes.exe is not a Windows application" :-) archive corrupted maybe?)

Weird that you got that error, maybe the download went wrong. Oh well, at least you got my point. :)

fano wrote:

Yes, I think configuration is the only thing can be out of the bundle... but in a clear path /boot/home/myApp/myApp.cfg not in a strange /etc path.

Yes, exactly. And if the app has a default configuration file (before the user has made any changes) it can be stored in the app's own directory, instead of storing it in /etc.

fano wrote:

In reality I have to say I've noted to have in my download directory decompressed (what a coincidence!) bsnes 0.70 and it is a QT application... in its direcoty I find 2 dll relating QT:

QtCore4.dll    657 Kb
QtGui4.dll    2574 Kb 

The entire bsnes folder is very huge... 10.2 MB! Wow is this we are speaking for days? For add QT dependency I have to add 3 MB? Well I can survive to this... in the worst case I install 100 QT applications I occupy 300 MB :-)

That's really big, my 500 GB drive can't handle it. :P 1000 Qt apps: 3 GB!

fano wrote:

Yes as Java is but they can self contained, yes... instead of the port tree, if the fake unixy directory is in the bundle itself?

MyApp.bundle
     ----- bin: myApp
     ----- lib: test.so qt_core.so
     ----- usr/share/pixmap: myApp.icon
     ----- etc --> /boot/home/MyApp/config/: myApp.cfg
     ----- var/log --> /boot/system/var/log

In this way the unixy compatibility links are gone... if application need them write it in the bundle!

Although my perfectionist side might still get a bit annoyed, this definitely looks tolerable. Only the app itself will look messy, the system won't be affected. :)

It seems we are pretty much in agreement on this, that's nice. There's one thing I want to add for all the supporters of the Linux-ish way. It seems that most of you think a bunch of third-party libraries is a natural thing for every app. The difference between Linux and Haiku is that Linux is a kernel and Haiku is an OS. In Linux land, everything but the kernel is extra, and that is why the kind of dependency resolution that Linux has makes sense there. Almost every app uses different APIs for everything. Haiku on the other hand is a complete OS, and almost any basic functionality you might want your app to use can be found in the OS itself. If an author decides to use a non-native API for anything, some might get annoyed in one way or another. But the smoothest way to handle it is for the author to include the thid-party libraries with the app.

Re: Package Management, First Draft

sparklewind wrote:

Weird that you got that error, maybe the download went wrong. Oh well, at least you got my point. :)

Yes strange... the archive seems OK as it decompress... the executable is NOT, mystery :-)

sparklewind wrote:

That's really big, my 500 GB drive can't handle it. :P 1000 Qt apps: 3 GB!

Yes, I don't know if it is only this example in which QT dependency is so small, but if it is... what is the matters? You want to put your application on a floppy? We're in 2011... I can understand if it was 180 MB, but 3?

sparklewind wrote:

Although my perfectionist side might still get a bit annoyed, this definitely looks tolerable. Only the app itself will look messy, the system won't be affected. :)

Yes but excluding developers no-one should see that mess, you see only a shiny icon, you should have to know that is in reality a folder with a unixy tree inside... in this way we simplify ports (not so much modification in the source you create the dirs it likes, eventually if it wants write out of the bundle you can use links as for the configuration in which is "fake" a native Haiku app as configuration is in /home/configuration/myApp but application thinks to write in etc...in the bundle itself!)

sparklewind wrote:

It seems we are pretty much in agreement on this, that's nice. There's one thing I want to add for all the supporters of the Linux-ish way. It seems that most of you think a bunch of third-party libraries is a natural thing for every app. The difference between Linux and Haiku is that Linux is a kernel and Haiku is an OS. In Linux land, everything but the kernel is extra, and that is why the kind of dependency resolution that Linux has makes sense there. Almost every app uses different APIs for everything. Haiku on the other hand is a complete OS, and almost any basic functionality you might want your app to use can be found in the OS itself. If an author decides to use a non-native API for anything, some might get annoyed in one way or another. But the smoothest way to handle it is for the author to include the thid-party libraries with the app.

Indeed this an important thing GUI application that not use NATIVE library should be
an exception: the majority should use Haiky kits and depending only on it!

I'm really not liking the whole "hands-off" approach to package file management. So I can't manipulate the files directly because the working tree is immutable, and I need to use the special tools to edit the contents of hpkgs, and possibly I'll have to mess around uninstalling and reinstalling the package too.

And really, the "package can be run just by double-clicking file" thing sounds like an afterthought and just not the same as Mac OS style bundles; I don't have high hopes for this system being implemented elegantly.

How will the system decide to install or run the package when you double-click it? How will the system manage file associations if the mount is temporary? How can you be sure every package maintainer will ensure this usage model works with their packages, especially if the "managed" approach is much more encouraged? What if the package contains not just a single application, but many applications, or also includes other components to provide to the system? Seemingly, if this is your preferred usage mode, you're probably inevitably going to end up screwed.

I was really hoping Haiku would do something simple that was just an extension of the "you handle apps directly" concept. Not this Unix/Linux-inspired idea where I have to go through a package manager for everything. *sigh*

Re: Package Management, First Draft

At first I was quite sceptical about the Haiku Package Manager. But this sure looks promising. Keep up the good work.
This draft makes quite a lot of good points. Removing /boot/apps makes the file tree harder to navigate. Moving etc, var and so on to a place of their own is definetly the thing to do.

I would really love a package to contain information about repository, where updates can be downloaded for this package.

Re: Package Management, First Draft or reason for a Distro

Changing the filesystem as you have proposed is the kind of move that will encourage a distrobution of Haiku with the more be-like file system and either no or a simpler package manager.

Be careful.

This will make Haiku even more Linux-like.

Re: Package Management, First Draft or reason for a Distro

The File organization is important

One must understand that the filesystem organization is as much a part of the user experiance as is the Deskbar, Tracker, and Window tabs. Haiku has enharited an eligant file structure that allows a new user to become an expert in a very short time. This proposed file structure change is counter-intuitive and will make all the maintanence tasks ( fixing broken software etc ) much more dificult. In my opinion we should remove /boot/common and place it back in /boot/home/config to be more BeOS like.

All of the reasons for the file organization that has been proposed can and suould be implimented using file attributes. Haiku has an extreemly powerful filesystem technology that is underused. There is no reason to muck up the file organization just to make a port or program package work properly. Use the file attributes instead of directory trees.

Re: Package Management, First Draft

Personally, I think a compromise between using package management for Unix-y software instead of Haiku software is a good one. If the intention is for it to be used for software like gcc and svn etc, it solves a real problem without impacting usability, although there are a few minor worries (applications ending up with convoluted install methods that use both, or only package management ever being used).

Sadly, reading this thread, I suspect that you can never find a compromise between the two points of view.

One side thinks that duplicate files, wasted HDD space, download sizes, security issues, and memory use is more important. As such, they want an improved version of Linux package managers.

The other side thinks usability, simplicity, consistency, and providing a level playing field for commercial software is more important. As such, they want an improved version of how Mac OS X and Windows essentially work.

I'm of the latter, and think the prior is fairly minor. I presume others see the prior as important, and the latter as a non-issue. I expect the prior will win, given that developers care more about it. My personal opinion is that I have 15 copies of SDL and 10 copies of Qt 4 on this machine from 3 years of use and have never worried about it once. I simply don't think think the prior issues will significantly impact end users, whereas the latter will.

I do have a small suggestion - I'm not quite convinced by '.hpkg', which is a little ugly. Perhaps '.haiku' would be better? Perhaps having a different extension for bundles, packages (that need resolution), and library packages would be useful.

I still think the idea of putting software settings in a package is a good idea, too, simply because the ability for settings to be backed-up/copied/removed in a single file would be amazing. Not sure about its feasibility, though.

Re: Package Management, First Draft

halo wrote:

Personally, I think a compromise between using package management for Unix-y software instead of Haiku software is a good one. If the intention is for it to be used for software like gcc and svn etc, it solves a real problem without impacting usability, although there are a few minor worries (applications ending up with convoluted install methods that use both, or only package management ever being used).

Sadly, reading this thread, I suspect that you can never find a compromise between the two points of view.

One side thinks that duplicate files, wasted HDD space, download sizes, security issues, and memory use is more important. As such, they want an improved version of Linux package managers.

The other side thinks usability, simplicity, consistency, and providing a level playing field for commercial software is more important. As such, they want an improved version of how Mac OS X and Windows essentially work.

I'm of the latter, and think the prior is fairly minor. I presume others see the prior as important, and the latter as a non-issue. I expect the prior will win, given that developers care more about it. My personal opinion is that I have 15 copies of SDL and 10 copies of Qt 4 on this machine from 3 years of use and have never worried about it once. I simply don't think think the prior issues will significantly impact end users, whereas the latter will.

I do have a small suggestion - I'm not quite convinced by '.hpkg', which is a little ugly. Perhaps '.haiku' would be better? Perhaps having a different extension for bundles, packages (that need resolution), and library packages would be useful.

I still think the idea of putting software settings in a package is a good idea, too, simply because the ability for settings to be backed-up/copied/removed in a single file would be amazing. Not sure about its feasibility, though.

I think its way to early to think about ways to install software, there isn't even a good formulary construct on how to install it yet. What I mean to say here is that this process lack quality controls altogether. with out those things, Haiku developers may as well stop now. Beucase no one will be able to guarentee anything works when it is installed.

some tennents to this must be established. things like protecting core System files and folders from being affected by installed applications.

these issues must be prcedurally resolved and unless they are, this whole package manager solution won't realy solve any real and very serious potential problems. It will only give the user a weapon to destroy the operating system.

If some start decrying this philosphy, I point them right to linux, which is a jumbled mess of incompatability becuase no rules and structures exist to prevent such problems. I also say that maybe they aren't going to develope the type of quality applications that new OS entering the software ecosystem really needs.

Its better to have less software selection with good quality and stability vrs lots of selection and gross negligence of operation.

Be smart haiku team, say no to this until there is a procedural way to protect the core of the OS and create some quality standards for what gets let in.

Its hard to undo a bad first impression.

Re: Package Management, First Draft

Actually, last I checked, Linux seems to be working just fine so long as you either 1) use the package manager, or 2) only use packages that you have installed yourself. Even then, on Linux and BSD, you can often intermix packaged and non-packaged software with a degree of success.

I take it you had a very nasty bad impression. So did I with a very specific piece of software, but I'm still using Linux and not letting that one bad experience get to me. It was, if anything, a nice cautionary lesson about being careful when poking around an operating system.

Re: Package Management, First Draft

Haiku is an operating system that is far behind other platforms (Mac OS, Windows, Linux) both in software support and community.

The best way forward for Haiku is to stop and think critically about package management by looking at existing and past approaches and the users of each.

This decision will dictate the uptake of Haiku into the future.

Bundling packages and their dependencies together such as Windows and Mac OS has proven to be lacking at best for power users and sometimes novice users. As it ends in having to resort to manual compilation and installation for those applications that do not have haiku bundles. So for a novice user the bundled system works well if the application bundle exists. And as Haiku is young and behind the game in comparision to other systems its difficult to play catchup building bundled applications. In short it will be an up hill battle.

So looking back we know the popular linux distributions (debian, ubuntu, redhat) also behind the game at their inception solved this problem by resorting to package managers and repositories. Why? Well its because it allowed for rapid management of packages in repositories by a community, allowed for hundreds of applications to conform to packaging standards, created a central location of applications and their dependencies (so you did not have to run around the net finding what you needed), allowed for one package maintainer to converse with other package maintainers (as thats what community is all about), gave the end user flexibility and ensured cleanliness on their system and in the end saved developers time. And saving developers and enthusiests time ensures ongoing growth and uptake by developers and of course novice users.

The package manager / repository paradigm allowed power users to configure software on installation but still be able to keep track of those things installed on their system, making development efforts easier. For novice users i believe this not as important as the only package management they perform is installing new software, and removing old software. More complex tasks such as custom build flags and alike are for power users.

Haiku needs to draw on what Linux has been through. Its only because of repositories and package managers that distributions like Ubuntu can exist. No other model would have worked, as is evident in that their are no Linux application bundle based distributions of any note worthiness.

Its also the case that a novice user does not care if its a bundled application (Mac OS, Windows) or an application to install through another application (such as Aptitude, Steam and Apples new App-Store). Novice users just care that they can read their email, play their music, and type their documents.

So why go for bundling applications and making binary installers if all it does is hinder the productivity of developers and other power users. Haiku should be encouraging the power users, and it should also encourage a clean system design. That is, the KISS principle. A complex system discourages developers and without developers the system is doomed.

Currently I use Arch Linux. This is a distribution that I have always admired. They have strived to make a package management system that is clean. There are two notable tools, pacman and packer. Pacman accesses the standard core repositories, while packer accesses the user repositories (AUR). Their is also a gui tool that makes package management easy for novice users. The greatness of Arch and their pacman and packer tool is how easy it is to install/remove/update/configure and find software. Its equially as easy to make your own package using the makepkg tool and a PKGBUILD file.

I strongly suggest Haiku investigates the Arch Linux approach, and thinks of the power users who will ultimately make this system desirable for the novice users.

If people want to know more about what i am talking about, please install Chakra (a novice version of Arch Linux) and play around.

Re: Package Management, First Draft

Dude -- for a first draft that is an awfully solid-looking plan. It solves the clunkiness issues a lot of people were afraid of, creates clear boundaries between operating system, group and user, provides a transparent method of working on the files within packages as if they're natively part of the filesystem, and allows UNIX and UNIX-like utilities to come along to the party so long as they bring their own beer.

I'm especially impressed with the packagefs speeds when dealing with those storage volumes. It'll be interesting to see what the speeds are like when dealing with tens or hundreds of terabytes of packages. Based on the test it seems to scale well. Are there plans to involve multithreading or background indexing of a packagefs? Are there any ideas on possibly "importing" or "broadcasting" a packagefs index ahead of time for people who want to operate packagefs mirrors or simply reduce the time a network machine would take to index the packages themselves? (Is there even a network version of packagefs?) I might be thinking a little far ahead, but it would be interesting if remote package repositories could be added to an entry in a packagefs filesystem and then be treated as if it's mounted locally, NFS-style but with the packagefs functionality, obviously.

I think having a separate tool to handle system updates is a good thing; it would be easy enough to say something like below:

kipup - package update tool

Action Parameters (Requires One of the Following)
-commit     
     download updates and apply them. If the files have not been downloaded, go download 
     them, otherwise install them now. 
-speculate < optional-outputfile >
     determine what updates are needed and go through the motions of applying them 
     , but don't actually do anything. Optionally, create a report after the operation 
     is complete and save it to a file; otherwise, display on standard output.
-summary-only < optional-outputfile >
     determine what components need to be updated and output the results to a file.   
     Otherwise, the results are printed on the standard output.
-prepare
     determine the needed updates and download all files, but do not apply them. 

Update Type Parameters (Requires One of the Following or a < package-name > )
-all
     update all components of the system that can be updated
-drivers < optional-class | optional-driver-name >
     update only system drivers. If you specify a driver class (i.e. fb, net, stream,   
     display, tty, others TBD), update only the drivers that are of that classification.
     If you specify a driver module name, only update that driver. If the class or name 
     of the driver is invalid, the optional parameter is ignored.
-sysapps < optional-sysapplication-name >
     update only core system applications. If you specify an application name and it is 
     part of the built-in system applications, then only it will be updated, otherwise the
     parameter is ignored.
-sysopt < optional-component-name >
     update optional system add-in components. If you specify a name, only update 
     that component. If the name is invalid, the optional parameter is ignored.
-updatesev < critical | severe | moderate | low | trivial | optional >
     update all components within an that satisfy the update severity class. This
     parameter is unique because it can be used with all other update types except
     when specifying a package name. 
< package-name >
     update only the component name specified. Does not work with any of the other update
     type parameters or with the optional parameter -updatesev

Examples

     Download the updates for the system, but do not apply them.

          kipup -prepare -all 

     Update only the driver for a QLogic 24xx Series Fibre Channel HBA.

          kipup -commit qla-24xx

     Update all fixed-block storage drivers that are critical severity.

          kipup -commit -drivers fb -updatesev critical

     Generate a report on the updates that the system requires for all system applications.
     Only worry about applications of severe or higher severity class.

          kipup -summary-only -sysapps -updatesev severe

     Do a dry-run to determine the results of upgrading all optional system add-ins and 
     save the file as addins-YYYY-MM-DD.output.

          kipup -speculate -sysopt addins-`date +%F`.output

     The same as above, except only those optional add-ins of critical severity or higher.

          kipup -speculate -sysopt addins-`date +%F`.output -updatesev critical

Something like that would allow flexibility in upgrading only particular parts of the operating system or allow someone with limited/pay bandwidth the ability to upgrade only key parts of the OS, though this might be a little overengineered. In some ways due to the fact mainly that hpkg files are going to be bundles, I can see this tool as acting close to how many WoW Add-In Managers behave. They let you know about Add-in updates, go download them, and then just place them in the Add-In's folder -- this is very similar to how you're describing the packagefs tree in function, if not in actual operation. If instead of having one huge package for a system update you allow for many smaller component packages that are essentially "Drop and Replace," then this could work for system updates, too.

Either way, it was fun working on this for about an hour.

I suppose until a lot of these UNIX applications get ported, they'll just have to continue bringing along their libraries; hopefully a lot of the key components of some applications, such as the APR, Neon, and other libraries, will get ported over in time.

Re: Package Management, First Draft

Good, interesting post. :)

Re: Package Management, First Draft

I know this isn't really the place to be suggesting this, but why is the "boot" folder even called "boot"...this should be named something like "system" and only have things that pertain to the OS.

Anything that's not "Average-User-Friendly" should be in the "system" folder. Apps could go in the "Apps" folder, and everything else could go into the Users/username/ folder. Any kind of user specific preferences can go into the Users/username/prefs/ folder and so on...

I think that haiku would benefit with it's "system" folder in a more "OS9" style, with folders like "libs" "Tracker-Addons", "Controls", "Screen-Savers", etc. instead of all that weird BeOS legacy stuff that's going on now. Until BeOS support is dropped, "invisible" symlinks could be used for most backwards compatibility if it was done right.

An "average-user" should NEVER, EVER have to open the "system" folder or use the terminal, etc. for ANY reason. Anything in the back-end should stay in the back, out of sight. This is my biggest issue with Linux. Please, don't make me type some random gibberish to make something work. Ever.

Also, on this subject, "dependencies" should be a word that "average-users" need-not know. "What do you mean I need to install A to use B?" This should all be done in the background. To use a program, just download and run. The OS can do everything it needs to behind-the-scenes automatically.

On the actual subject of package management, I like the OSX style of ".app"'s. Most .apps are actually just folders containing all of their contents. Therefor decompression at run-time is not an issue because they are already decompressed. For haiku, any dependencies could still be checked when the app is first launched and installed if necessary.

Of course Package management should also have a form of update handling too. Not just for the system, but EVERY program should be able to check for the latest version, and download it automatically whenever it is run.

Uninstallation would be a breeze, just through away the ".app". The dependencies wouldn't need to be uninstalled because something else may need them.

Anyway, I just wanted to get that off of my chest... Keep up with the good work! I'm lovin most of it so far! ;)

Re: Package Management, First Draft

dbozan99 wrote:

why is the "boot" folder even called "boot"...this should be named something like "system" and only have things that pertain to the OS.

You've missunderstood the paths. "/boot" is not a folder but the boot volume. There is a system folder, at /boot/system. Haiku generally mounts non-boot volumes by name, like