Package Management for Haiku

Blog post by zooey on Mon, 2011-01-03 18:49
,

"Make everything as simple as possible, but not simpler."

Starting my month of paid Haiku-work today, I was quite astonished to see the many emotional comments that have been added to the announcement. Clearly, the topic seems to be one of heavy likes and dislikes ...

Having read all those comments and most of the discussions we had earlier (on the wiki and on the mailing lists), I felt the need to start my blog with the quote given above - as, for me, that pretty much summarizes what there is to say about package management ...

Package Management - what is it?

Basically, package management is the means to tame and control that flee circus of all the different software components that make up your running system. Without any package management, each computer user would have to maintain those software components yourself, which after some time pretty much unavoidably leads to a more or less inconsistent (if not completely borked) system. In order to circumvent this, most OSes these days incorporate some kind of package management. Most Linux distributions come with pretty elaborate package managers, the BSDs have their ports tree, which is just another (albeit a bit simpler) way of managing packages. OSX pretends to do without package management only to rely on fink (plus others) to provide support for managing packages through the backdoor. With all those modern mobile OSes managing their software through a 'market' application, that pretty much leaves Windows as the lonely OS without package management (unless you consider Windows updates just that). But even on windows, many applications have chosen to implement an update service all by themselves, in effect providing a (rather crude) form of package management.

The parts of a package management system

Here's a list of the components that usually make up a package management system:

PACKAGE BUILD RECIPE (examples: .bep, control, .spec)
One or more files containing info about a specific package that describe how to to create a binary package from the sources. A recipe contains info about the dependencies the software has and which files (programs, libraries, data files) should be put where during installation.

PACKAGE BUILDER (examples: apt-build, haikuporter, rpmbuild)
A program used for building a specific package from its source, carefully following the actions defined in the recipe in order to convert the sources into a binary package.

PACKAGE FILE (examples: .deb, .ebuild, .hpkg, .pkg.tar.xz, .rpm)
A file type and data format that contains all the files and meta information that make up a specific binary software package.

PACKAGE ACTIVATOR (examples: dpkg, rpm, pkg_add, Haiku's package-fs)
A program that can activate/deactivate one or more packages (activation may involve actual installation of files, but not necessarily so). Keeps track of which packages have been activated.

REPOSITORY
A place where to store info about the available packages, usually along with the packages themselves, too.

PACKAGE MANAGER (examples: apt, emerge, pacman, smart, yum, zypper)
The heart of it all - a program which reads one or more given repositories (usually over network) and can be asked to download and activate available packages. It will automatically resolve any dependencies those packages may have on other packages. Additionally, the package manager can update packages and even complete systems.

These days, it is common to only ever communicate with the package management system through the GUI of the package manager - all the other components need not be visible to a user at all (depending on the involvement the user wants to have with system administration and/or development).

So why do some people hate package management that much?

Honestly, I don't really know - I'm using package management nearly on a daily basis (e.g. during my administrative work on the Haiku infrastructure). Without a package management system helping me out, many of my everyday tasks would be much more complicated and error-prone.

I reckon most of the agitation around package management stems from the fact that package management is a complicated matter. One can try hard to hide this fact, but in the end, it will always show through that some hard work has to be done behind the scenes (usually by the people packaging the software). People wish it were simpler and complain loudly, pointing at solutions that most of the time just hide the problems instead of solving them.

Of course, several earlier implementations of package management were really a PITA, I remember having to hunt for RPMs on the net that were dependencies of some other program I just wanted to try out. Doing that once can be frustrating already, but having to do it repeatedly made me want to wipe the dust of my C64 ...

But today, package management usually is unobtrusive and just works. However, I am still seing problems here and there, which I would like to avoid on Haiku, of course.

Requirements for each component

In order to get a baseline for my research work on the existing implementations, I thought it'd be a good idea to start with a list of requirements for all the individual components. I am then going to look at each implementation and determine just how well it fits Haiku's needs.

I am pretty sure I am going to miss a few requirements here, so please comment if you think anything is missing!

PACKAGE BUILD RECIPE

  • preferably contains info about from where to download the sources from
  • specifies which patches to apply
  • contains several basic attributes of the software (name, version, description, homepage, supported target platforms, ...)
  • specifies the build options and build script
  • specifies the dependencies of this software for several stages/scopes (build, install, test, ...)
  • specifies the set of resolvables that the resulting package provides (which may depend on the build options given)
  • supports giving different info for some target platforms (x86-gcc2 vs. ppc)
  • contains info about the licenses applicable to the software

-

PACKAGE BUILDER

  • uses a recipe to create a binary package for a given target platform
  • sets up a chroot()-like environment for each build, containing only the software's build-dependencies
  • enters the chroot()-env for the build in order to reduce the risk of the build process "picking up" installed software, which would create unintended (and untracked!) dependencies.
  • must support signing the resulting package with a given GPG-key
  • nice-to-have: support a mode for automatically invoking the test suite of the software and checking the result with expectations
  • nice-to-have: provides a helper program to convert any buildable software into a package file (by means of a trivial recipe or without any recipe whatsoever) - the resulting "faked" package can then be activated and will take part in dependency resolution

PACKAGE FILE

  • a single file containing all the info required for installing (i.e. activating the software contained in) the package
  • must support signatures with one or more GPG-keys
  • contains hashes for all files contained in the package in order to be able to validate installed files againts the original package contents
  • should be perfectly tailored for the requirements of the package activator, i.e. the activator dictates the format, not the builder

PACKAGE ACTIVATOR

  • show list of installed packages
  • install/update packags
  • remove/purge packages (remove keeps config files, purge deletes those, too)
  • verify packages against status quo
  • activate packages system-wide or per-user
  • nice-to-have: support installation of multiple versions of a package on the same level (system or user) - this probably needs to be supported by the individual packages in order to work
  • nice-to-have: automatically make backups of config files before package update
  • nice-to-have: show list of config files for a package (and allow to backup/restore/purge them)

REPOSITORY DETAILS

  • has name, description and URL
  • a repository must provide concise info about the packages it contains and their dependencies
  • provides a (GPG-)key used for signing all the packages in the repo
  • should preferably be accessible by a web-browser
  • there may be a correlated website, which offers additional services on top of the technical requirements, e.g. screenshots, reviews, user comments, etc.

PACKAGE MANAGER DETAILS

  • maintains (add, list, remove) a set of prioritised repositories (including user-defined repos)
  • show list of files contained in package
  • show the package owning a specific file
  • show all attributes (origin, version, release, resolvables, dependencies) of all packages
  • supports all actions of the package activator, with additional dependency resolving
  • supports activation of local packages (that were created locally or downloaded by some other means) with full dependency resolution
  • always computes all required actions and shows them to user before applying them
  • can search for packages with given attributes (name, resolvables)
  • can verify validity of whole dependency tree and offer actions to solve any problems
  • automatic update interval for repository information can be configured and switched off
  • supports listing and getting rid of dependency-only packages (which were being activated only as dependencies of some other package)
  • supports system updates (nice-to-have: actively, by offering to switch to a new release)
  • nice-to-have: supports limiting actions to a specific repository
  • nice-to-have: supports packet "pinning", i.e. specifying restrictions for specific packages with respect to their origin, release or version

Which implementations to look at?

I have currently planned to look at the following package managers (and their associated repository format[s]) in detail:

  • apt (with dpkg and rpm backends)
  • pacman
  • smart (support for many package formats)
  • yum (supports rpm only, AFAICS)
  • zypper (supports rpm only, AFAICS)

Additionally, I'll inspect at least these package formats and package activators:

  • dpkg/deb
  • hpkg/package-FS
  • rpm

Finally, I will spend some time with the different package build infrastructures, especially the ones used by Debian (apt-build), openSUSE (rpmbuild) and Arch Linux (pacman).

The special focus during these investigations will be on how easily each of those tools could be adjusted to work on/for Haiku - after all, many of these tools have been designed for a specific Linux distribution, which Haiku simply isn't (and doesn't want to be). On the other hand, I really hope that at least one of the package managers proves to cater for most of our needs, as writing one from scratch would be a *lot* of work.

Please do not hesitate to send your ideas my way ...

Comments

Re: Package Management for Haiku

by umccullough - 2011-01-03 21:40

Just wanted to add, for those who are fervent about one type of package over another, that I don't see any reason why a given package couldn't technically include all its dependencies with it (ala OS X bundles)... personally, I detest that solution as it provides a real PITA for upgrading shared libs (you must download a new bundle for every app that uses them - or risk stability/security issues for those that you don't), but it could be done that way.

As Oliver so clearly demonstrates, there's a lot more to package management than just tracking dependent libs (which is what people immediately jump to). Package Management also doesn't require a central repository that all software must come from... although some aspects of it are clearly designed to support that strategy.

Since Package Management for Haiku is being looked as a lot of different aspects that can and will help Haiku users (and developers!) to improve their experience - I don't understand why so many people get all up in arms about it. Ultimately, what is being done at HaikuPorts, and Haikuware alike is already package management in some part - and the sooner we can start to pull these bits together into a smooth working system, the sooner Haiku will become more usable for average people.

Thanks for taking the time to lay all this out Oliver!

Re: Package Management for Haiku

by tonestone57 - 2011-01-03 22:50

I am for package management but will tell you why others are not.

"So why do some people hate package management that much? Honestly, I don't really know"

  1. Linux complexity and bloat - massive program to handle somewhat simple job & many different versions of same libraries
  2. Broken packages (not working):
      3. a) Dependency hell - cross dependency issues (missing libraries) or older library dependency (breaks program with newer library version)
      b) Application incompatibility with OS. Application may not work on different versions of Haiku. (missing symbol error)
  3. Multiple repos (confusing & adds to complexity - understanding what each one is for & adding them)
  4. Websites, like Haikuware, and installoptionalpackage script work fine for installing progams and time could better be spent elsewhere. On higher priority areas to get Haiku to R1 quicker
  5. Fear that Haiku will lose uniqeness and become more like Linux

I don't require a response. I only wanted to point out what the major issues were and why some people are opposed to package management on Haiku.

I am somewhat opposed to importing a Linux package manager into Haiku. I prefer from scratch to avoid bloat, complexity and make sure it works the Haiku way but will keep on open-mind towards your findings.

Re: Package Management for Haiku

by umccullough - 2011-01-03 23:21

Edit: This was meant to be a reply to tonestone's post, but I guess I didn't use the "Quote" button.

The funny thing about the list you mention is: Almost all of those items will be a worse problem *without* package management..

1) Package management doesn't create this problem, software development/porting practices do. This problem already exists in Haiku due to the abundance of ported software, but the user must deal with it manually.

2) I hear people complain about broken downloads not working on Haiku all the time - missing dependencies, ported apps compiled with hacks that don't work on stock Haiku (such as symlinking libroot->libm, etc.) How can package management make this worse than it already is?

3) Is there really a single place to obtain all Haiku software now? Is having a single place to obtain all software even a good thing? Update: besides, package management doesn't imply everything must come from a repo. I download .deb files all the time and install them on my Linux box.

4) Haikuware may be a good place to obtain packaged software, but it certainly isn't a solution to package management. installoptionalpackage.sh is a temporary hack that will go away when a real solution is in place. As an example: there's no way to remove a package once installed, there's no way to use installoptionalpackage to install software other than was is defined in Haiku's official "OptionalPackages"... There's no clean way to update to a newer version of a package and know that it was done properly. installoptionalpackage can't even do some of the more complex tasks that Haiku's OptionalPackages requires. Besides, these are, in a way, basic parts of Package Management, that are severely lacking in features/functionality. Once you get used to the idea that they are, you realize how important that aspect is to everyday use.

5) Package Management is not going to do this - porting every FOSS app under the sun to Haiku is more likely to cause this problem. To keep Haiku unique, more native applications that use Haiku's included functionality are required. Package Management is just the aspect of installing/uninstalling/maintaining software on a given machine - it has nothing to do with Linux specifically. Even Microsoft provides package management tools like "Add/Remove Programs" and MSI with Windows... does that make it more like Linux?

Anyway, I think there's some kind of stigma around the term "Package Management" which is unfortunate, since I'm certain everyone out there uses some form of it regularly on their OS of choice.

Re: Package Management for Haiku

by tonestone57 - 2011-01-04 18:40

@ Urias
Please note, those were the general (negative) issues I noticed from "Haiku Inc. accepts contract for package management" thread that now has 52 comments to it. I myself am for some type of package management in R1 but wanted to just point these out to show why people were against the idea itself. Those were what I was seeing from the other posters pushing back against package management on Haiku.

I agree that those same problems exist without package management not counting the Linux related ones. Many *think* that package manager causes these problems but they really happen at developer level. People using outdated programs requiring older libraries.

A well written package manager & good package testing should address all of those concerns. Of course, nothing is perfect and there will be issues (hopefully less) but similar ones if you go download the applications and libraries from a website.

Some people believe that Haikuware + installoptionalpackage are good enough for R1 and package manager should be pushed to R2. That way R1 gets out sooner. Question of when to release one. I think good to have it in R1 but create something simple and not too complex to start with.

Anyways, I'm glad Oliver is working on this and happy to see his findings and analysis when finished with his research.

@ others
Some also suggest using bundles but they don't understand that only resolves dependency issues but you miss out on security and bug fixes to libraries by using the older versions, use more memory and disk space. I wouldn't call that efficient which is very important to me. Dependency issues and system updating are the hardest issues to deal with in package management. Updating a Haiku system with memory loaded files isn't easy like an application which you quit and no longer run in memory.

Re: Package Management for Haiku

by X512 - 2011-01-04 19:18
tonestone57 wrote:

Some also suggest using bundles but they don't understand that only resolves dependency issues but you miss out on security and bug fixes to libraries by using the older versions, use more memory and disk space.

I think that it is problem of software developer. Developer can use new libraries in next releases if it needed and new libraries had been tested with software. New library can broke application or make it unstable.

Re: Package Management for Haiku

by cjst - 2011-01-03 23:47

Haiku absolutely does not need package management. At least not like it exists on Linux, BSD and so forth. To install an application on Haiku one should just have to download the right archive and unarchive it in their app folder. What Haiku needs in fact is a way for the OS to update itself automatically from the internet, and perhaps a way to ensure the integrity of the OS files. So something akin to a package manager could be used to solve both of the aforementioned problems. But it must not try to solve problems that Haiku does not and should not have. Such as for example dependencies. Haiku programs should use the standard libraries that come with haiku. When a third party library is needed, then it's bundled in the zip file with the program. To delete a program, delete it's folder. To upgrade it just delete the folder and unarchive the new version. As simple as that.

So instead of considering things like RPM, DPKG and so on you might want to look into how chrome, firefox, windows and many other program update themselves. Haiku would need something more generic than these. That could perhaps be reusable by third party applications. But not something as overly complex so as to satisfy alls the whims of developers like on Linux/BSD.

About ports: they really should adapt themselves to Haiku and not the other way around.

Re: Package Management for Haiku

by jest - 2011-01-06 10:10
cjst wrote:

Haiku needs in fact is a way for the OS to update itself automatically from the internet, and perhaps a way to ensure the integrity of the OS files........

Haiku programs should use the standard libraries that come with haiku. When a third party library is needed, then it's bundled in the zip file with the program. To delete a program, delete it's folder. To upgrade it just delete the folder and unarchive the new version. As simple as that.

About ports: they really should adapt themselves to Haiku and not the other way around.

+1 - The big issue for me is separation. Separation from programs I have installed vs the Haiku.

For me no 3rd party program should be writing files into OS folders or causing an update of any file in them.

Re: Package Management for Haiku

by tsteve - 2011-01-20 23:05

This is exactly right, BeOS did it the right and easy way, simply open the zip file, and click on the program. Windows 95 wasn't that bad to begin with. But then you started having problem when MS started watching what DLL files were loaded, and counting them etc, then you got the stupid uninstall programs that had to ask, do you want to delete this file ? another application may be using it ?,,, but it still worked, and wasn't too much of a headache. but then Install Shield had to try and build the better mouse trap, and more and more, people had problems trying to uninstall something... At that time it was mostly about hard drive space, and double files,,, Then finally came the day, when MS decided to make something better. Something that could install and uninstall, but also update, and in theory even go backwards. MSI, sinse that time it has been pur hell, if you install something and later update it, it will never uninstall correctly and completely again. Then when IE or Office was installed, but did not work, you were screwed, you could not uninstall it, you could not simply delete it, and you could not install over top of it. You basically had to format and reinstall Windows... Now the registry was half that problem, with it telling explorer that certain files were in use, and could not be deleted etc. But my point here, like every time I write anything here, is that simplicity has to rule. It cant be any other way. Evey day Haiku gets a little closer to being just another Linux in the world, or just as messed up as Windows
With BeOS, you simply download a zip file, unzipped it, and used it, or used the BeOS package manager, that was young, but not too bad. The reason it wasn't too bad, is it didn't do much. Everything I'm reading here today, is just little steps to Haiku being just another linux clone, and for the 100th time. Linux is free, and still cant get 10% use in the wild. Haiku will be exactly the same, if it keeps following in Linux's footsteps

Re: Package Management for Haiku

by Zeno Arrow - 2011-01-04 00:10

zooey, thank you for taking the time to post.

I am in favour of a package manager, but I can see the flaws with the current ones too, so hope I can help.

Firstly, people like to have as much control and knowledge over their file system as possible. What happens when you install an app using a package manager? It downloads a bunch of files then sprays them across your file system, into locations you did not choose, and leaves you virtually no choice when it comes to uninstalling apps apart from to use it.

I'm in favour of package managers for making day-to-day installation operation easy, but I also like to have as much control over how I handle my own system as possible. It's honestly a breath of fresh air when you can download an app that works without installation (PuTTY on Windows for example), means I can put the app wherever I want. The .dmg approach that OSX uses acknowledges this feels satisfying, but doesn't solve the issue of bloat from non-shared libraries.

I posted the outline of how to fix this on the previous blog topic, I'm happy to expand on how it would work, what it boils down to is leveraging the potential of OpenBFS to improve the package management experience. If you have doubts it'll work, please feel free to ask how I'd anticipated fixing the issues you percieve. Thanks.

Re: Package Management for Haiku

by thatguy - 2011-01-04 02:13
Zeno Arrow wrote:

zooey, thank you for taking the time to post.

I am in favour of a package manager, but I can see the flaws with the current ones too, so hope I can help.

Firstly, people like to have as much control and knowledge over their file system as possible. What happens when you install an app using a package manager? It downloads a bunch of files then sprays them across your file system, into locations you did not choose, and leaves you virtually no choice when it comes to uninstalling apps apart from to use it.

I'm in favour of package managers for making day-to-day installation operation easy, but I also like to have as much control over how I handle my own system as possible. It's honestly a breath of fresh air when you can download an app that works without installation (PuTTY on Windows for example), means I can put the app wherever I want. The .dmg approach that OSX uses acknowledges this feels satisfying, but doesn't solve the issue of bloat from non-shared libraries.

I posted the outline of how to fix this on the previous blog topic, I'm happy to expand on how it would work, what it boils down to is leveraging the potential of OpenBFS to improve the package management experience. If you have doubts it'll work, please feel free to ask how I'd anticipated fixing the issues you percieve. Thanks.

seriously, what bloat ? where is this bloat comming from ? why is this bloat here ? who says its bloat ? I call it core stability. If I have to give up 5 mb to avoid having some hack developer overwrite or endager critcal system librarys like opengl etc, so be it.

the whol problem is increably avoidable. Just put out a standard for how to package and install applications, ports, IDe, applications layers whatever.

Installing should really be as easy as unzipping or deleting.

This whole debating is infuriating and honestly a good slap in the face of lots of haiku supporters.

PACKAGE MANAGEMENT ISN'T NEEDED

a sane way to package applications is.

people need to stop with the bloat talk already, haiku is like 500mb at best installed and with all the software in the current eco system it wieghs in under 20gb with the source and build tools. which are like 50% of that.

So its going to be a long time before its ever any kind of issue.

Don't go fixing something that isn't broke.

Now this is bloatware and why Haiku will generally almost always be a user installed OS.

http://www.youtube.com/watch?v=3hDdw_PA7xw

Re: Package Management for Haiku

by kvdman - 2011-01-04 05:52

This seems a bit backwards. Without an IDE, an easy way to develop apps, and strict guidelines for developing apps on Haiku, a package manager, seems useless. So, now you have a package manager, but what happens when developers compile their apps against an ever changing Haiku revision, submit it to the repository for the package manager only for users to find the app crashes because it can't find the needed libraries? I've seen this again and again even from Haiku developers!

It should really just be as simple as it is on OS X. Download, drag to applications and it works. Why do I have to download or use a package manager to make my software work?

As far as I'm concerned:

"Make everything as simple as possible, but not simpler."

A package manager makes things more confusing for users; it's not simple. Simple is downloading an app, and it works; no matter where it's from. Haiku should find some way to accomodate this.

I regret my initial statement that this project is a waste of money. I find (as wel as 'final poll voters') that this isn't a priority. There are work-arounds, and I think package management could best be used for OS updates and security enhancements, not software management. This could all be solved by including standard current and legacy libraries required by apps, and a good set of standard applications.

Re: Package Management for Haiku

by darkwyrm - 2011-01-04 00:39

This is a big task that you're undertaking. The best of luck to you!

Re: Package Management for Haiku

by hobsii - 2011-01-04 01:01

I'm no longer a BeOS or Haiku user, but I can't resist to comment on this. Why can't we just download, unzip and drag an app to the Applications folder? I like the way it works on MacOS.

Re: Package Management for Haiku

by Knutsi - 2011-01-06 15:41

I completely agree. I grew up on Macs, and the mechanism for installing applications (no need to install) and system extensions, fonts etc. (drag to system folder) was so simple it made people unafraid to do such things. I remember thinking when using Windows 95+ that "why do I have to run this installer thing?", and noticing that my mother would resist installing software because of the installer.

Point: everyone knew Finder from day to day work with files, and when you knew Finder, you knew all you needed to install extensions and applications. THAT's simplicity. One pattern to cover it all, and you were forced to learn it anyway.

But, times change, and people will now expect some kind of app store (Win 8 will have one, and people know THIS pattern now from phones). So, I think this is the way to go. An app store. With all the package management you need integrated, and an option to install from a file so we don't have to have "freedom from porn" (http://gawker.com/5539717/steve-jobs-offers-world-freedom-from-porn) enforced.

Re: Package Management for Haiku

by Aud - 2011-01-04 06:36

I don't know what this 'package-fs' thing is exactly, but it gives me the idea of a filesystem layer (a la FUSE) that provides semi-automatic maintenance services to applications (and etcetera) when they are placed in a specific folder/mountpoint.

I'm very much in agreement of liking the feel of Mac OS X much more than anything else. I've used Gentoo/emerge quite extensively in the past, and I love the power it gives me (version selection, USE flags, large repository, lots of pre-stable packages, eselect), but hate having to go through a special interface and not be able to handle the files directly myself (lest the package manager thinks I've broken my system, oh noes!). OS X is lacking in the area of package repositories, and in relying on apps/components updating themselves.

So summed up, the kind of package manager I would totally love is one where I can just drop software into place, allows me to keeps apps and components neatly bundled up, allows me to directly handle files myself, yet provides helpful update notifications, and gives me a simple-looking tool to apply the updates (which should allow me to choose what I want installed, not necessarily the "latest and greatest" or the most stable versions). Having package repositories would be a bonus, but not as important, and I think is much more useful for developers than standard end-users.

Re: Package Management for Haiku

by fest3er - 2011-01-04 07:27

Zooey

You seem to have omitted 'list of dependencies' from your Package File section.

Package management is absolutely required, even though it can be an absolute pain in the stern. May I suggest Package Management System as an overall description? (Surely PMS describes the situation well enough.)

You are right, for all the reasons you mentioned. And more. Suppose I am writing Frombulation!, open source software for designing widget interfaces. What I hack together is all well and good. But I need a formalized way of keeping track of dependencies, how to build it, how to assemble its install archive, how to uninstall it, and how to prepare the documentation. More important, the gal who someday takes over maintaining it is going to need to know everything I know about preparing the software's dependencies, building the software, assembling the software into a package, installing the software, preparing the documentation, uninstalling the software, et alia.

Young folks don't have a problem remembering all this minutiae. (Sigh. Listen to me; not too long ago I was one of them young whippersnappers. :)) But you get to be 40 and 50, and older, you just aren't going to remember all those details in a couple months. A good package management system will provide structure and order to the tasks of software development, distribution and maintenance. And it helps developers do something else I've been known to carp and kvetch about. Remember the three most important things everyone, including software authors, needs to do every day of their chosen careers, avocations and hobbies: 1) Communicate. 2) Communicate. 3) Communicate. Communicate to yourself what you know now, because you'll probably forget in a few months. Communicate to future maintainers what they'll need to know to take over your job. Communicate the requirements of your software to the people who use it so they don't have to chase all over learning from scratch what you already know. A PMS provides this communication. A good PMS does it unobtrusively, smoothly, and cleanly.

My bias against RPM goes back 15 years or more; I never really liked it, and still don't. Perhaps it has matured in the interim. I can't recall the last time I had to chase a dependency with Debian Stable, or even Testing. Debian's PMS handles many architectures very well. And like all of Man's creations, it isn't perfect; but it works well.

Good luck with your endeavour! And don't forget to keep us posted!

Re: Package Management for Haiku

by zooey - 2011-01-04 09:18
fest3er wrote:

You seem to have omitted 'list of dependencies' from your Package File section.

Ah, you are right. But it's such a central feature of package files that I'm sure I won't forget when checking out different package file implementations :-)

Re: Package Management for Haiku

by Zenja - 2011-01-04 07:33

The shared library fiasco on Linux is due to Linux being nothing more than a kernel - everything else is optional, from file system to desktop environment. This creates a dependency management problem, which package managers attempt to resolve.

Haiku, however, ships as a complete system, with a well defined set of bundled libraries. Anything not bundled with Haiku core, is strictly not a Haiku component. If an application needs it, then the component should ship with the application itself, bundled in the applications lib directory. It doesn't matter if we waste disk/memory resources to have 2/3 identical copies of said library. The other additional benefit of not sharing libraries is that we DON\T accidentally break binary compatibility of a particular app by updating a shared library. If the user is concerned about a vulnerability with a particular library, they should pressure the application developer to update their app with a newer version.

What Haiku does need is a method to update Haiku core components, and this is what the community is requesting (IMHO). 3rd party applications/bundles should not be handled by Haiku core package manager. In this regard, I think that Microsoft and Apple are doing the right thing with System/Windows Update. IMHO, the Linux method is unnecessary.

Another thing that I'd love to see is something similar to Apples AppStore. The AppStore is a centralised repository for downloading VALIDATED applications, with a community section for reviews. The benefits of using an application store is that you can get notifications of application updates. But the biggest difference between the AppStore and a Package Manager is that the AppStore handles application bundles as independent entities - there is no sharing of libraries / components.

At the end of the day, KISS. Keep it simple.

Re: Package Management for Haiku

by Zenja - 2011-01-04 07:38

One more think. Oliver, I've noticed that you've talked about source code repositories while discussing package management. I think the community will be hostile to such a concept. Haiku cares about binary compatibility, and most end users do not want to recompile each application - if we wanted to do that, we'd be running Gentoo or LFS.

Re: Package Management for Haiku

by zooey - 2011-01-04 09:40
Zenja wrote:

One more think. Oliver, I've noticed that you've talked about source code repositories while discussing package management. I think the community will be hostile to such a concept. Haiku cares about binary compatibility, and most end users do not want to recompile each application - if we wanted to do that, we'd be running Gentoo or LFS.

I wasn't talking about source code repositories but about the infrastructure required by package maintainers to actually produce packages from upstream sources and Haiku-specific patches (basically what's going on at haikuports today).

Don't worry, I am not at all interested in forcing a source-based package management system (like portage/emerge) upon Haiku users ;-)

Re: Package Management for Haiku

by zooey - 2011-01-04 09:34
Zenja wrote:

The shared library fiasco on Linux is due to Linux being nothing more than a kernel - everything else is optional, from file system to desktop environment. This creates a dependency management problem, which package managers attempt to resolve.

Haiku, however, ships as a complete system, with a well defined set of bundled libraries. Anything not bundled with Haiku core, is strictly not a Haiku component. If an application needs it, then the component should ship with the application itself, bundled in the applications lib directory.

Well, I wish it were this easy: Are you saying that every Perl/Python/Ruby application is going to have to provide the compiler, runtime environment and all the required modules with it? Ok, probably not, since it could be argued that since Haiku provides Perl & Python, the compiler, runtime environment and a set of modules are part of core Haiku. But if so, what happens if the app requires a specific version of any of these? In order to get rid of the dependency problem, you'd have to bundle all of them together for even the tiniest of scripts.

Trying to summarize: an application bundle is just one concept, which happens to work neatly for native Haiku applications that do not have many dependencies. On the other hand, app bundles do not work neatly for everything else, really. Maybe there's an approach that successfully combines the advantages of bundles with decent dependency resolution and easy security updates (and I'm already aware of some projects that at least try to get that done) - I'm going to have a look at those, too.

Re: Package Management for Haiku

by cjst - 2011-01-04 10:01
zooey wrote:

Well, I wish it were this easy: Are you saying that every Perl/Python/Ruby application is going to have to provide the compiler, runtime environment and all the required modules with it? Ok, probably not, since it could be argued that since Haiku provides Perl & Python, the compiler, runtime environment and a set of modules are part of core Haiku. But if so, what happens if the app requires a specific version of any of these? In order to get rid of the dependency problem, you'd have to bundle all of them together for even the tiniest of scripts.

Perl, Python and Ruby and so on are irrelevant on a Desktop OS. These are scripting languages for RAD on servers. Nothing more. So it doesn't matter. They are included with Haiku mostlikely because they are required to compile something because Linux people have the extremely annoying habit of implementing stuff in any language they happen to like at the moment. Just look at Windows. On Windows nobody cares about Perl, Python or Ruby. If some guy wants to play with that it probably means he's smart enough to install them before trying to run any scripts. Please, let's not lay the groundwork for Haiku becoming yet another exotic UNIX-variant :\

Re: Package Management for Haiku

by Zeno Arrow - 2011-01-04 11:10
zooey wrote:

Maybe there's an approach that successfully combines the advantages of bundles with decent dependency resolution and easy security updates (and I'm already aware of some projects that at least try to get that done) - I'm going to have a look at those, too.

Yes, that solution exists, and if you'd taken the time to look at what I'd written before you'd have a better idea of how.

The package format just needs to be a compressed folder with some extra meta information on how the program is run/installed. I'll refer to the package format as HPF for now (Haiku Package Format). Compression format to use? Well KGB Archiver is open-source and offers efficient compression, see the test results for yourself (the downside being relatively high resource requirements):
http://sourceforge.net/projects/kgbarchiver/
http://kgbarchiver.net/?page=tests

7zip maybe a better option as it offers two modes of operation depending on how much RAM is available:
http://docs.bugaco.com/7zip/7zC.txt
"1) Default mode. In that mode 7z Decoder will read full compressed
block to RAM before decompressing.

2) Mode with defined _LZMA_IN_CB. In that mode 7z Decoder can read
compressed block by parts. And you can specify desired buffer size.
So memory requirements can be reduced. But decompressing speed will
be 5-10% lower and code size is slightly larger."

Anyhow, leaving aside compression formats, what about the meta data. Well it depends on what the program requires, but let's take a common form, which will be executable with bundled libraries. The executable has meta data embedded as file system attributes, with the path to each needed library defined as a key field-value pair. The key field is the library attribute name, and the value is the path to this library on the system.

It's probably worth showing how this works with an example. Let's say we bundle a LISP interpreter (version 1.01) with our app. The lisp interpreter has the name attribute set as 'lispint1.01'. The executable has 'lispint1.01' listed as a requirement, along with a default location for it (relative location probably best for default value). Now if you try to run the HPF package without uncompressing it, as long as the Haiku system has been made aware of how to deal with HPF without requiring a special app, it can read the meta data and can work like a OSX DMG file, i.e. run without installing, because it knows where to find the needed libraries.

However, let's say you want to install the app. Easy. You've got two choices, either you can install using the package manager, which will put the files within the HPF within sane system locations (as well as updating the meta data so the app knows where to find the files), or you can extract the HPF archive, copy the files to where you want them, and update the meta data yourself (if required, which won't always be needed if the HPF file used relative addresses for library paths).

I'm just scratching the surface of what's possible with this approach, but I hope the penny is beginning to drop.

Ultimately, like you said before, you want to make package management as simple as possible, but no simpler. Package management at its core is file management, just a specialised form of file management for applications. If you want to fix a file management problem, the logical place to do so is in the file system, not to mention it can make things easier to understand. You're sitting on a goldmine with OpenBFS, emulating Linux package management would be a wasted opportunity.

If you haven't already, have a look at GoboLinux:
http://www.gobolinux.org/
A video about how to improve package management by improving the Unix file system is linked to on the GoboLinux blog:
http://blog.gobolinux.org/
http://2009.r2.co.nz/20100118/50181.htm

As before, comments/criticism welcome.

Re: Package Management for Haiku

by zooey - 2011-01-04 14:07
Zeno Arrow wrote:
zooey wrote:

Maybe there's an approach that successfully combines the advantages of bundles with decent dependency resolution and easy security updates (and I'm already aware of some projects that at least try to get that done) - I'm going to have a look at those, too.

Yes, that solution exists, and if you'd taken the time to look at what I'd written before you'd have a better idea of how.

What I meant to say is: maybe people suggesting approaches on how to combine the advantages of bundles with decent dependency resolution, security updates and effective sharing of executable images are right and one of these approaches actually works. I'm aware of gobolinux and of what you've written in the other blog entry. However, someone has to prove that one of these ideas actually works in real life without having negative side-effects.

[ ... ]

Zeno Arrow wrote:

The package format just needs to be a compressed folder with some extra meta information on how the program is run/installed. I'll refer to the package format as HPF for now (Haiku Package Format). Compression format to use? Well KGB Archiver is open-source and offers efficient compression, see the test results for yourself (the downside being relatively high resource requirements):

Haiku already has a well-specified package format, please look here.

[ ... ]

Zeno Arrow wrote:

It's probably worth showing how this works with an example. Let's say we bundle a LISP interpreter (version 1.01) with our app. The lisp interpreter has the name attribute set as 'lispint1.01'. The executable has 'lispint1.01' listed as a requirement, along with a default location for it (relative location probably best for default value). Now if you try to run the HPF package without uncompressing it, as long as the Haiku system has been made aware of how to deal with HPF without requiring a special app, it can read the meta data and can work like a OSX DMG file, i.e. run without installing, because it knows where to find the needed libraries.

Letting an executable specify it's dependencies via file system attributes is an interesting idea. However, if the app specifies the explicit version ('lispint1.01'), the system couldn't run the program if lispint1.02 would happen to be installed. The exact ways on how dependencies are being specified by an executable and how these can be resolved (most probably by the runtime loader) deserves some careful thinking and experimenting.

[ ... ]

Zeno Arrow wrote:

Ultimately, like you said before, you want to make package management as simple as possible, but no simpler. Package management at its core is file management, just a specialised form of file management for applications. If you want to fix a file management problem, the logical place to do so is in the file system, not to mention it can make things easier to understand. You're sitting on a goldmine with OpenBFS, emulating Linux package management would be a wasted opportunity.

Hm, source control is just another form of file management - still, no current vcs works at the file system level and the ones that did are considered obsolete today. Don't get me wrong, I'm not saying you aren't right, but simply stating (and repeating) a claim isn't going to prove it right or wrong.

The example you've given about installing an app bundle is a rather trivial one. Bringing an application package into life is just one facet of package management, another one would be to automatically determine the set of missing dependencies and download them on-demand when a user starts a bundled application. My task is about finding a solution for the whole package management problem, not just application bundles. Linux package management systems happen to be the most elaborate out there - I'd be plain stupid not to look at them and try to reuse as much of their components as possible (as long as they makes sense on Haiku).

Re: Package Management for Haiku

by Zeno Arrow - 2011-01-04 16:19

@zooey
Thanks for your response.

"Haiku already has a well-specified package format, please look here"

Thank you for the info, using HPKG should be fine, like you say it's a well-specified format.

"What I meant to say is: maybe people suggesting approaches on how to combine the advantages of bundles with decent dependency resolution, security updates and effective sharing of executable images are right and one of these approaches actually works. I'm aware of gobolinux and of what you've written in the other blog entry. However, someone has to prove that one of these ideas actually works in real life without having negative side-effects."

To me the approach to combine advantages of package bundles, etc... is to make formal installation optional. Imagine being able to run HPKG without installation, just by double clicking on it. How is this possible? I'll get to that in a moment.

As for dependency resolution, the key is being able to add and subtract what files are included in a package based based on what you already have installed, with attribute meta data pointing to where to find the files on the system. In other words, in a package either you have a file or a link to where to find it. How is this possible? Again, will explain in a moment.

As for proving the GoboLinux approach to a more structured file system works without side-effects, why not install it in a VM and see for yourself. Quoting from the website:
http://gobolinux.org/?page=at_a_glance
"There is no rocket science to this: /bin is a link to /System/Links/Executables. And as a matter of fact, so is /usr/bin. And /usr/sbin... all "binaries" directories map to the same place. Amusingly, this makes us even more compatible than some more standard-looking distributions."

To make it clear, there are two seperate things I'm trying to fix regarding package management. One is having a saner file system layout (being able to find all files in logical places), which relies on having formal porting/coding guidelines. The second is being able to control and customise your experience with executables by giving users the choice not to use one central tool. I only mention both because it's the right time to fix them both.

"Letting an executable specify it's dependencies via file system attributes is an interesting idea. However, if the app specifies the explicit version ('lispint1.01'), the system couldn't run the program if lispint1.02 would happen to be installed. The exact ways on how dependencies are being specified by an executable and how these can be resolved (most probably by the runtime loader) deserves some careful thinking and experimenting."

The app can specify which version of a library/runtime it needs. If it needs 'lispint1.01' then the app must have it, regardless of whether 'lispint1.02' is installed as well. Going beyond this, if a version of an application is shown to work equally well with both 'lispint1.01' and 'lispint1.02', both can be listed as attributes, and whichever is installed can be used.

"Hm, source control is just another form of file management - still, no current vcs works at the file system level and the ones that did are considered obsolete today. Don't get me wrong, I'm not saying you aren't right, but simply stating (and repeating) a claim isn't going to prove it right or wrong.

The example you've given about installing an app bundle is a rather trivial one. Bringing an application package into life is just one facet of package management, another one would be to automatically determine the set of missing dependencies and download them on-demand when a user starts a bundled application. My task is about finding a solution for the whole package management problem, not just application bundles. Linux package management systems happen to be the most elaborate out there - I'd be plain stupid not to look at them and try to reuse as much of their components as possible (as long as they makes sense on Haiku)."

Okay, now it's time to explain how I foresaw fixing the dependency issues. Two things:

1. As I hinted at before, packages would be broken up into parts, the system is intelligent enough to know whether you need a file or already have it, and if you have it the package simply points to it.

2. This is the absolute key, and I hope when I explain this all the other parts will make sense. The key is handling package management using a HPKG datatype.

Now from what I understand, Haiku can use datatypes (a concept inherited from AmigaOS), if I'm wrong on that correct me. For those who aren't familiar with datatypes, they are a type of shared functionality where a program written to make use of datatypes can extend it's functionality through new datatypes. For example, imagine you write a desktop publishing app that can use datatypes, at first you just use your own format for saving work, but eventually someone writes a .pdf datatype, now your app can save to .pdf, and you didn't need to change a thing.

Why is a HPKG datatype beneficial? Simply put, it means package management adapts to the way you want to work. Imagine Web+ can use the HPKG datatype. Now when you download a HPKG through Web+ it knows which files you have installed and which files it needs to download, and creates a custom version of the HPKG file, with all files and links needed to run an app.

Now we want to install a HPKG. Let's say we want to do that through the file manager, which is possible if the file manager can use the same HPKG datatype that Web+ can. We right-click on a HPKG, select install, files are installed to default locations (defining these default locations is part of the reason I brought up the GoboLinux approach). The app knows how to find all these files through the file system attributes, which are transferred from the HPKG file to the executable (extra attributes for libraries written so know which executables rely on them).

Now we've had enough of an application, and want to uninstall it. We could do this a number of ways. How about we drag the executable file to the Trash/Recycle Bin, which can also make use of the HPKG datatype. It knows what you are trying to do and asks if you'd like to remove the unique dependencies also. You say yes, the files are gone.

How about updating an app? You could just right click on your Application folder and select update all. However, we can do better than that, let's say you only want to update a single app, go to the app folder, right click and look for updates for it alone.

How about update notifications? No problem, you can have an optional update daemon running, which gives you information about updates, and guess what, it can use the HPKG datatype to install those updates too.

How about package bundles? Well, it's just a different way of handling the HPKG data, all of which can be achieved through the HPKG datatype.

I personally believe this should be the 'Haiku' way of offering package management, it leverages Haiku's strengths and fits in with experiences offered elsewhere in the OS. For example, I think it's pretty cool that it's possible to read e-mail without needing to open an e-mail app, simply because the file browser understands the e-mail format (through datatypes too I believe). Integrating package management into file management would be a great feature.

For those who like having a central package management app, they can have it too, it's just a central app that makes use of all of the features contained within the HPKG datatype. It's really the perfect solution to the package management issue, in my opinion.

I appreciate this might seem like a lot of work, but it doesn't have to be, some of these features can be implemented later, but getting the design of the package management framework right at this time means the more ambitious features are easier to implement later on.

If something I've said doesn't add up, you need to tell me where you think this approach will be less than optimal, and I can address those concerns, so please think about what I'm trying to lay out and see how it fits in with your own views. Thank you.

Re: Package Management for Haiku

by hobsii - 2011-01-04 11:29
Zenja wrote:

Haiku, however, ships as a complete system, with a well defined set of bundled libraries. Anything not bundled with Haiku core, is strictly not a Haiku component. If an application needs it, then the component should ship with the application itself, bundled in the applications lib directory.

Why isn't this a good solution? It's not a retoric question, I just don't know.

Re: Package Management for Haiku

by Zeno Arrow - 2011-01-04 11:42
joaomcarvalho wrote:
Zenja wrote:

Haiku, however, ships as a complete system, with a well defined set of bundled libraries. Anything not bundled with Haiku core, is strictly not a Haiku component. If an application needs it, then the component should ship with the application itself, bundled in the applications lib directory.

Why isn't this a good solution? It's not a retoric question, I just don't know.

1. Because the vast majority of open-source apps were not written with the Haiku APIs in mind.
2. Because it doesn't promote efficiency with regards to keeping your apps up to date.

Re: Package Management for Haiku

by zooey - 2011-01-04 14:20
hobsii wrote:
Zenja wrote:

Haiku, however, ships as a complete system, with a well defined set of bundled libraries. Anything not bundled with Haiku core, is strictly not a Haiku component. If an application needs it, then the component should ship with the application itself, bundled in the applications lib directory.

Why isn't this a good solution? It's not a retoric question, I just don't know.

Because there will always be applications that require a lot of other software that isn't part of Haiku's base system. Is Haiku going to contain SDL? Perl/Python/Java and which of their respective modules?

More fundamentally, if every bundle comes with all the libraries it requires, those libraries won't be shared no more, i.e. they waste some disk space and (more importantly) are being loaded into memory once per instance. When apps all use the same library file, however, that library will only ever be loaded into memory once. Considering the trend that more and more functionality is being provided by shared libraries, the amount of wasted memory could become rather large.

Re: Package Management for Haiku

by Zenja - 2011-01-05 13:22

Have you seen how the OpenGL extension mechanism works? When a manufacturer has a new hardware feature, they expose the functionality via a vendor specific OpenGL extension. When another manufacturer also implements the hardware feature, the Architecture Review Board get together and decide to promote the extension from vendor specific to EXT. After a few years they reevaluate whether the market is using this extension, and if it's popular, they promote it to ARB status, before finally moving it into the core.

For Haiku, we can use a similar mechanism. Take SDL as an example. If only a small number of apps are using SDL, it remains bundled with the application (/apps/name/lib). When a set number of apps require library, it gets promoted to /optional/lib. Once it gains serious traction, it moves to /common/lib. If it becomes "compulsory" to have, then it's time to move it to /system/lib.

Re: Package Management for Haiku

by Zeno Arrow - 2011-01-05 14:46

@Zenja
But that's still inefficient. Let's use SDL as an example library, if user feedback shows roughly 50% of the userbase uses at least one SDL app then by your logic it should be included with the OS as enough people will make use of it. But what about those other 50% who now have a library installed that they'll likely never use, what do they gain from this? This is just one library, problem can easily multiply. Also, how is the SDL library going to be updated? The logical way would be to use a package manager, but you don't want that, so what do you propose? Download and copy over the old library? What about if the library and configuration files spreads itself out over the file system, which isn't unheard of.

Package management solves all this by making sure you only have the files you need.

Re: Package Management for Haiku

by sarienpalth - 2011-01-07 10:07

EDIT: After reading the second page, I've determined that this post is probably useless. If the moderators of this thread don't feel that this post adds anything positive to the conversation, then please feel free to delete it. -- SP 20100107 0507 ET

Actually, this raises another good question/point: should the idea of being able to drop a "Resource Bundle" in a location and have it "just work" also be applied to Library Frameworks (for some reason I wanted to say Extentions)? If this is the case, then being able to have multiple versions of a library coexist with each other in a sane way would be possible. At that point, there would be a golden age in computing and complete end-user laziness would be entirely possible. :P

If the "Drop and Run" scenario is implemented, then add-in management installation is simple enough for end users to do it themselves without the need of an add-in manager. However, if there are specific file system search paths that have to be updated or if the end-user does some weird stuff and places Application and/or Library Resources in strange places, then there will have to be some user changes to the operating system, though minor ones.

The thing to keep in mind is just because Application and Library Resources become "bundles" or otherwise single points on the file system doesn't mean you can't have an add-in manager to install them for you. All that creating Resource Bundles do is make it easier for a user to also install Application and Library Resources on their own.

To combine the ideas of an add-in manager and using "Resource Bundles," I can see a single application (say, called Meditate for ease of notation) performing as 1) an add-in manager and 2) a system update and upgrade facilitator. Theoretically, all that Meditate would do in a particular case is place the add-in in a well-defined location and make the necessary updates to the Application Index and/or Library Index. This isn't very different from what a package management system does; it's just not flinging files all over the place. Also, keep in mind that an add-in manager allows users to be even lazier and not worry about downloading add-in dependencies manually. Instead, the user can rely on the add-in's metadata information for determining dependencies rather than their own understanding of the Application or Library Resource.

In short, I think the people who are crying wolf about package management are really more annoyed with how package installation is handled, not with the overarching concept of package management itself. After all MacOS X bundles aren't pretty on the inside either, they just look pretty as a desktop icon (the underlying ugliness of which is invisible). I think Haiku could really do something groundbreaking if it considered using Haiku Packages in a "Drop and Run" way for both Application and Library Resources and implementing what I've been calling "add-in management." Some major obstacles might include some parts of the Haiku Package File Format being redone (for example, for add-in updates, perhaps there could be a HCHG file type that only includes the files within the add-in bundle that need to be changed or patched? There may be no change necessary to the Haiku Package format, but I need to go read the specifications.)

Re: Package Management for Haiku

by X512 - 2011-01-04 17:47
Zenja wrote:

Haiku, however, ships as a complete system, with a well defined set of bundled libraries. Anything not bundled with Haiku core, is strictly not a Haiku component. If an application needs it, then the component should ship with the application itself, bundled in the applications lib directory.

Totally agree.

zooey wrote:
hobsii wrote:
Zenja wrote:

Haiku, however, ships as a complete system, with a well defined set of bundled libraries. Anything not bundled with Haiku core, is strictly not a Haiku component. If an application needs it, then the component should ship with the application itself, bundled in the applications lib directory.

Why isn't this a good solution? It's not a retoric question, I just don't know.

Because there will always be applications that require a lot of other software that isn't part of Haiku's base system. Is Haiku going to contain SDL? Perl/Python/Java and which of their respective modules?

More fundamentally, if every bundle comes with all the libraries it requires, those libraries won't be shared no more, i.e. they waste some disk space and (more importantly) are being loaded into memory once per instance. When apps all use the same library file, however, that library will only ever be loaded into memory once. Considering the trend that more and more functionality is being provided by shared libraries, the amount of wasted memory could become rather large.

Nowdays it isn't a broblem to put all depedencies in application distribution at all. Modern drives have terabytes of memory. All software don't take more 1% of them. More important problem is simple software installation and it reliability.

For example Gimp and Inkscape for Windows includes all depedencies(gtk+ etc) and it install and work fine.

zooey wrote:

and (more importantly) are being loaded into memory once per instance.

I don't think that you really need more than 10 applications at the same time. Haiku isn't a server OS.

Re: Package Management for Haiku

by akaposi - 2011-01-04 12:20

I think it's worth to have a look at declarative package management (and in a wider aspect, declarative system configuration management). A good example is the Nix package manager for Linux (and the Linux distribution built on top of it, NixOS): http://nixos.org/nix/
Good luck and thanks for your work!

Re: Package Management for Haiku

by jido - 2011-01-04 14:52

Aren't all the cited package managers based on the same principles? As previous poster suggested, you should look at different solutions (nix...)

PACKAGE BUILDER
nice-to-have: provides a helper program to convert any buildable software into a package file (by means of a trivial recipe or without any recipe whatsoever) - the resulting "faked" package can then be activated and will take part in dependency resolution

Yes please!

PACKAGE ACTIVATOR
activate packages system-wide or per-user

How does this work? Is there any existing package activator which provides both options?

nice-to-have: show list of config files for a package (and allow to backup/restore/purge them)

Is there any existing package management which maintains that information? If not, why?

PACKAGE MANAGER DETAILS
maintains (add, list, remove) a set of prioritised repositories (including user-defined repos)

I oppose this.

I think packages should be retrieved (1) from individual sources, or (2) from a single central repository (which can be user-defined eg. for development needs). At the most you could use redirection, so the "central repository" is really a URL providing a list of repositories to get software from. But the package manager should never show more than one repository. I hate having to manage repositories in Linux.

When getting a package from a web site or physical media: there is only one Haiku, so there is no confusion which packages can be installed on which distribution as we see in Linux.

always computes all required actions and shows them to user before applying them

Could you default to hide actions which are simply "copy file to location...", as they are not much of interest when quickly checking the list of actions.

supports listing and getting rid of dependency-only packages (which were being activated only as dependencies of some other package)

For what purpose? Are dependency-only packages left behind when uninstalling all dependent packages?

That gets me thinking... Relative to the discussion about self-contained applications: what if there are two types of packages, "application" packages (can install anywhere, does not write outside its target directory) and "system" packages (installs at predefined paths, touches the system). Application packages can depend on system packages. System packages are preferably retrieved from the central repository.

Re: Package Management for Haiku

by zooey - 2011-01-04 15:53
jido wrote:

Aren't all the cited package managers based on the same principles? As previous poster suggested, you should look at different solutions (nix...)

Yes, you are right - during my reading up today, I have added several more package managers to my list. I will write more about that in a new blog entry later today.

jido wrote:

PACKAGE ACTIVATOR
activate packages system-wide or per-user

How does this work? Is there any existing package activator which provides both options?

Haiku's package-FS supports this (at least in principle). Basically, you can drop packages into two different folders in order to get them activated, one for system-wide package activation and another folder for activating the package just for yourself.

jido wrote:

nice-to-have: show list of config files for a package (and allow to backup/restore/purge them)

Is there any existing package management which maintains that information? If not, why?

apt/dpkg does preserve this info to that extent that it does not remove config files unless asked to do so explicitly ('purge'). The basic idea behind this feature is that it would be nice to be able to downgrade back to an earlier version of a specific software and get the old state of the configuration back (instead of the potentially upgraded and incompatible configuration of the newer version).

jido wrote:

PACKAGE MANAGER DETAILS
maintains (add, list, remove) a set of prioritised repositories (including user-defined repos)

I oppose this.

Then simply don't use it (I imagine the default setup to only refer to one repository). Other people, however, might be interested in checking out testing versions of some packages. For use cases like that, support for more than one repository is a requirement.

[ ... ]

jido wrote:

When getting a package from a web site or physical media: there is only one Haiku, so there is no confusion which packages can be installed on which distribution as we see in Linux.

That is incorrect, as Haiku already supports two different target platforms (x86-gcc2 and x86-gcc4). So there already is potential incompatibility across packages for different platforms.

jido wrote:

always computes all required actions and shows them to user before applying them

Could you default to hide actions which are simply "copy file to location...", as they are not much of interest when quickly checking the list of actions.

I didn't mean individual file actions, but a summary of which packages are going to be installed, upgraded and/or removed.

jido wrote:

supports listing and getting rid of dependency-only packages (which were being activated only as dependencies of some other package)

For what purpose? Are dependency-only packages left behind when uninstalling all dependent packages?

Yes, they usually are being left behind. I suggest that the package manager should immediately ask to get rid of them or at least offer means to list and remove them at a later stage.

jido wrote:

That gets me thinking... Relative to the discussion about self-contained applications: what if there are two types of packages, "application" packages (can install anywhere, does not write outside its target directory) and "system" packages (installs at predefined paths, touches the system). Application packages can depend on system packages. System packages are preferably retrieved from the central repository.

Something along those lines might indeed be necessary, especially if package-FS doesn't scale well with many packages.

Re: Package Management for Haiku

by jido - 2011-01-05 09:47
zooey wrote:
jido wrote:

nice-to-have: show list of config files for a package (and allow to backup/restore/purge them)

Is there any existing package management which maintains that information? If not, why?

apt/dpkg does preserve this info to that extent that it does not remove config files unless asked to do so explicitly ('purge'). The basic idea behind this feature is that it would be nice to be able to downgrade back to an earlier version of a specific software and get the old state of the configuration back (instead of the potentially upgraded and incompatible configuration of the newer version).

Yes, nice.

Quote:
jido wrote:

PACKAGE MANAGER DETAILS
maintains (add, list, remove) a set of prioritised repositories (including user-defined repos)

I oppose this.

Then simply don't use it (I imagine the default setup to only refer to one repository).

Not good enough. If the functionality is there, it will be used and abused.
For testing, you could use indirection as I suggested-- how hard is it to put up a file with a list of repositories on a web server? On the other hand, a software developer can't ask that of all its users.

Re: Package Management for Haiku

by Clebin - 2011-01-04 16:04

It scares me that we're just swapping one flawed system for another.

I believe package management has hindered the development of commercial software for Linux over the last 10 years. The perceived solution to that is a kind of centralised App Store built onto the package management. That goes even more against the grain of open-source, turning distros like Ubuntu into closed eco-systems like Apple's.

I believe in commercial software, but I also want openness in distribution. With package management, I worry that the two are almost mutually exclusive.

I don't care about a few meg of hard-disk space, but if we're worrying about resources, what about the time and bandwidth used when continually updating apps and frameworks that aren't being used? How much unused data is downloaded from Apple's App Store on a daily basis, and how much will it grow as users keep every app they've ever bought?

I like that traditional Mac apps check for updates when you run them, and then download, update and relaunch as needed. I don't see that Linux's daily upgrade cycle offers much advantage. If self-updating were formalised as an API in the OS, that would be better. If apps also registered the files they used for an easy uninstallation, it would be better still.

I understand that the traditional means of distribution can still be used alongside a package manager, but what are we encouraging as the default? Because whatever is encouraged is what we'll end up with.

A package manager is brilliant for OS updates and server-type software like PHP. I love that I can add modules using the package manager. But for desktop apps, I prefer that I can organise them as I want, run multiple versions of Firefox easily and have them self-update. So perhaps some distinction needs to be made between the two.

Haiku has the opportunity to address the weaknesses of every current OS's means of distribution, but all this presents is a copy of what Linux currently has. I personally think that's a wasted opportunity.

Chris

Re: Package Management for Haiku

by Clebin - 2011-01-04 16:31

By the way, my solution to my above post:

* Restrict the package manager to OS updates in R1.

* In R2, introduce app-bundles and APIs for self-updating and uninstalling apps and encourage developers of desktop apps to adopt that approach. Maybe allow Unixy server-type software to use the package manager, but only provide a command line tool by default.

* Encourage distro developers to keep to this principle.

Many will disagree, but that's my favoured solution to have the best of all worlds.

Chris

Re: Package Management for Haiku

by halo - 2011-01-04 20:32

I largely agree with Clebin. Clearly Haiku needs a good method of distributing, installing and updating software.

However, I have several concerns about Linux-style package management with dependency resolution. They are:

1. It encourages reliance on dependences, and that's a bad thing. You start with a small, fast, core operating system. Then some piece of software decides it needs Python, so it installs Python. Another piece of software requires Qt. Since most people have Python and Qt installed, a new application developer isn't going to worry about using both Python and Qt himself. Eventually, you get a huge tangle of dependencies (including older versions) that can never be removed, resulted in several gigabytes of bloat coming with any Haiku system. On top of that, no-one bothers to use your nice static native APIs. Why bother? The other APIs are better documented, non-native ports from other OSes are much easier to do, and there is pretty much no downside. Besides, why bother improving the Haiku API when you can simply import a better one? This is one of the main reason that Linux distributions are terrible. By encouraging dependencies, you are effectively turning Haiku into another Linux distribution.

2. It creates a two-tier system which favours open-source software, making it very hard for commercial software developers. By making it really easy to install open-source software with its 15 zillion GPL dependencies, half already pre-installed, you make it very difficult for commercial software to compete meaning that no-one is ever going to bother. I don't regard Haiku as an operating system steeped in "free software" ideology at its heart, yet that's what a package manager effectively makes it. What Haiku needs isn't a "package manager", it needs an "App Store" or "Marketplace" much more.

3. The way all Linux package managers fundamentally work is absolutely awful. They do magic, litter crap all over your system, and cause absolute havoc when they go wrong. They are fundamentally unfit for purpose, and completely user-unfriendly. This also applies to Windows. Mac OS X is better, but not perfect. Since Haiku doesn't have to be fitted around decades of backwards compatibility, it should use something that actually works well. Standalone executable packages, to me, is the only thing that actually makes sense.

4. Making a good 'package manager' UI is really, really hard. They are mostly horrible to use. The only decent successful ones are Apple's App Store, Valve's Steam and, perhaps to a lesser extent, AppBrain when coupled with Google Marketplace. All the Linux ones I've used are all horrible - they present you with a big list of cryptically-named software with no indication of what's good, what's bad, and what's popular. They barely offer any screenshots, or any indication of how long something will take to install. They are completely unfit for purpose.

I was hoping that the Haiku developers realised these sorts of problems and that's what made it different to Linux distributions, but perhaps not.

@tonestone57: You end up trading some security for user-friendliness. Since Haiku is a desktop OS, I think that's a good trade-off, and Apple, Microsoft and Google clearly think so too. I think Haiku has always placed much more emphasis on user-friendliness than security given that there's a lot of interface completed yet not much in the way of security.

Re: Package Management for Haiku

by halo - 2011-01-05 06:46

Thinking about it, why not implement Mac-style self-contained executable packages first (for R1) and then, if still desired, build a package manager on top of that (for R2)? A package manager sounds like a strict superset of the basic package functionality to me.

Re: Package Management for Haiku

by drcouzelis - 2011-01-04 21:01

Will it be possible to update the entire Haiku operating system with the package manager?

Re: Package Management for Haiku

by hobsii - 2011-01-04 21:20

How was software install on BeOS R5? I think most apps were installed with some kind of of install wizard. And I remember some kind of software centre were we could remove apps. Right?

Re: Package Management for Haiku

by X512 - 2011-01-04 21:31
hobsii wrote:

How was software install on BeOS R5? I think most apps were installed with some kind of of install wizard. And I remember some kind of software centre were we could remove apps. Right?

Yes. BeOS have package manager called SoftwareValet. It uses own binary format pkg. Haiku have tool to install this packages(and remove too, but it isn't included in image).

Re: Package Management for Haiku

by hobsii - 2011-01-05 10:26

Yes, now I remember! It even had a Shop... http://toastytech.com/guis/b5pevalet.png

Re: Package Management for Haiku

by andrea.paiola - 2011-01-04 22:02

"As Mac OS X" is too simpler?

Re: Package Management for Haiku

by hobsii - 2011-01-05 10:43

Because it works with no hassles... Seriously, in my opinion it has only two things missing: first, when you delete an app (the package that contais all the app) file the system should prompt for deletion of related stuff on Application Support; and second, a common auto-update system for third party applications. But since I don't uninstall so many apps, an most have auto-update integrated, it's not a huge problem. I still prefer it to Linux apt or Windows wizards.
You can also install some some kind of package management for Unix ports. It's called MacPorts, but I always avoid it and try to find self contained solutions.

Re: Package Management for Haiku

by Moopark - 2011-01-05 06:48

Why are you looking at Linux package management systems for a desktop OS?!?

If you don't understand why apple got it right -- download the app, it's "installed" -- then you're the wrong person to be looking into this problem.

You solve the application distribution problem (it's NOT a "package management" problem) and grow the ecosystem of native applications by:

- Providing a comprehensive set of native APIs that provide ALL of the functionality required by most applications
- Maintain binary compatibility across releases so that applications that rely on your libraries continue to work without requiring lockstep upgrades of all the software users have installed.
- Let app developers bundle minor non-standard libraries directly with their apps
- Let app developers use an in-app update framework (like on the Mac) so that they can very simply ship updates whenever they want.

Looking to Linux for the answer is just plain dumb. Linux package management was designed as a way of dealing with an already disastrously complex and incompatible software ecosystem, and while it does the job on a server, it does NOT belong in a desktop OS.

Re: Package Management for Haiku

by Zeno Arrow - 2011-01-05 11:17

@Moopark
Have you actually used a package manager? It makes it really easy to find apps, install them, update them and remove them. They make keeping your system up to date easier, compared to static app bundles. If you'd still prefer app bundles, fine, but don't force your inefficiencies on me.

Like I said in earlier posts, I'm in favour of user choice, and the solution I outlined allows supporting both traditional package management and app bundles, depending on what the user prefers. If people stopped and read what I said they'd understand we can all have what we want.

@others
To those who would prefer an App Store-type setup compared to a package manager, you make me laugh because the basic differences are only skin deep, they're practically the same thing! All you really want is a package manager with a simpler interface that allows you to pay before you download, that's all easy to implement.

Re: Package Management for Haiku

by jido - 2011-01-05 11:58
Zeno Arrow wrote:

@others
To those who would prefer an App Store-type setup compared to a package manager, you make me laugh because the basic differences are only skin deep, they're practically the same thing! All you really want is a package manager with a simpler interface that allows you to pay before you download, that's all easy to implement.

No word about payment, but when it comes to a "store": please make that a single store! Or at least a single URL (see my comment above). We don't want to manages stores, repositories or whatever they are called. All useful software should be found at a single location or location set. Don't repeat mistakes of the past.

Re: Package Management for Haiku

by Zeno Arrow - 2011-01-05 12:23

@jido
What difference does it make if it's one repository or many? You'd still be able to access them all through a single, consistent interface.

I get the feeling you don't know what a repository is. Think of it as a download location. I might want to look through all the apps from location A, B and C, or I might want to only rely on location A for all my apps. The point is, it's up to you.

Or using a store analogy, imagine you could sell iPhone apps direct to customers (i.e. Apple don't get a cut) but still using the App Store interface. This would be possible if Apple allowed 3rd party repos (which of course they never will, though Cydia/Rock offer a way around that). The point is, with Haiku, you can choose how many download sites you want to use, it's up to the user to decide.