- Debugger: Getting mixed signals
- 'Packaging Infrastructure' Contract Weekly Report #4
- Haiku monthly activity report - 06/2015
- 'Packaging Infrastructure' Contract Weekly Report #3
- 'Packaging Infrastructure' Contract Weekly Report #2
- GCI 2014 winners trip report (mentor side)
- TeX Live and LyX; Changes to the boot code
- 'Packaging Infrastructure' Contract Weekly Report #1
- Beginning of 'Packaging Infrastructure' Contract
- Haiku monthly activity report - 05/2015
Package Management: New Contract Starts
My new contract has now officially started. At the moment it looks like its duration will be at least three months. A big thanks to everyone who donated to make that possible!
On the technical side not that much has happened since my previous blog post a week ago. Oliver has continued to work on cross-building and, given the potential for surprises and "interesting" problems, probably will do so for a while longer.
I have added support for the new package file format features (declaring settings files, users, groups, and post-install scripts) to haikuporter. I have also added a checker for our packaging policy (at least a part of it), so that it can now already be detected while building a package whether it violates the policy and therefore might cause trouble later.
Furthermore I've started implementing a dependency analyzer in haikuporter that figures out which packages we actually need to cross-build so that all other packages can be built. Part of the implementation will also be reused for another missing feature: A build recipe only declares the minimum version of the package's dependencies. ATM these declarations end up unmodified in the package, which is not correct, though. E.g. if the minimum requirement for a certain library is version 1.0, the package might actually be built against version 1.4. Since new features of the later version might be used implicitly or even explicitly, the resulting package will require the newer library. So we need to replace the minimum requirements declared in the recipe by the actual versions we've used when building the package.
Once that feature has been implemented (barring potential further issues) we will once again -- and hopefully for the last time -- go through all build recipes to update them and build new packages. I'm not sure yet which of the remaining tasks we'll tackle after that. Possibly some build system work (e.g. getting the hybrid builds working again), maybe already pushing for integration with repositories.
As was mentioned in the recent Haiku, Inc. news post, pre-built images of the package management Haiku are now available for testing. I believe Matt builds them weekly, so the second one should hit the server some time later today. Thanks to reports by Humdinger I've looked into and fixed a few issues the first image has; like most Wifi firmware being missing from the image and queries not working correctly in some cases. I also made the Installer usuable again, so it will be possible to install Haiku from the upcoming image using that application. I recommend doing that, if only to enjoy watching how fast the installation process is.
Issues with the package management Haiku can be reported as usual via Trac. There's now a version "R1/Package Management" the tickets can be assigned to.
- bonefish's blog
- Login or register to post comments
Comments
Re: Package Management: New Contract Starts
Thanks for the progress update. Thanks for all the work you and Oliver are doing.
I've noticed that the HaikuPorts site does not have any updated info on the new recipe format. Is the format finalized? Where will the new HiakuPorts recipe format be documented?
Re: Package Management: New Contract Starts
To date, all the package management-related docs have been located here: http://dev.haiku-os.org/wiki/PackageManagement
I suspect the finalized recipe format will wind up there as well.
Re: Package Management: New Contract Starts
Re: Package Management: New Contract Starts
Looks like another wheel reinvented. The page lacks example to look at. Is it also possible to provide a comparison to the similar format (YAML?)? YAML subset may be a good thing - it might be easier to produce, edit with existing tools.
Re: Package Management: New Contract Starts
Oliver has documented the recipe format (including examples): http://ports.haiku-files.org/wiki/HaikuPorterForPM/BuildRecipes
Regarding the other statements/questions of your comment, I'm a bit baffled. What wheel has been reinvented? We're using shell scripts, which are simple and powerful. Using something like YAML would only complicate things significantly without any benefit I can think of. I don't see why you would edit the recipes with anything but a text editor.
Re: Package Management: New Contract Starts
A lot of questions after the first look.
Regarding package format. What for save bytes in files header? uint64 for every field will allow to read the numbers as an ordinary array. Why big endian? Isn't x86 is the most popular platform for Haiku?
Why using own format at all? How it will be better then existing stuff? Why not just use .zip format with meta information as files. For example http://www.python.org/dev/peps/pep-0427/ The package format looks like tight binary chunk with a lot of variable data types. This increases algorithmic complexity when parsing it and makes it harder to develop tools that work with Haiku packages on other platforms.
About haikuporter. What is the license of it? Is there analysis for existing solutions for dependency resolutions? Is there something that can be already reused? This dependency resolution component could be handy for Python packaging tools.
It is good to see some alternatives popping up. Very curious. =)
Re: Package Management: New Contract Starts
Answering your questions:
Saving bytes in the header saves bytes in the file. So there should be a good reason not to do it.
I don't see why one would want to read the header fields as an array.
What endianess is used doesn't really matter as long as it is fixed. Big endian is nicer to read in a hex editor and it is the "network endianess".
I'm not aware of any file format suitable for our purposes. There might be one, but I didn't find any when doing research for packagefs. Something zip based may sound like a good option, but it doesn't support random access to file data. E.g. if you have a 100 MiB file, reading the last few bytes shouldn't require us to decompress the rest of the data first.
I can't quite follow your assessment of our file format. I also don't see why anyone would develop tools that work with Haiku packages on other platforms.
Regarding haikuporter's license, I was about to answer MIT, but looking at the source files, their copyright header doesn't mention any license.
The dependency resolution uses pkgman, which uses the package kit, which in turn uses libsolv. As is the package kit, pkgman, and the respective code in haikuporter are tied to Haiku repositories/packages/package infos. So, no, you won't be able to reuse much (if any) of this for other purposes. Most of libsolv is repository/package format agnostic. To support a new format you basically only need to add the code that reads your format and converts the meta data to libsolv's internal format. I believe there already are python bindings for libsolv.
Re: Package Management: New Contract Starts
Thanks for the libsolv explanation. I am going to read about satisfiability soon. Other comments are below.
Answering your questions:
Saving bytes in the header saves bytes in the file. So there should be a good reason not to do it.
I don't see why one would want to read the header fields as an array.
48 saved bytes per package (https://gist.github.com/techtonik/5833449)? IMHO that doesn't worth it. 48 bytes per package kill the advantage to easily parse the binary format outside of C, without complicated structure definitions.
What endianess is used doesn't really matter as long as it is fixed. Big endian is nicer to read in a hex editor and it is the "network endianess".
I don't agree about "doesn't really matter" for cross-architecture file format. But the argument about hex dump readability is a good one. I like it.
I'm not aware of any file format suitable for our purposes. There might be one, but I didn't find any when doing research for packagefs. Something zip based may sound like a good option, but it doesn't support random access to file data. E.g. if you have a 100 MiB file, reading the last few bytes shouldn't require us to decompress the rest of the data first.
Are you sure that .zip doesn't allow random access? As far a I can see HPKG uses the same DEFLATE compression format - http://tools.ietf.org/html/rfc1951 And also I don't see how can you decompress the last few bytes without storing separate dictionary for each chunk. This can increase the size by much more than a hundred bytes that you are trying to save. =) Are you sure that random access by file is not enough?
I can't quite follow your assessment of our file format. I also don't see why anyone would develop tools that work with Haiku packages on other platforms.
Think about web tools in Python, Ruby etc., or do you want to run unchecked C code on your server or code community package management, validation and analysis tools in C?
Regarding haikuporter's license, I was about to answer MIT, but looking at the source files, their copyright header doesn't mention any license.
And what is the policy for Haiku code? Is it going to be proprietary? =)
Re: Package Management: New Contract Starts
48 saved bytes per package (https://gist.github.com/techtonik/5833449)? IMHO that doesn't worth it. 48 bytes per package kill the advantage to easily parse the binary format outside of C, without complicated structure definitions.
Besides that don't really see why one would want to read the package format in python, I'm pretty sure, it would be less complicated to create package kit bindings e.g. via SWIG than to write a package format parser from that scratch.
Are you sure that .zip doesn't allow random access? As far a I can see HPKG uses the same DEFLATE compression format - http://tools.ietf.org/html/rfc1951 And also I don't see how can you decompress the last few bytes without storing separate dictionary for each chunk.
There is a dictionary per chunk.
This can increase the size by much more than a hundred bytes that you are trying to save. =)
Using chunks is not for saving bytes. It's to allow for fast random access.
Are you sure that random access by file is not enough?
Yes.
I can't quite follow your assessment of our file format. I also don't see why anyone would develop tools that work with Haiku packages on other platforms.
Think about web tools in Python, Ruby etc., or do you want to run unchecked C code on your server or code community package management, validation and analysis tools in C?
And what is the policy for Haiku code? Is it going to be proprietary? =)
Haiku code is mostly MIT. After discussion on the HaikuPorts development list, it has been agreed, that that's also what everyone assumed for haikuporter.
Re: Package Management: New Contract Starts
I donated, but nor for this. Package management, while nice, is really low, low, _low_ priority. The way it worked in BeOS was just fine for the most part.
I'll continue donating, of course, but I wish I could do it with a proviso.
Re: Package Management: New Contract Starts
I disagree so much. Not managing packages is ok when you have little bit of software, it's plain nightmare when there's a ton of apps. Especially considering ported stuff like Qt software, Java-based and who knows what else. What about regular system updates or upgrading to new versions without reinstall or need for USB sticks and Live CDs? What about easy as hell online catalog (like App Store) using the repositories of haiku packages, downloading & instaling with One Click? Package Management is one of the most important stuff after internal core system parts, stability and hardware support.
Priorities
I think completion of package management and the new debugger (C++ source-level debugger) are the two last, distinct milestones on the road leading up to Release 1.
The current priorities listed on the page linked below likely still apply, for the most part:
http://dev.haiku-os.org/wiki/FutureHaikuFeatures
The idea is to make Release 1 not just liveable for people who knew and loved the BeOS, but to provide the tools and infrastructure that developers have come to expect from an operating system these days, and that includes package management, a debugger and tools to make apps available in different languages and locales.