Blogs

End of package management contract

Blog post by zooey on Sun, 2013-11-10 16:47

From mid-June to early November, I have spent 320 more hours working on package management. After having worked on the bootstrap support in HaikuPorter (which Ingo has already mentioned in his blog entry), I have spent some time reworking the way how Perl and Pythong organize their modules: Scripting languages with module support usually expect to be able to build modules from source and install them somewhere into the hierarchy of the scripting language. In our case, they can no longer just put the built (site-specific) modules into the "standard" folder, as that one is read-only. So the configuration of Perl and Python had to be changed, such that modules built from source are being installed correctly into a writable folder (in the non-packaged hierarchy). Perl already supports the notion of a vendor-modules folder, where packaged Perl modules will be installed (an example of such a beast are the perl modules contained in the git package). That idea has been extended to Python (such that it will automatically pick up the python modules provided by the mercurial package). Separating packaged modules from local (built from source) modules is an idea that should be followed by all (not only scripting) languages ported to Haiku.
The updated perl, python, git, mercurial and scons packages that are the result from that work have not yet been published, but I will do that during the next couple of days.

The package management branches of both haikuporter and haikuports have been merged into the respective master, i.e. all porting work done in there will produce packages. Several new 'testing-' branches will be added to the haikuports repository soon in order to be able to separate work on the bleeding edge from the work that tries to stabilize a given set of packages for a release.

Finally, I have started with work on the infrastructure required for automatic building of repositories and (at a later stage) packages. The first effect of this is that whenever a developer changes the version of a package used by Haiku's build system, the respective repository will be created automatically. Those repositories are being published to http://packages.haiku-os.org/haikuports/master, from where pkgman will download packages.

Summarizing those months of work: I think we have managed to put most of the work behind us, but we haven't reached the goal just yet. There's still more work to do and I personally will return to continuing that work in my spare time.

Live from Alchimie X

Blog post by mmu_man on Sat, 2013-11-09 17:48

A quick hello from the Alchimie X demoparty where I gave a short talk this morning about Haiku news (PM merge, Sam460 port...) ;-)

Slides are here (french, sorry).

Lots of fun here !


Photo by @AmigaImpact.

WebKit weekly report #6

Blog post by PulkoMandy on Fri, 2013-11-08 07:32

Hello world!

Sorry for no report last week, I was not in front of the computer on Friday.

Anyway, I got the HTTP authentication working last week. This was the last missing feature in the Services Kit version of WebKit when compared to the current cURL one. The next step is to fix the new rendering bugs.

The rendering side of things is mostly built into WebKit, so I didn't want to fix it on the old version we are still running. So, I have started merging WebKit changes all the way to the current revision. Unfortunately, our WebKit repository wasn't created the right way, and the commit hashes for WebKit commits didn't match the ones for the official WebKit repo. I had to create a new repository, and manually match the commits and play with git rebase, merge and cherry-pick to rewrite all ourwork against the official WebKit commits. This took some time, as there are 120000 WebKit commits in our repository, plus our own changes.

I have removed the old repository and uploaded a new one. My work happens in the 'rebased' branch, which is the default one.

With the repository rebased onto the official WebKit, it is now possible to use git merge to merge commits directly from the official WebKit mirror.. Our port was about 2 years behind, in WebKit world that's more than 30000 commits. I didn't merge those all at once, instead I'm working with ranges of about 2000 to 3000 commits. While one range is merging and compiling, I can review the commits for the following one, and take note of the important changes. This helps a lot knowing what happened when I need to do a merge or fix the build.

I have merged commits up to april 2013 so far. This is the point where the Chromium port gets removed, and later on the wxWidgets port. The GTK one switches to CMake as a build system (the same we use now). These are the changes that creates more conflicts with us, so I'm going to merge this part in smaller chunks (I tried a big one and got a build error I didn't know how to solve).

Following the removal of the Chromium port, WebKit entered a phase of cleanup and simplification. This includes removing Chromium-specific stuff, making the code simpler in some places, and also using some C++11 features that help with detecting errors at compile time.

One important change is the introduction of "platform strategies". Before that, porting webKit involved implementing many classes where the header file was shared accross platforms, but not the C++ sources. This led to a lot of #ifdefs-guarded platform-specific code all over the place. The platformStrategy class makes all the platform specific code located in the same place, and the shared code go elsewhere. Our port will be much easier to keep working this way.

So, which new features are brought in by this huge merge ? Well, none so far. Most of the features in WebKit are optional, and to keep things easier, we disabled most of them. As features are sometimes removed when no port uses them, I'm forst going to merge all the changes, then see what can easily be enabled. Some of the features involve only multi-platfomr code, but others require some porting.

Anyway, the up-to-date WebKit is faster, smaller, and has much less bugs, which are already very good things. When I'm done with the merge I will have a look at running the WebKit tests, as I'm pretty sure this will help catch some bugs and at least give us a better idea of the work left to do. Running a testsuite is now much easier thanks to the use of CMake, where the guis working on the EFL port made some changes to the build system to help with setting this up. I will also test all the websites I have listed on my TODO/check list, to make sure things that used to work are still working.

I hope to have something in shape for a merge next week or the one after that. The remaining part of the month will be dedicated to enabling some of the extra features, but with only 2 or 3 weeks left I will not be able to work on the most complex ones. Haiku, Inc currently can't fund another month of development, so it looks like I'll have to stop there, unless there are more people donating money this month.

WebKit weekly report #5

Blog post by PulkoMandy on Fri, 2013-10-25 06:50

Hello there !

Well, not so much progress on WebKit this week. I spent most of the time working on CMake code to get it to generate hpkg files. I got something that works well enough to link Web+ to it, so I can test things with the actual browser instead of HaikuLauncher.
Today I added the cookie jar persistence, so Web+ remembers all the cookies when you exit and relaunch it. I also started working on HTTP authentication. These are the two features I couldn't test with HaikuLauncher, as it lacks some code for them (saving cookies on exit, and showing the HTTP authentication password prompt window).

I also implemented the protocol handler for file: URLs, so now I can browse the Be Book and Web+ default home page works as well.

With the HTTP authentication fixed, I think I will be on par with the current code, so I could merge this into Haiku now for you all to use. However, there are some new rendering glitches that maybe I should fix first. What do you think?

Scheduler work progress

Blog post by Paweł Dziepak on Mon, 2013-10-21 16:14

Thanks to generosity of Haiku supporters, I will be able to continue my work on scheduler in November. It's high time I wrote a report about what has already been done. As it was mentioned before my work can be tracked in the scheduler branch in my repository. Commit descriptions and some comments in the scheduler code contain more detailed motivation behind some of the decisions I had to make. In this post, though, I will concentrate how my work looked so far and what I plan to do next.

WebKit weekly report #4

Blog post by PulkoMandy on Fri, 2013-10-18 07:07

Time for a report again !

So, over last week-end and monday morning I finally got Ninja working. I already said some words about it, Ninja is meant to replace Make for building projects. It has less features, because it is designed to run files that are generated, rather than hand-written. In my case, CMake generate the Ninja failes. I had problems building Ninja that turned out to be abug in our Python port. I didn't fix it, but I found a way to work around it.

The use for all this buildsystem work? Well, we're now using one of WebKit standard build systems (CMake), and using it in a way that's fast! CMake can also generate makefiles, but running these would take about a minute just to figure out which files to recompile. With Ninja, this is done in a matter of seconds, because the simpler rules are faster to parse. Ninja was actually written just for that purpose. One minute may not seem much when the full build of WebKit takes about an hour, but when I'm working on just one or two files to track down a bug, the edit/compile/test cycle is now much faster.

So, with this out of the way I was able to more easily try out things and fix the issues as I found them. I mentionned the Opera cookie testsuite in last week report : most of the tests are now passing, and we even get better results than the old Curl backend in some places. The remaining issues are rather minor ones, but I can fix them if you can find a website where they actually matter.

In one of the blog post comments someone mentioned that Windows Live Mail (or Outlook, as it's now called) didn't allow to display messages. I tried to do that with HaikuLauncher and the new code and noticed I couldn't even login to the website. Getting this to work was a bit tricky, leading to fixes in 3 different places: settings cookies from Javascript code, proper handling of URL redirections, and telling WebKit that "aspx" files served without a mimetype are web pages.

I already told you about the cookies, so here are some more details about the two other issues: the URL redirection problem happened when an HTTP server gave us a 302 answer (that is, a redirection) with a relative URL. The HTTP specification says an URL should be used, most likely starting with "http://". But the URL specification was later replaced with the URI one, which allows URIs without that part, for example "/path/to/page.aspx?param=foo". Our BUrl class wasn't able to understand such things and was desperately looking for the "//" part, leading to an endless loop. I replaced the homegrown parser with a regular expressionbased one, using the regex that is kindly provided inthe RFC for URIs. I had to fix our RegExp class in the shared kit (this is a collection of internal classes that are not, yet, part of the official API). So, I had to fix the RegExp class as it didn't handle optional capture groups in the regular expression properly. This includes things like: the URI may, or may not start with "scheme:", then, maybe there will be an authority part, of the form "//authority", then a path, which is anything following the authority and up to th first ? or # or the end of the URI, and then there is the query and fragment, and they are also optional. RegExp allow to write this in a more compact and computer understandable format and get the parts extracted to separate strings.

As for the "aspx" file handling, the HTTP protocol says that the server should give a "content-type" header with the MIME type of the file it's going to send. This fits well with the use of MIME types in Haiku and would be very nice. However, Microsoft servers for login.live.com sometimes don't include the header. Web+ used to assume the file was to be displayed on screen in that case. You may already have seen it trying to render a video file as plain text, as a huge page full of strange characters. Well, that issue was fixed, and if in doubt, it will now download the file to disk instead. A list of well-known file extensions is used to attempt a guess when the "content-type" is missing. I added aspx to that list and mapped it to text/html, so, the URLs ending in aspx will now be shown on screen.

I had a run of testing after all these fixes and things are getting much better. It helped fix some more issues in github and gist, and I could also browse some documents in Google Drive. I think we still have some problems with file upload, however, I'll have to look into that. Uploading video to youtube, as someone pointed out in the forums, also fails.

I caught a crash when opening a new window in HaikuLauncher, but I'm not sure if that would be a problem for Web+. Also, I wanted to work on HTTP authentication (the login/password prompt) and this is not available in HaikuLauncher, so a working Web+ will be needed. To get there I wanted to build a package for WebKit. To do this, I first tried writing an haikuporter recipe. I didn't get very good results with this. First,it means working separately from my development git checkout, so it takes twice the sapce on disk (and we're talking multiple gigabytes) and it takes forever to checkout. Second, there's no easy way to use distcc with haikuporter, so things are keeping my CPU very busy for a long time. And since this isn't my development tree, there are a lot of things to rebuild.

So, I decided to try a different solution: using CMake built-in support for generating packages. I tried that yesterday and found out that it wasn't working. I tried on a very small test project, and with help from the CMake IRC channel I found the issue: our CMake port is lacking some features that are available in the Linux version, and not needed on Windows. This confuses the Ninja generator which isn't ready for that case. This is a rather technical issue related to the way executables and libraries are linked together. This is different in the tests (the lib and executable sit together in a directory) and installed version (they are installed somewhere on the filesystem and the OS, more precisely the runtime_loader) must bind them together. On Linux, CMake makes up for that by rewriting part of the executables when installing or packaging them. This isn't enabled in the Haiku version, so it will instead try to link executables again when installing them. However, the Ninja generator is skipping the link step (it was never tested in that case) and tries to install or package non-existing files. This shouldn't be too hard to get working just like on Linux, more news next week !

There is another missing step to make this complete: CPack, the CMake component that generates packages, knows how to make tar archives (with various compression schemes), zip archives, and deb and rpm packages. This is not very useful for us, so I started adding support for hpkg files there. This will make it super-easy to get a package built from any project using cmake: "make package", or "ninja package", depending on the generator you decided to use.

Anyway, I've uploaded an unsupported build in a manually crafted zip file so you can try things for yourself: http://pulkomandy.tk/drop/HaikuWebKit_20131018.zip
I'm waiting for your reports on how well it's working. Note you need a fairly recent build of Haiku (a nightly from today or yesterday should do) as I made some changes to the BHttpRequest class again, to fix some API design issues that led to memory leaks. That class was also added to the Haiku Book.

That's it for this week!

WebKit weekly report #3

Blog post by PulkoMandy on Fri, 2013-10-11 06:53

Hello again, it's time for another report !

I made pretty good progress this week.

The issues I had last week with POST data are fixed. I had removed a non-working piece of code but replaced it with another that was broken in a different way. The problem was the way POST data was added to the http request. Fixing this properly required some changes to the Services Kit API. I removed some classes to make things simpler and introduced a stub for the central BUrlProtocolHandler class, which takes an Url as a parameter and builds a request for it using the appropriate protocol. The BUrlProtocolHttp class was renamed to BHttpRequest, and the API was tweaked to use multiple methods to configure it, instead of a single SetOption(option_name, value) method. This allows seting options with multiple parameters, and is more type-safe.

I started writing some documentation in the Haiku book, for both the Service Kit and the new Network Kit. BeOS already had a Network Kit, and the same API is available in Haiku, but we also have a newer, more flexible and more powerful API. Unfortunately it is undocumented (and unfinished), so there is not a lot of users for it. I hope the documentation will help change that. I'm far from done, however, with just 3 classes available in the Haiku Book (http://api.haiku-os.org).

I finally uploaded WebKit dependencies to the package manager. So, with a recent nightly build, it will be even easier to get started with building WebKit. As I had them around, I also added packages for vim and Caya.

With all this set, I can log in to gmail/google mail again. This means things are working rather well. I even got the web chat to show my online contacts, something that isn't working in the current versions of Web+. I'm also able to log into github, and I found another set of non-working things there (most of them were already broken in older Web+).

I'm now trying to run the tests from http://testsuites.opera.com/cookies. These will help me find and fix some more bugs with cookies management. I already found that cookies set to expire before 1970 would stay forever, because of a bug in the BDateTime class from Support Kit. There are some other issues detected by this test, and it does not make much requests, which makes it easier to debug than an actual production website.

No preview build this week yet, but I'll try to update the WebKit recipe at haikuports so I can cook a package for you to try.

Oh, I forgot to mention our patches to CMake were upstreamed. Version 2.8.13 will have all of them and should build out of the box on Haiku. Also, augiedoggie provided me with a most package, so I can cross that out the TODO list (which is here: https://gist.github.com/pulkomandy/6685664#file-bnetapi-webkit-bugs-md)