Blogs

Hello there!

So, this wasn't a fun week. Last week I had finished merging all commits from WebKit main repo. I got Web+ to run fairly well with these changes and I wanted to merge this into Haiku. However, while this works for the gcc2-hybrid version, I was not able to build a package for gcc4 and gcc4h. As a result, there were no nightlies published (the script wants all the architectures to work for a given revision before it gets published).

When trying to build the gcc4 package, I ran into several different problems: haikuporter triggering a KDL when running webkit "make install", the CMake script I used to build the gcc2h packages not working as it should on gcc4, and a few more. Since I don't usually run a gcc4 install, I had to do all this from a temporary install on an SD card, which is very slow at compiling, or, actually, at doing anything. Even a simple rm -r can take a minute or so to complete.

Meanwhile, I'm making some (slow) progress on getting the testsuite to run. For this to run I have to fix a tool called DumpRenderTree. This is a small executable wrapped around WebKit that renders a web page, and dumps to the console either a tree structure of it, or a plain-text dump of the page contents. It should also generate PNGs of the pages in some case. The testsuite runs DumpRenderTree on a set of pages and compares the results with "expected" files, and makes sure everything matches.

The problem I have is DumpRenderTree is always dumping the render tree instead of the plain text dump the "expected" files have. To switch between the modes, DumpRenderTree expects the page to call Javascript code to configure the output. The javascript interface can also do some other useful things, like clearing cookies, or otherwise interacting with the tool.

Since the WebKit API is specific to each port, the DumpRenderTree tool must be rewritten to use each of these APIs. I updated our version, modelling it on the EFL one. I now have something that should work, but somehow the JS callbacks aren't working.

I tried building WebKit with all assertions enabled to mak sure everything was right. I had to fix some minor harmless things to get HaikuLauncher to run, but DumpRenderTree would hit asserts over and over again. I tried disabling one, only to hit the next. Something looked strange, with methods as basic as isMainThread() failing (this is just comparing a TLS value with a global). I added some traces, and noticed the TLS was ok, but the global was changing value. I spent some time trying to look for something overwriting it, but then I noticed it was also changing address. This shouldn't be possible. Well, that hinted me to the actual cause for the problem: some files are getting linked twice into DumpRenderTree. what happens is we first build a big "libWebKit" library. This exports only our WebKit API, and inside it, there is also all the core code (WTF, JavaScriptCore and WebCore), the cross-platform parts of WebKit. The idea is we want the library to only export our official API, and not let apps mess with the internals.

However, DumpRenderTree does need to mess with the internals. It adds custom JS APIs for the app to access, reads internal data to extract frame content as text, and takes some shotcuts calling WebCore directly to do things in a way that works on all platforms. So, DumpRenderTree links not only our libWebKit, but also the WTF, JSC and WebCore libs again. Since we build these as static libs, we end up with code both in DRT itself and in libWebKit. Things then start breaking apart because these can't see each other. We get duplicated static variables, initializations called on one side that set variables the other side can't see, etc.

So, I moved back to the shared mode. This is what we used to do when WebKit was built with Jam. In this mode, WTF, JSC and WebCore are built as shared libraries, and libwebKit only links against them instead of embedding them. This allows things to exist only once in the running executable and DRT to access all the internal as it should.

But... that didn't work. I hit a command-line length limitation.

In the Jam build, we used to build multiple small static libraries (webcore1.a, webcore2.a, etc) and then link them together in a big shared library. This allows us to send a smaller set of files to ar to create the static libs and stay under the maximal command-line length. CMake uses a different trick to keep command lines short: it uses response files. These were added to gcc and the binutils to solve the very same problem on Windows. The idea is to put the command-line args into a file, and give it to the tool on the command line using te @file syntax. This works well with ar (when building the static libs), but our gcc will unfold the thing before calling some of the internal tools, and we hit the problem again when invoking collect2 (this is an internal executable used by gcc) or ld. It seems this is because gcc doesn't know that the linker it is calling is compatible with response files, so it tries unfolding the command line to be more compatible.

I'm now rebuilding gcc with the --with-gnu-ld configure option. This should force it into using response files when invoking the linker, and finally get our libraries built. Not this will change the ABI again and the haikuwebkit packages for both gcc2hybrid and gcc4 will have to be rebuilt. I'm not sure yet if we should do this only for the tests, or if we should also do it for the distributed libraries. It makes sense to distribute the ones we have tested, but the single-library solution should be faster (less symbol lookup) and use less memory at runtime.

If the gcc rebuild doesn't work, I'll have to use the same trick we did with Jam, making invasive changes to the cmake build. This would make it more difficult to merge with the official WebKit sources in the future.

Also, some sidenotes:

• My patches to CD, IM, and IUP have been upstreamed. We now have a portable UI toolkit that uses native widgets
• Kallisti5 is working on providing a server that runs Haiku. We will use this once he is done setting up to run the CMake testsuite/nightly builds, and run the WebKit tests. This is a step towards getting parts of our port merged upstream again.

From mid-June to early November, I have spent 320 more hours working on package management. After having worked on the bootstrap support in HaikuPorter (which Ingo has already mentioned in his blog entry), I have spent some time reworking the way how Perl and Pythong organize their modules: Scripting languages with module support usually expect to be able to build modules from source and install them somewhere into the hierarchy of the scripting language. In our case, they can no longer just put the built (site-specific) modules into the "standard" folder, as that one is read-only. So the configuration of Perl and Python had to be changed, such that modules built from source are being installed correctly into a writable folder (in the non-packaged hierarchy). Perl already supports the notion of a vendor-modules folder, where packaged Perl modules will be installed (an example of such a beast are the perl modules contained in the git package). That idea has been extended to Python (such that it will automatically pick up the python modules provided by the mercurial package). Separating packaged modules from local (built from source) modules is an idea that should be followed by all (not only scripting) languages ported to Haiku.
The updated perl, python, git, mercurial and scons packages that are the result from that work have not yet been published, but I will do that during the next couple of days.

The package management branches of both haikuporter and haikuports have been merged into the respective master, i.e. all porting work done in there will produce packages. Several new 'testing-' branches will be added to the haikuports repository soon in order to be able to separate work on the bleeding edge from the work that tries to stabilize a given set of packages for a release.

Finally, I have started with work on the infrastructure required for automatic building of repositories and (at a later stage) packages. The first effect of this is that whenever a developer changes the version of a package used by Haiku's build system, the respective repository will be created automatically. Those repositories are being published to http://packages.haiku-os.org/haikuports/master, from where pkgman will download packages.

Summarizing those months of work: I think we have managed to put most of the work behind us, but we haven't reached the goal just yet. There's still more work to do and I personally will return to continuing that work in my spare time.

Hello world!

Sorry for no report last week, I was not in front of the computer on Friday.

Anyway, I got the HTTP authentication working last week. This was the last missing feature in the Services Kit version of WebKit when compared to the current cURL one. The next step is to fix the new rendering bugs.

The rendering side of things is mostly built into WebKit, so I didn't want to fix it on the old version we are still running. So, I have started merging WebKit changes all the way to the current revision. Unfortunately, our WebKit repository wasn't created the right way, and the commit hashes for WebKit commits didn't match the ones for the official WebKit repo. I had to create a new repository, and manually match the commits and play with git rebase, merge and cherry-pick to rewrite all ourwork against the official WebKit commits. This took some time, as there are 120000 WebKit commits in our repository, plus our own changes.

I have removed the old repository and uploaded a new one. My work happens in the 'rebased' branch, which is the default one.

With the repository rebased onto the official WebKit, it is now possible to use git merge to merge commits directly from the official WebKit mirror.. Our port was about 2 years behind, in WebKit world that's more than 30000 commits. I didn't merge those all at once, instead I'm working with ranges of about 2000 to 3000 commits. While one range is merging and compiling, I can review the commits for the following one, and take note of the important changes. This helps a lot knowing what happened when I need to do a merge or fix the build.

I have merged commits up to april 2013 so far. This is the point where the Chromium port gets removed, and later on the wxWidgets port. The GTK one switches to CMake as a build system (the same we use now). These are the changes that creates more conflicts with us, so I'm going to merge this part in smaller chunks (I tried a big one and got a build error I didn't know how to solve).

Following the removal of the Chromium port, WebKit entered a phase of cleanup and simplification. This includes removing Chromium-specific stuff, making the code simpler in some places, and also using some C++11 features that help with detecting errors at compile time.

One important change is the introduction of "platform strategies". Before that, porting webKit involved implementing many classes where the header file was shared accross platforms, but not the C++ sources. This led to a lot of #ifdefs-guarded platform-specific code all over the place. The platformStrategy class makes all the platform specific code located in the same place, and the shared code go elsewhere. Our port will be much easier to keep working this way.

So, which new features are brought in by this huge merge ? Well, none so far. Most of the features in WebKit are optional, and to keep things easier, we disabled most of them. As features are sometimes removed when no port uses them, I'm forst going to merge all the changes, then see what can easily be enabled. Some of the features involve only multi-platfomr code, but others require some porting.

Anyway, the up-to-date WebKit is faster, smaller, and has much less bugs, which are already very good things. When I'm done with the merge I will have a look at running the WebKit tests, as I'm pretty sure this will help catch some bugs and at least give us a better idea of the work left to do. Running a testsuite is now much easier thanks to the use of CMake, where the guis working on the EFL port made some changes to the build system to help with setting this up. I will also test all the websites I have listed on my TODO/check list, to make sure things that used to work are still working.

I hope to have something in shape for a merge next week or the one after that. The remaining part of the month will be dedicated to enabling some of the extra features, but with only 2 or 3 weeks left I will not be able to work on the most complex ones. Haiku, Inc currently can't fund another month of development, so it looks like I'll have to stop there, unless there are more people donating money this month.