WebKit weekly report #48 - More Locale Kit, buildbot and upstreaming efforts

Blog post by PulkoMandy on Fri, 2014-10-10 06:44

Hello world!

Localekit and ICU migration

Last week I wrote the report while I was debugging a deadlock in ICU 53.1. I spent some time debugging this and I found the issue. ICU calls native functions to handle some aspects of timezones (tzset, localtime, and a few others). However on Haiku we implement these functions using ICU. This didn't work too well as ICU tried to lock a lock it was already holding during the initialization of timezone data. Fortunately the calls can easily be disabled in ICU, and that was enough to fix the issue.

While I was working on this, ICU version 54 was released. So I reworked my patches on that new version (fortunately not too much work was needed), and I uploaded the ICU 54 packages to the repositories.

With this updated version, I could continue work on improving the Locale Kit. I added parsing support to BDateFormat and BTimeFormat, which I needed for the data/time picker in WebKit. I finished the implementation of the BMessageFormat class for a first version (this raised a few concerns about design choices in the Locale Kit which may need some rework wrt locking and thread safety). I started making use of BMessageFormat in some places, and soon the translations will provide correct plural support.

I also made some other changes to the APIs, such as the addition of a format generator to ICU. This allows applications to ask things like "I want to know the appropriate date format to show only hours, minutes, and day of week". The Deskbar clock makes use of this now, and should put the fields in the correct order in all locales.

WebKit buildbot and upstreaming effort

You may know that Haiku, Inc. is the owner of 15 Mac Minis donated by the Mozilla fundation and used as build and test bots. They power most of our buildbot infrastructure. One of them was assigned to the WebKit porting effort and this week I worked on setting it up. Since our port isn't upstreamed yet, I have set up my own build master for this, and you can visit it at http://build.pulkomandy.tk . Unfortunately, the build times out because the machine is too slow (we are running Haiku in a virtual machine on rather old hardware there). We will see if the build can be made faster, or a more powerful machine found.

The build bot was a requirement for considering upstreaming our WebKit port. So I started discussing this on the WebKit mailing lists. They decided that it is not time for a complete upstreaming yet, however they suggested we start sending patches to the core parts of WebKit we had to change to get it working on Haiku. There aren't many of those, but the goal is granting one or ideally, two Haiku developpers commit access on WebKit repositories. No one else could review the Haiku specific parts of our code.

I spent part of the week reviewing our changes and submiting the relevant ones. Two small patches were already upstreamed, a tird one is still being discussed, and I will continue this work next week.

Random debugging

Using Haiku as my main system is sometimes a bit of an adventure. I had to fix some annoying issues before I could get work done:

  • Bash was crashing when parsing haikuports recipes with utf-8 characters. This was tracked to a missing bounds check in our widechar type management (wctype).
  • Our buildbot slave package was badly rebuilt after the Python 2.7 update and failed to run. Now it works again.

Bonus material

Here are some tasks I did outside of my paid work time for Haiku this week. They deserve some public announcement.

  • fRiSS, the RSS and ATOM feed reader for Haiku, was fixed for the current network kit APIs and moved to http://github.com/pulkomandy/fRiSS . Fork it and improve it!
  • I started working on the slides for my talk about WebKit at BeGeistert. I hope to meet you all there.
  • I did some work on getting DocumentViewer to run with a current version of MuPDF.
  • I fixed a BeOS compatibility issue in unarchiving BMenuFields, to get VNC Viewer running on Haiku. It seems much slower than the Linux version, however.
  • I added an haikuports recipe for mksh. In these days of shellshock, it may be a good idea to use an actively developped shell with a clean codebase.

... and that closes the report for this week.

Comments

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

Thanks for the report! The buildbot slave shouldn't timeout because the machine is too slow. A crash or a network outage are more plausible IMO.

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

No, there is a timeout of "20 minutes with no output". It can be made longer, but 20 minutes to compile a file does not seem very good anyway. A network outage results in a different error, "exception: slave lost". And it was not a compiler crash, as there was no debug_server window asking what to do on the slave.

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

If they're really slow, and old enough for user upgrades, replacing the hard drive with a SSD one will speed it up quite a bit. It may be wearing on the SSD, but you can also use a SSD swap file to increase the effective memory size.

But 20 minutes to compile a file sounds extreme and likely is some other problem rather than general slowness.

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

I switched the buildbot to Ninja instead of Make, and that seems to have fixed the timeout issue. But overall it's still quite slow. The machine has 2GB of RAM, of which 256M are reserved by the graphics chipset. And we run Haiku in a virtual machine (kvm) with 1GB of RAM. When building WebKit on 2 cores this is barely enough. So first of all we will see about adding more RAM to that machine. This may be good enough for now.

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

Amazing!
Reading your post, seeing the work you do, I was thinking: I should also do again work for Haiku and update my programs, and just when continuing reading, I saw you were doing the DocumentViewer too. That was a big surprise. It's a big surprise that it's still considered useful.
(I stopped programming for haiku since I have the new laptop which doesn't boot Haiku, and since then I'm still tempted from time to time, to install it on a VM on linux)

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

I needed to open a PDF file. The backend in DocumentViewer is much faster than BePDF, however I must admit I don't like the "ribbon" user interface much. It looks like a simple toolbar (without tabs) would be enough as there aren't too much buttons. But it works well anyway and it's nice to have a good PDF viewer available, and as important to me as having a good web browser.

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

Related to build bot:
Haven't you considered using google compute engine?
I have been writing my own http/https/websockets server and clients, and running my bots (related to math, and bitcoin) on the google compute engine, and it works pretty good, and the price is really good.

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

There is PDF.js also.... which could render PDFs in the browser might be a good stability test as well as its a fairly large js application.

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

PDF.js doesn't work yet, unfortunately. Our support is slowly improving but it still needs some work.

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

Sounds like you are running in pure emu mode. Perhaps this can help? http://support.apple.com/kb/TS2744 (Enable HVM in firmware)

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

No, we are using kvm on Linux and it works fine. The problem is not enough RAM is available on the machine, which makes things slower than needed. It takes about 6 hours to buildWebKit in such conditions, which is similar to what you would get natively on similar hardware.

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

ICU developer and wrote-an-app-for-BeBits-14-years-ago-person here.

Is Haiku still working with gcc2? That's probably the main issue with upstreaming all of the patches. Otherwise, make sure we have any other patches. See: http://bugs.icu-project.org/trac/search?q=haiku
Please, file bugs, write our mailing list, etc. Thanks!

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

Hi, Here is the current patch for ICU: http://bb.haikuports.org/haikuports/raw/dc6079731828f6a6db160ee6ef5a9a84... As you can see most of this are problems with inner classes and using declarations not being handled quite right. And the remaining part of the patch (disabling the check for native functions) was not done in a very clean way. There seem to be way to disable the automatic check for tzname and tzset in the configure script. So, would you consider upstreaming some or all of this?

Re: WebKit weekly report #48 - More Locale Kit, buildbot and ...

most definitely .. not the gcc2 part but as discussed, let's clean up the rest, thanks!