Services Kit

WebKit weekly report #19

Blog post by PulkoMandy on Fri, 2014-02-14 07:37

Hello everyone!

This week I worked on stabilization and small improvements of WebKit. There are a few new features, as well.

The crash with cursors I mentionned last week is fixed. I had forgotten to copy an object in the copy constructor, leading to a double delete. I continued working on the clipping code, and fixed the issues with www.haiku-os.org and a few other websites. But, I can't get it to work with haikuports, Trac, and now gmail is also broken. I don't want to do a release until we have a fix for that.

I also did some more work on supporting css shadows. With the updated clipping code, the ugly black box that was sometimes visible is now gone (it's clipped out). However, we also must draw the shadow itself. I did most of the implementation on WebKit side, but it needs support for a drawing mode (SourceIn) that app_server doesn't handle yet. So, I left that disabled for now, until we can get it to render the expected way.

An easy fix was adding the support for HTML5 "file" API. I just had to turn a compile flag on, and this gives us 10 more points on html5test.com. It also gets imgur.com image upload working.

I merged some changes from WebKit, without much problems. This brings the usual small fixes and cleanup, without too much code breakage this time. There are a lot of code cleanups going on at WebKit, making the codebase simpler and also faster.

I also did some work on the Network Kit HTTP backend. We now support gzip/deflate compression of HTTP data. This make some web pages load much faster, and also fixes issues with some websites serving compressed data even though the browser doesn't advertise support for it. While working on this, another problem in the HTTP code was discovered, there was a possible stack overflow because we were using a gcc extension to C++. This is now fixed, hopefully improving stability of the web browser.

With all those small issues fixed, it's time for another testsuite run. I hope I can find some test that fail because of our remaining clipping problems, as this would help me identify the issues more easily than with a complex web page such as gmail. I'd really like to squash at least the new issues introduced by this clipping change, so we can have a release that I'd qualify as stable. I'm trying to not start work on too much other features until we get this sorted out. Once that release is done, we can resume way on features that need more changes.

Next up on the TODO list: support for affine transformations (stippi added this to the app_server), which will improve SVG rendering a lot and possibly fix some other issues. Support for shadows, and the missing SourceIn composite drawing mode. When we get these two out, we should have much better rendering. There are probably other missing features, but they are yet to be identified.

See you next week!

WebKit weekly report #15

Blog post by PulkoMandy on Fri, 2014-01-17 08:46

Hello again!

No big new changes this week, but a lot of small fixes and improvements.

I reviewed the growing issues list for Web+ on the bug tracker, and fixed several of them. Most of these were small and rather easy to fix bugs (I kept all the harder ones for later). Here is a list with comments, not that the issues were hard to track, but this is also a way to learn a bit more about the WebKit codebase.

Web+ crashed when trying to upload a file to GMail. This was a bug in our BFormDataIO code we use for serializing the form data into the HTTP stream. It missed the case where the first element in the form was a file, and tried to read from it without initializing it first. The FormDataIO class is used so we don't have to put the whole form data in memory in order to send it. It handles each form element one at a time, with special case for files, which are streamed from disk in small chunks, rather than loaded into memory.

Web+ also crashed when trying to decode a huge image. The test case for this is a 93MB JPEG file that expands to 700+MB of pixel data. ShowImage manages to display that, however Web+ tries to do incremental decoding, showing an incomplete view of the image as it is loaded. Our implementation of this is not optimal, as there are at least two copies of the data, one in a BBitmap and one as a raw byte array. For now I fixed the crash, but we abort the decoding and just show a blank page instead. We may want to review the image code to lower the memory use.

I implemented listing local directories in the Services Kit. WebKit has support for rendering directories as part of their FTP handling. Returning files list in one of the formats FTP listings use (there is no standard for this, but a few common formats in use) makes WebKit parse it and generate an HTML page for listing. There are still some problems with encoding (WebKit doesn't seem to expect UTF-8 filenames in those listings), but things should be working now. I also fixed some problems with symlinks in the file:// protocol handler.

Some drawing glitches were fixed (again). We're now in a state where all improvements will require adding support to BView.

There was also a problem with opening links in a new tab from inside a frame. I also added shift+middle click as a way to open a link in a new tab and immediately switch to it (middle click alone opens the tab in the background).

I did several fixes to Cookie management. The most important one is there was a bug in the code for getting cookies for a specific website. A misuse of our StringHash class (this is a simple class that allows using a string as a hash for a hash map) led to memory corruption. We were trying to set the key for the map to a substring of the previous key to implement domain exploration (so a site at www.example.com can access cookies set at example.com - but not for just 'com'). Basically, the HashString freed the old key, then tried to copy characters from it to the new key, using memcpy. This is a classic use-after-free problem, that didn't always create problems in normal run, but was very obvious when running the browser with libroot_debug. Another fix was the proper implementation of CookiesForDOM. This is one of the two methods for accessing cookies. We used the same code as for the access from Javascript, but CookiesForDOM must also include "HttpOnly" cookies. Finally, a third bug was wrongly parsing the expiration date for cookies using the local time zone, whereas they need GMT dates. Depending on your timezone, this lead to cookies expiring too late (you probably didn't notice) or too early, sometimes right in the past. For example, some banking website use short-lived cookies (1 hour or less) as a timeout system. In my GMT+1 timezone, the cookie was expired immediately and I couldn't even access the login screen.

Some fixes went into the SSL support. One case of crashing was fixed, we were deleting the OpenSSL connection context before the network thread had a chance to exit, leading to a crash when leaving an https page before it finished loading. I also started work towards proper support for certificate checking. SSL connections didn't do any checking for certificates, and actually didn't even load the certificate store, making the SSL host authentication useless (you still get the encryption, but you can't make sure you're sending things to the right server). I implemented the Network Kit side of things, but now I must get this exposed in the Services Kit, then in WebKit, and finally add a nice dialog in Web+ asking what to do. Then, I must get the answer back to the network kit and continue or stop the connection with the unsafe host.

On WebKit side, I did a lot of small - but useful! - usability enhancements.

We got the error reporting for non-http connections working again. When trying to open a non-existing file:// URL, you now get a "file not found" message instead of a blank page.

The URL bar now always has an icon (the default is a little globe), to avoid the URL jumping to the right when the favicon gets shown. I also fixed some glitch pixels below the text in that bar, when using small font sizes.

I reworked some of the bookmark loading code. Now, bookmarks load in the current window, instead of the first window they can find in the workspace. If you open several of them at the same time, it works as expected. There was a race condition leading Web+ to try opening several bookmarks in the same tab, with of course only the last one showing up. Another problem was it was not possible to use symlinks in the bookmarks folder, as the BNavMenu we use for bookmarks wasn't traversing them. This now works as expected.

The search page in Web+ is now configurable. This means you can switch to goodsearch.com and help raise some money for Haiku while searching! Or, you can use the local version of Google or whatever search engine you prefer. The bug that made us unable to search for UTF-8 strings was also fixed, so you don't have to search in english anymore. And, there was also some progress with IDN domains, but the complete fix for this will have to wait for the next update to the WebKit package.

So, what's next? I will continue working on better SSL support, as this is currently set as an alpha blocker. I also plan to have a look at doing a bookmark bar. I tried doing this as a BMenuBar + BNavMenu, but these classes aren't designed for multiple inheritance, so I have some refactoring to do there. Or maybe I should go with another approach.

The "network lock-up" bug and missing BView features are also still fairly high on the TODO list, but these will need more time as I'm not as familiar with the code in these areas.

I didn't do this for some time, but let's also talk about non-working-hours time I also spent on Haiku. I did some Haikuporter recipes for XRick, OpenTTD, and a few other games. I also finally made a recipe for libusb, and others have used this to compile libftdi and avrdude. This isn't quite working yet, but I hope someone gets it going so I can finally do some hardware hacking on Haiku (did I hear blinkenlights?).

See you next week!

WebKit weekly report #11

Blog post by PulkoMandy on Fri, 2013-12-13 07:41

Hello everyone.

Some progress again this week.

First of all, I finally updated the haikuwebkit package, for both gcc2hybrid and gcc4. This means you can download things with WebPositive again, and all the bugfixes since the last release are in as well.

On the testsuite side, I got the crash-report feature to more or less work thanks to help from Rene and Ingo. There was one improvement to Debugger to allow extracting the stack trace when using the --save-report option on a thread that's not (yet) crashed. I tried various ways of plugging this into the testsuite, but it turned out catching the signals is the right thing to do. I also had to override debugger() in the testsuite system to call abort, and raise the expected signal, instead of dropping directly into debugger.

Most of the crashes reports are now working, but sometimes Debugger fails exiting and stays waiting for something. I have to manually kill it to get the testsuite to continue. Well, the cases where it works are already quite helpful, helping me to catch some bugs in my image saving code for the "pixel test" stuff, and also improving the logic to decide when to run the next test - we were sometimes doing this before the current one had finished running its javascript code.

Writing of the TestExpectations file continues, I have enough entries in it now to filter some of the tests out and make the final report useable. WebKit reports tests that didn't run as expected (crashes, things that failed, and things that passed when we told in the TestExpectations that they should fail). With 15000 tests not passing, the WebPage is quite huge, but now small enough Web+ manages to render it. I'm exploring the results and either adding rules to TestExpectations, or fixing the bugs when I've also seen them create issues on some web pages.

One of the fixes I did is support for gradients (in CSS and SVG). We had some code written, but apparently not too well tested. It was using the wrong range for the gradient stops (our API expects 0 to 255, and the code used 0 to 1). Also, the gradient semantics are a bit different, in our gradients, if there is no stop at the start or end of the gradient, they will draw random colors. CSS expect the first and last stops to extend to the gradient edge.

I already identified some other problems in both the testsuite and WebKit code itself. I'm not going to fix all the issues, so my goal is to write as much of the TestExpectations as I can, trying to describe why each of the tests fail and grouping them together. The goal is to get everything to run "as expected" (even if that means actually failing). This way, the impact of any change can easily be identified by the number of tests fixed or broken.

I'll still have a look at the crashing tests. If we can get those fixed, the testsuite will run more smoothly and we can expect the actual browser to be more stable as well.

WebKit weekly report #9

Blog post by PulkoMandy on Fri, 2013-11-29 09:26

Hello world!

This week I got the testsuite running. The fixed build of gcc I was talking about last week worked, and I can now link a working DumpRenderTree executable. The first test run wasn't very good, with DRT crashing quite often, some javascript alerts popping on my screen, and ended after running about 9000 tests in an app_server freeze.

I'm adding some of the missing features to DumpRenderTree: alerts are now dumped to the standard output like other ports do, so they can be compared as part of the testing process. I skipped the test that crashes app_server (the testsuite system has support for skipping tests, and a few other things). I also did some smaller changes like setting the default font size to 16px for the tests (the browser still defaults to 14px, which is a bit unusual, but nothing in the css spec enforces a default size of 16px) ; and changing the JS console messages format for DRT so they match other ports (the main change is to not include the file name in the message, this way the test gives the same result if run from a different directory).

I'm now working on support for PNG dump of web pages in DRT. Some tests are testing things that aren't easily seen in a text dump, such as CSS gradients. To test this, there are two systems used in WebKit testsuite: the first one is comparing the rendering with a reference PNG. This works well, but there are often minor changes to the rendering, making it hard to keep the test result up to date with the browser. The PNG files are also often platform specific, due to different antialiasing or drawing algorithms, font rasterizers, and the like. So, a better system called "reftests" is used. The idea is the same, but instead of comparing with a fixed PNG image, DRT generate PNGs of two different pages, and checks that they render exactly the same. When there are two ways of rendering the same thing, this works, and follows the browser changes automatically.

Our results are currently not very good, with a lot of failing tests. I expect the support for pixel-tests and reftests to improve this a bit. Note that the goal isn't to reach 100% success: none of the other platforms do that. Instead, there is a "TestExpectations" file which tells which test should fail or succeed. The actual results are compared with the file. When fixing a bug, you can then easily verify that you don't introduce new test failures, and possibly that you make existing tests pass when they failed before.

The WebKit project requires each commit to either introduce a new test, or fix an existing one, unless youhave very good reasons (sometimes it isn't possible to test something, or the test would take too long to run).

In order to run the complete testsuite, I will also need a running HTTP server. The current scripts support either lighttpd or apache. One option is porting one of these, the other is modifying the scripts to accept something else, such as PoorMan or Cherokee, for which we already have an Haikuports recipe.

With the testsuite running, we will now have a way to check that future work on our WebKit port don't break things that were already working, like it happened with the version bundled in current nightlies - I bet you noticed the crashes.

I didn't work only on the testsuite, and also did some small improvements on the network layer. I'm adding support for data: URIs, and improving the MIME type identification system, which is currently restricted to HTTP. I also started looking at support for gzip compression of HTTP data. This is suppoed to be optional, but some websites actually require it. It shouldn't be too hard to implement and could bring us an even faster browsing experience, anyway.

WebKit weekly report #7

Blog post by PulkoMandy on Fri, 2013-11-15 08:39

Hello world!

This week I reached a major milestone in my work, as I'm done merging all WebKit commits all the way to november 2013. HaikuLauncher is still running fine, and I will make the small required changes in WebPositive so it works fine again. Expect an updated WebPositive in trunk very soon now.

This week merges were as boring as the previous week ones: addition of the NiX port, removal of the Qt one (next version of Qt will go with Blink), replacement of a lot of never-null pointers to use references instead, rename of the KURL class to URL, and some API changes. WebKit has started using some C++11 features to help with cleaner and faster code.

I've also started looking at the test system. WebKit uses a tool called DumpRenderTree that will produce a text representation of the render tree. This allows running tests that are not dependant on any platform. There are also 'pixel tests', where WebKit renders a web page, and compares the results with a PNG file. And finally, there are some javascript-only tests for JavaScriptCore.

I've updated our version of DumpRenderTree (since WebKit has no common API on different platforms, each platform must provide an implementation for it). I've got it to run in the mode where you give it a single test on the command line, which is good for manual testing, but the testsuite instead feeds test names on the standard input, and this somehow crashes our version currently. I'll debug that and run the 33000+ tests to get a better idea of where we are.

There will be some more work to do before we can run the complete testsuite, for example tests involving HTTP require an apache web server port (anyone has a recipe for this ?), and many methods in the test frameworks aren't implemented yet. These allow the tests to access some browser settings from Javascript, for example they can disable cookies, change cache policies, and so on. This allows testing of WebKit with all possible settings.

I want to get the testsuite running to help with future work on adding new features and the merges we'll have to do. This will help getting a better idea of the status of our port, and make sure we don't break things when merging new versions of WebKit in. Also, the tests are usually simple and the failing one will point to the features we are missing, making the port much easier to fix and improve.

WebKit weekly report #5

Blog post by PulkoMandy on Fri, 2013-10-25 06:50

Hello there !

Well, not so much progress on WebKit this week. I spent most of the time working on CMake code to get it to generate hpkg files. I got something that works well enough to link Web+ to it, so I can test things with the actual browser instead of HaikuLauncher.
Today I added the cookie jar persistence, so Web+ remembers all the cookies when you exit and relaunch it. I also started working on HTTP authentication. These are the two features I couldn't test with HaikuLauncher, as it lacks some code for them (saving cookies on exit, and showing the HTTP authentication password prompt window).

I also implemented the protocol handler for file: URLs, so now I can browse the Be Book and Web+ default home page works as well.

With the HTTP authentication fixed, I think I will be on par with the current code, so I could merge this into Haiku now for you all to use. However, there are some new rendering glitches that maybe I should fix first. What do you think?

WebKit weekly report #4

Blog post by PulkoMandy on Fri, 2013-10-18 07:07

Time for a report again !

So, over last week-end and monday morning I finally got Ninja working. I already said some words about it, Ninja is meant to replace Make for building projects. It has less features, because it is designed to run files that are generated, rather than hand-written. In my case, CMake generate the Ninja failes. I had problems building Ninja that turned out to be abug in our Python port. I didn't fix it, but I found a way to work around it.

The use for all this buildsystem work? Well, we're now using one of WebKit standard build systems (CMake), and using it in a way that's fast! CMake can also generate makefiles, but running these would take about a minute just to figure out which files to recompile. With Ninja, this is done in a matter of seconds, because the simpler rules are faster to parse. Ninja was actually written just for that purpose. One minute may not seem much when the full build of WebKit takes about an hour, but when I'm working on just one or two files to track down a bug, the edit/compile/test cycle is now much faster.

So, with this out of the way I was able to more easily try out things and fix the issues as I found them. I mentionned the Opera cookie testsuite in last week report : most of the tests are now passing, and we even get better results than the old Curl backend in some places. The remaining issues are rather minor ones, but I can fix them if you can find a website where they actually matter.

In one of the blog post comments someone mentioned that Windows Live Mail (or Outlook, as it's now called) didn't allow to display messages. I tried to do that with HaikuLauncher and the new code and noticed I couldn't even login to the website. Getting this to work was a bit tricky, leading to fixes in 3 different places: settings cookies from Javascript code, proper handling of URL redirections, and telling WebKit that "aspx" files served without a mimetype are web pages.

I already told you about the cookies, so here are some more details about the two other issues: the URL redirection problem happened when an HTTP server gave us a 302 answer (that is, a redirection) with a relative URL. The HTTP specification says an URL should be used, most likely starting with "http://". But the URL specification was later replaced with the URI one, which allows URIs without that part, for example "/path/to/page.aspx?param=foo". Our BUrl class wasn't able to understand such things and was desperately looking for the "//" part, leading to an endless loop. I replaced the homegrown parser with a regular expressionbased one, using the regex that is kindly provided inthe RFC for URIs. I had to fix our RegExp class in the shared kit (this is a collection of internal classes that are not, yet, part of the official API). So, I had to fix the RegExp class as it didn't handle optional capture groups in the regular expression properly. This includes things like: the URI may, or may not start with "scheme:", then, maybe there will be an authority part, of the form "//authority", then a path, which is anything following the authority and up to th first ? or # or the end of the URI, and then there is the query and fragment, and they are also optional. RegExp allow to write this in a more compact and computer understandable format and get the parts extracted to separate strings.

As for the "aspx" file handling, the HTTP protocol says that the server should give a "content-type" header with the MIME type of the file it's going to send. This fits well with the use of MIME types in Haiku and would be very nice. However, Microsoft servers for login.live.com sometimes don't include the header. Web+ used to assume the file was to be displayed on screen in that case. You may already have seen it trying to render a video file as plain text, as a huge page full of strange characters. Well, that issue was fixed, and if in doubt, it will now download the file to disk instead. A list of well-known file extensions is used to attempt a guess when the "content-type" is missing. I added aspx to that list and mapped it to text/html, so, the URLs ending in aspx will now be shown on screen.

I had a run of testing after all these fixes and things are getting much better. It helped fix some more issues in github and gist, and I could also browse some documents in Google Drive. I think we still have some problems with file upload, however, I'll have to look into that. Uploading video to youtube, as someone pointed out in the forums, also fails.

I caught a crash when opening a new window in HaikuLauncher, but I'm not sure if that would be a problem for Web+. Also, I wanted to work on HTTP authentication (the login/password prompt) and this is not available in HaikuLauncher, so a working Web+ will be needed. To get there I wanted to build a package for WebKit. To do this, I first tried writing an haikuporter recipe. I didn't get very good results with this. First,it means working separately from my development git checkout, so it takes twice the sapce on disk (and we're talking multiple gigabytes) and it takes forever to checkout. Second, there's no easy way to use distcc with haikuporter, so things are keeping my CPU very busy for a long time. And since this isn't my development tree, there are a lot of things to rebuild.

So, I decided to try a different solution: using CMake built-in support for generating packages. I tried that yesterday and found out that it wasn't working. I tried on a very small test project, and with help from the CMake IRC channel I found the issue: our CMake port is lacking some features that are available in the Linux version, and not needed on Windows. This confuses the Ninja generator which isn't ready for that case. This is a rather technical issue related to the way executables and libraries are linked together. This is different in the tests (the lib and executable sit together in a directory) and installed version (they are installed somewhere on the filesystem and the OS, more precisely the runtime_loader) must bind them together. On Linux, CMake makes up for that by rewriting part of the executables when installing or packaging them. This isn't enabled in the Haiku version, so it will instead try to link executables again when installing them. However, the Ninja generator is skipping the link step (it was never tested in that case) and tries to install or package non-existing files. This shouldn't be too hard to get working just like on Linux, more news next week !

There is another missing step to make this complete: CPack, the CMake component that generates packages, knows how to make tar archives (with various compression schemes), zip archives, and deb and rpm packages. This is not very useful for us, so I started adding support for hpkg files there. This will make it super-easy to get a package built from any project using cmake: "make package", or "ninja package", depending on the generator you decided to use.

Anyway, I've uploaded an unsupported build in a manually crafted zip file so you can try things for yourself: http://pulkomandy.tk/drop/HaikuWebKit_20131018.zip
I'm waiting for your reports on how well it's working. Note you need a fairly recent build of Haiku (a nightly from today or yesterday should do) as I made some changes to the BHttpRequest class again, to fix some API design issues that led to memory leaks. That class was also added to the Haiku Book.

That's it for this week!