WebKit

Hello there,

Well, somehow quiet and regular activty this week. Not too much exciting things, but progress is being made.

I updated WebKit to early december version. This is not the latest one, but the guys at WebKit started using even more C++11 as Visual Studio on Windows finally gets more support for it. So, enter std::chrono and some std::thread stuff. Unfortunately, our version of gcc4 seems to be missing some of these. I'm now closely watching the work of korli and mt to bring us gcc4.8.

I made some long overdue cleanup to the build system, removing some manual (and undocumented) steps and fixing some bugs in the process. The about box will now report the proper WebKit version, once again using the standard CMake scripts to get it, instead of a custom shell script. Moreover, this seems to have fixed Google detecting us as a mobile browser. Some other websites still show that problem, however.

The work on completing the testsuite expectations continues. The progress is slow, as reviewing each of the tests, and looking for possible solutions before marking them all as failing is a very time consuming process. It is also a bit boring and unrewarding, so I'm also spending some time looking at some other things.

I attemted to fix the lockup issue. If you use Web+ you probably have noticed that it oftens becomes unresponsive and seems frozen for a minute or so. I now know what's happening: this triggers when you try to navigate away from a page thet's still loading, or performing an XmlHttpRequest or some other kind of network activity. If the socket is blocked on a connect() call, we have no way to unlock it from there. What WebKit does is it tries to close the connection, and the way it's implemented in our HTTP backend, we have to wait for the network activity to stop, before we can free the connection object. If we free the object without waiting for the thread to terminate first, the thread will crash when the connect call finally times out.

There is a well-known trick to avoid this issue, making the connect non-blocking, and then using select or poll to wait on it. Unfortunately, the Haiku network stack has some bugs that prevent this from working. I'm not yet up to speed on that area of the Haiku code yet, but I'll try to improve the situation.

On the rendering side, I'm also hitting various limitations of the BView API. First of all, the latest build I uploaded last week introduced two new rendering glitches, but fortunately I could get them fixed without reintroducing the box-shadow bug htat led to black rectangles drawing all over the place.

The missing bits in BView currently are support for transform matrices, which are used a lot for drawing SVG and maybe canvas elements ; and support for arbitrary clipping. I had a try at implementing the latter using ClipToPicture, but this lacks antialiasing and is implemented in a suboptimal way, leading to laggy scrolling. It also has problems with the way scrolling is implemented in WebKit, because the clipping is itself clipped (doh!) to the visible part of the view. So, when you try scrolling, it isn't updated and prevent things from drawing if they weren't already visible before scrolling.

I discussed these BView issues with Stippi and there is clearly no way to solve these issues without improving the BView API to add support for arbitrary transformations and clipping shapes. While the initial implementation of this shouldn't be too hard, there may be a few difficulties, for example with the way view coordinates are converted to window or screen ones. It may be a bit tricky to get this all working well.

There are other issues that can be solved only on the WebKit side, for example the history navigation problems. I'm not sure why this is happening, but sometimes, clicking on a link doesn't add it to the navigation history, making the next/previous buttons a bit unreliable to use.

Finally, there are a few known crashes that I'm still tracking. One of them happens when decoding big images or animatd gifs, another one when trying to attach a file to a mail on gmail, and one happens when trying to stop an https connection, and tends to be triggered easily when navigating gmail or github. I'll try to get these crashing issues fixed before I start playing with new features, so we can at least have a stable version of WebKit to do some web browsing, even if some features are still missing.

Hello everyone!

I was a bit bored of messing with the testsuite so this week I looked into "real" issues. The merge of a new WebKit version to trunk last week led to a few more bugreports, and I also looked at some very old ones to see if I could do something. Turned out the answer is yes, and for some of them, the fixes were also rather simple. So let's see what we have:

• Copy&Paste now doesn't add random text after the useful data
• CSS box shadow doesn't leave an ugly artifact next to the box. It still doesn't draw the shadow, however.
• Implemented a minimal version of "complex text layout". This gets "optimize-legibility" text to draw (for example on play.google.com). Most places where text wasn't showing were because of this.
• Disabled scrollbars look as expected, instead of a strange grey rectangle with no buttons around it.
• Web Workers don't crash the browser anymore.

That last issue is caused by a stack alignment problem. What happens is the stack in Haiku is aligned to 4 bytes (32 bits, or the size of a CPU register). This used to be enough. But, when compiling some parts of WebKit, gcc generates code that uses SSE2 instructions like MOVDQA to access data on the stack. This instruction, working with SSE registers, requires 16-byte alignment (128bits). Our stack isn't aligned this way, so the instruction triggers an hardware error and the application crashes. Fortunately, gcc allows specifying that some functions need to check that the stack alignment is correct before running, so I could avoid te issue.

While I was working on all this, AnEvilYak added a feature I was missing in debug_server: it is now possible to tell it to automatically save a debug report instead of asking the user what to do. This is very helpful for the testsuite, as it can now run without me having to click the "save report" button. I had tried to implement this from inside the testsuite app, by catching the signals and trying to run Debugger myself, but that wasn't working well. So, I removed that code and we're now using the new debug_server support, which works much better. The testsuite scripts know to look for the debug reports on the desktop, and will move them to the right place in the test results. With this new way to save reports, we have a disassembly of the exact place where the crash happened, making it a bit easier to understand the errors.

With this out of the way, I went back to analyzing the testsuite results. I made some progress in writing our TestExpectations file, and now the html report is small enough to be browsable in Web+, which makes it much easier to compare the test results. One of the problem I had was that the pixel tests often got out of sync, and the engine would compare the expected output from one test with the actual output of the previous one. I found the problem (we weren't syncing the offscreen view before getting the offscreen bitmap) and also fixed one memory leak in that part of the code.

I now have 6000 more tests to analyze before I finish the TestExpectations file. When this is done, I will have a try at updating WebKit against the trunk again (we're already 1700 revisions behind), and see if that triggers any regressions - or maybe, fixes some of our probems.

Hello world!
You may have heard I'm in for another month!

So, this week I made some more progress with the testsuite system. The major improvements are finished support for the image comparison system I was talking about in last week report. The DumpRenderTree tool now dumps a PNG image of the webpage (this was quite easy to do thanks to the offscreen BView to BBitmap rendering and the translation kit) when requested (not all tests use the feature). Then, another tool called ImageDiff reads two images, and computes the difference. If the difference is big enough, it generates a new image highlighting areas that don't match.

I had to re-add the x86 translators to the x86_gcc2hybrid build so I could make use of them in the gcc4-only WebKit. Fortunately this wasn't too hard either, with just some jamfile tweaks required to make it work again.

The other thing I worked on on DRT side is support for saving crash reports. The problem is, several tests are crashing the DumpRenderTree tool. Without further handling, this results on the testsuite waiting on the debug_server alert asking wether to save a debug report, kill the team, or open Debugger. It is quite annoying to click "kill" an hundred of times as the testsuite runs. So, I'm now catching the signals (SIGSEGV and SIGABRT) from DRT, and from the signal handler I ask Debugger to save a crash report for the current thread. I also modified the python side of the testsuite so it reads reads these reports and move them to the right place. They are now stored with the test results, renamed to the name of the test that made things crash. This is quite useful to see what happened, and execute the crashing tests manually. There's still one problem: Debugger will currently not save any stacktrace, because it does that only for stopped/crashed threads. And since I'm catching the signals, the threads aren't in stopped state anymore when using this trick.

With these fixes out of the way, we're now passing over 52% of the tests. This may look quite bad at first but there are several factors to this, other than failing tests. Most of these come from features that we don't implement yet, such as WebGL, video and audio support, accessibility, battery status (yes, there is a javascript API allowing webpages to query that), web notifications, and so on. There are also a few minor problems like off-by-one size errors that make a lot of the CSS tests fail. And there are strage things like some colors not matching what was requested (a 0,255,0 green gets drawn as 0,254,0 for some reason, making most of the canvas test fail).

The next part of the process is reviewing these results and completing the TestExpectation file. This involves saying things like 'yes, we expect "battery status" tests to fail'. And there are some more fine-grained indications, either "we don't plan to implement it" or "here's the bug in our bugtracker". The testsuite tool can then notify people that some tests passed when they shouldn't have (meaning a bug got fixed somewhere), or on the reverse, some test doesn't pass anymore (so something was broken by your changes). People can update the TestExpectations in the first case, or fix the problems in the second one. There is also a way to tell that we expect some tests to crash the test engine, or to skip them altogether (I use this to skip the one test that crashes app_server).

While reviewing the test results, I also catched and fixed one bug in our WebKit drawing code. This was a corner case of canvas drawing functions, probably not very important, but it would have not been easy to catch without the testsuite. This set of simple pages focusing on a single feature at a time are great to hint you as to where to look for problems.

So, on to analysing the results and maybe fixing some more issues.

On a more interesting note, I finished the support for data: URIs, and added cookie storage for file:// ones. I will provide updated haikuwebkit packages next week including these changes and fixing the easy-to-trigger crashes in the current Web+.