webpositive

WebKit weekly report #19

Blog post by PulkoMandy on Fri, 2014-02-14 07:37

Hello everyone!

This week I worked on stabilization and small improvements of WebKit. There are a few new features, as well.

The crash with cursors I mentionned last week is fixed. I had forgotten to copy an object in the copy constructor, leading to a double delete. I continued working on the clipping code, and fixed the issues with www.haiku-os.org and a few other websites. But, I can't get it to work with haikuports, Trac, and now gmail is also broken. I don't want to do a release until we have a fix for that.

I also did some more work on supporting css shadows. With the updated clipping code, the ugly black box that was sometimes visible is now gone (it's clipped out). However, we also must draw the shadow itself. I did most of the implementation on WebKit side, but it needs support for a drawing mode (SourceIn) that app_server doesn't handle yet. So, I left that disabled for now, until we can get it to render the expected way.

An easy fix was adding the support for HTML5 "file" API. I just had to turn a compile flag on, and this gives us 10 more points on html5test.com. It also gets imgur.com image upload working.

I merged some changes from WebKit, without much problems. This brings the usual small fixes and cleanup, without too much code breakage this time. There are a lot of code cleanups going on at WebKit, making the codebase simpler and also faster.

I also did some work on the Network Kit HTTP backend. We now support gzip/deflate compression of HTTP data. This make some web pages load much faster, and also fixes issues with some websites serving compressed data even though the browser doesn't advertise support for it. While working on this, another problem in the HTTP code was discovered, there was a possible stack overflow because we were using a gcc extension to C++. This is now fixed, hopefully improving stability of the web browser.

With all those small issues fixed, it's time for another testsuite run. I hope I can find some test that fail because of our remaining clipping problems, as this would help me identify the issues more easily than with a complex web page such as gmail. I'd really like to squash at least the new issues introduced by this clipping change, so we can have a release that I'd qualify as stable. I'm trying to not start work on too much other features until we get this sorted out. Once that release is done, we can resume way on features that need more changes.

Next up on the TODO list: support for affine transformations (stippi added this to the app_server), which will improve SVG rendering a lot and possibly fix some other issues. Support for shadows, and the missing SourceIn composite drawing mode. When we get these two out, we should have much better rendering. There are probably other missing features, but they are yet to be identified.

See you next week!

WebKit weekly report #15

Blog post by PulkoMandy on Fri, 2014-01-17 08:46

Hello again!

No big new changes this week, but a lot of small fixes and improvements.

I reviewed the growing issues list for Web+ on the bug tracker, and fixed several of them. Most of these were small and rather easy to fix bugs (I kept all the harder ones for later). Here is a list with comments, not that the issues were hard to track, but this is also a way to learn a bit more about the WebKit codebase.

Web+ crashed when trying to upload a file to GMail. This was a bug in our BFormDataIO code we use for serializing the form data into the HTTP stream. It missed the case where the first element in the form was a file, and tried to read from it without initializing it first. The FormDataIO class is used so we don't have to put the whole form data in memory in order to send it. It handles each form element one at a time, with special case for files, which are streamed from disk in small chunks, rather than loaded into memory.

Web+ also crashed when trying to decode a huge image. The test case for this is a 93MB JPEG file that expands to 700+MB of pixel data. ShowImage manages to display that, however Web+ tries to do incremental decoding, showing an incomplete view of the image as it is loaded. Our implementation of this is not optimal, as there are at least two copies of the data, one in a BBitmap and one as a raw byte array. For now I fixed the crash, but we abort the decoding and just show a blank page instead. We may want to review the image code to lower the memory use.

I implemented listing local directories in the Services Kit. WebKit has support for rendering directories as part of their FTP handling. Returning files list in one of the formats FTP listings use (there is no standard for this, but a few common formats in use) makes WebKit parse it and generate an HTML page for listing. There are still some problems with encoding (WebKit doesn't seem to expect UTF-8 filenames in those listings), but things should be working now. I also fixed some problems with symlinks in the file:// protocol handler.

Some drawing glitches were fixed (again). We're now in a state where all improvements will require adding support to BView.

There was also a problem with opening links in a new tab from inside a frame. I also added shift+middle click as a way to open a link in a new tab and immediately switch to it (middle click alone opens the tab in the background).

I did several fixes to Cookie management. The most important one is there was a bug in the code for getting cookies for a specific website. A misuse of our StringHash class (this is a simple class that allows using a string as a hash for a hash map) led to memory corruption. We were trying to set the key for the map to a substring of the previous key to implement domain exploration (so a site at www.example.com can access cookies set at example.com - but not for just 'com'). Basically, the HashString freed the old key, then tried to copy characters from it to the new key, using memcpy. This is a classic use-after-free problem, that didn't always create problems in normal run, but was very obvious when running the browser with libroot_debug. Another fix was the proper implementation of CookiesForDOM. This is one of the two methods for accessing cookies. We used the same code as for the access from Javascript, but CookiesForDOM must also include "HttpOnly" cookies. Finally, a third bug was wrongly parsing the expiration date for cookies using the local time zone, whereas they need GMT dates. Depending on your timezone, this lead to cookies expiring too late (you probably didn't notice) or too early, sometimes right in the past. For example, some banking website use short-lived cookies (1 hour or less) as a timeout system. In my GMT+1 timezone, the cookie was expired immediately and I couldn't even access the login screen.

Some fixes went into the SSL support. One case of crashing was fixed, we were deleting the OpenSSL connection context before the network thread had a chance to exit, leading to a crash when leaving an https page before it finished loading. I also started work towards proper support for certificate checking. SSL connections didn't do any checking for certificates, and actually didn't even load the certificate store, making the SSL host authentication useless (you still get the encryption, but you can't make sure you're sending things to the right server). I implemented the Network Kit side of things, but now I must get this exposed in the Services Kit, then in WebKit, and finally add a nice dialog in Web+ asking what to do. Then, I must get the answer back to the network kit and continue or stop the connection with the unsafe host.

On WebKit side, I did a lot of small - but useful! - usability enhancements.

We got the error reporting for non-http connections working again. When trying to open a non-existing file:// URL, you now get a "file not found" message instead of a blank page.

The URL bar now always has an icon (the default is a little globe), to avoid the URL jumping to the right when the favicon gets shown. I also fixed some glitch pixels below the text in that bar, when using small font sizes.

I reworked some of the bookmark loading code. Now, bookmarks load in the current window, instead of the first window they can find in the workspace. If you open several of them at the same time, it works as expected. There was a race condition leading Web+ to try opening several bookmarks in the same tab, with of course only the last one showing up. Another problem was it was not possible to use symlinks in the bookmarks folder, as the BNavMenu we use for bookmarks wasn't traversing them. This now works as expected.

The search page in Web+ is now configurable. This means you can switch to goodsearch.com and help raise some money for Haiku while searching! Or, you can use the local version of Google or whatever search engine you prefer. The bug that made us unable to search for UTF-8 strings was also fixed, so you don't have to search in english anymore. And, there was also some progress with IDN domains, but the complete fix for this will have to wait for the next update to the WebKit package.

So, what's next? I will continue working on better SSL support, as this is currently set as an alpha blocker. I also plan to have a look at doing a bookmark bar. I tried doing this as a BMenuBar + BNavMenu, but these classes aren't designed for multiple inheritance, so I have some refactoring to do there. Or maybe I should go with another approach.

The "network lock-up" bug and missing BView features are also still fairly high on the TODO list, but these will need more time as I'm not as familiar with the code in these areas.

I didn't do this for some time, but let's also talk about non-working-hours time I also spent on Haiku. I did some Haikuporter recipes for XRick, OpenTTD, and a few other games. I also finally made a recipe for libusb, and others have used this to compile libftdi and avrdude. This isn't quite working yet, but I hope someone gets it going so I can finally do some hardware hacking on Haiku (did I hear blinkenlights?).

See you next week!

WebKit weekly report #14

Blog post by PulkoMandy on Fri, 2014-01-10 08:34

Hello there,

Well, somehow quiet and regular activty this week. Not too much exciting things, but progress is being made.

I updated WebKit to early december version. This is not the latest one, but the guys at WebKit started using even more C++11 as Visual Studio on Windows finally gets more support for it. So, enter std::chrono and some std::thread stuff. Unfortunately, our version of gcc4 seems to be missing some of these. I'm now closely watching the work of korli and mt to bring us gcc4.8.

I made some long overdue cleanup to the build system, removing some manual (and undocumented) steps and fixing some bugs in the process. The about box will now report the proper WebKit version, once again using the standard CMake scripts to get it, instead of a custom shell script. Moreover, this seems to have fixed Google detecting us as a mobile browser. Some other websites still show that problem, however.

The work on completing the testsuite expectations continues. The progress is slow, as reviewing each of the tests, and looking for possible solutions before marking them all as failing is a very time consuming process. It is also a bit boring and unrewarding, so I'm also spending some time looking at some other things.

I attemted to fix the lockup issue. If you use Web+ you probably have noticed that it oftens becomes unresponsive and seems frozen for a minute or so. I now know what's happening: this triggers when you try to navigate away from a page thet's still loading, or performing an XmlHttpRequest or some other kind of network activity. If the socket is blocked on a connect() call, we have no way to unlock it from there. What WebKit does is it tries to close the connection, and the way it's implemented in our HTTP backend, we have to wait for the network activity to stop, before we can free the connection object. If we free the object without waiting for the thread to terminate first, the thread will crash when the connect call finally times out.

There is a well-known trick to avoid this issue, making the connect non-blocking, and then using select or poll to wait on it. Unfortunately, the Haiku network stack has some bugs that prevent this from working. I'm not yet up to speed on that area of the Haiku code yet, but I'll try to improve the situation.

On the rendering side, I'm also hitting various limitations of the BView API. First of all, the latest build I uploaded last week introduced two new rendering glitches, but fortunately I could get them fixed without reintroducing the box-shadow bug htat led to black rectangles drawing all over the place.

The missing bits in BView currently are support for transform matrices, which are used a lot for drawing SVG and maybe canvas elements ; and support for arbitrary clipping. I had a try at implementing the latter using ClipToPicture, but this lacks antialiasing and is implemented in a suboptimal way, leading to laggy scrolling. It also has problems with the way scrolling is implemented in WebKit, because the clipping is itself clipped (doh!) to the visible part of the view. So, when you try scrolling, it isn't updated and prevent things from drawing if they weren't already visible before scrolling.

I discussed these BView issues with Stippi and there is clearly no way to solve these issues without improving the BView API to add support for arbitrary transformations and clipping shapes. While the initial implementation of this shouldn't be too hard, there may be a few difficulties, for example with the way view coordinates are converted to window or screen ones. It may be a bit tricky to get this all working well.

There are other issues that can be solved only on the WebKit side, for example the history navigation problems. I'm not sure why this is happening, but sometimes, clicking on a link doesn't add it to the navigation history, making the next/previous buttons a bit unreliable to use.

Finally, there are a few known crashes that I'm still tracking. One of them happens when decoding big images or animatd gifs, another one when trying to attach a file to a mail on gmail, and one happens when trying to stop an https connection, and tends to be triggered easily when navigating gmail or github. I'll try to get these crashing issues fixed before I start playing with new features, so we can at least have a stable version of WebKit to do some web browsing, even if some features are still missing.

WebKit weekly report #13

Blog post by PulkoMandy on Sat, 2014-01-04 12:16

Hello everyone!

You probably already read the news on the homepage: I'm continuing to work on WebKit for January.
Maybe you noticed there was no report last week, as I was visiting family and didn't get much work done. I'm not counting that week as paid work for Haiku.

Most of the work I did during the last two week revolves around the testsuite stuff. The testsuite engine got support for tests that need some time before the reports are parsed. We used to dump the page right after loading, but in some cases the test wants us to wait a bit and notify us when it's done. The test uses Javascript to notify the testsuite system of this. We now get better (and more reliable) results for many tests that run some javascript code before giving their results. On the downside, the testsuite now needs more time to run, about 3 hours on my computer (using 2 cores).

I also added some support for dumping pages with frames. Only the top level would be dumped before, and of course that didn't quite match what the references for the tests expected.

I'm now starting to get a list of mostly actual test failures. A lot of them are because our font metrics aren't exactly the same as on other platforms. This leads to slightly different size for some text runs, and ultimately a different dump for the page. Some of the tests use the Ahem font to get predictable results, but not all of them. The ones that don't may need platform-specific test results to avoid the problem.

We are also failing a lot of the tests that use SVG. While the fixed gradient support improves things, it's not quite enough. The SVG drawing in WebKit is one of the places where transform matrices are used to scale, pan and rotate objects when drawing them to the view. This is another thing our BView can't do yet. I started looking at what's needed for this, and briefly discussed with stippi, our app_server expert, over IRC. However, I think it's better to get the testsuite ready before I do the change. This will allow seeing how much tests we can get to pass, and if it breaks other things (giving me an hint on what to work on next). The testsuite will also be useful when merging changes from WebKit again. We're now some month behind, but our current builds are fairly stable, and I don't want to break everything by merging the latest changes without having a way to track the regressions (and possibly the bugfixes).

I'll be uploading a new webkit build next week. This will include the gradient and box-shadow fixes, as well as the web worker crash fix, and some other things I did since last time. This should be a "reasonably good" version with enough stability to last for some time. I'll try to fix the remaining issues with the border color bleeding on the background, and the broken back/forward management, as these are the most visible and annoying of the remaining problems. I'll have to learn about the back/forward code, which, as you expect coming from WebKit, is more complicated than you'd think, for performance reasons. There is an in-memory cache for recently viewed pages, that avoids reloading everything when you press the "back" button. This includes the page itself, but also the position of the scrollbars and a few other things. While this seems very natural when using the browser, it's actually quite tricky to implement. The page can't be easily snapshotted because of JavaScript and other things that may be running, and the anchor used for restoring the scrolling may itself point to javacript-generated (or otherwise dynamically added) content.

Well, quite a lot of work to do, still.

WebKit weekly report #11

Blog post by PulkoMandy on Fri, 2013-12-13 07:41

Hello everyone.

Some progress again this week.

First of all, I finally updated the haikuwebkit package, for both gcc2hybrid and gcc4. This means you can download things with WebPositive again, and all the bugfixes since the last release are in as well.

On the testsuite side, I got the crash-report feature to more or less work thanks to help from Rene and Ingo. There was one improvement to Debugger to allow extracting the stack trace when using the --save-report option on a thread that's not (yet) crashed. I tried various ways of plugging this into the testsuite, but it turned out catching the signals is the right thing to do. I also had to override debugger() in the testsuite system to call abort, and raise the expected signal, instead of dropping directly into debugger.

Most of the crashes reports are now working, but sometimes Debugger fails exiting and stays waiting for something. I have to manually kill it to get the testsuite to continue. Well, the cases where it works are already quite helpful, helping me to catch some bugs in my image saving code for the "pixel test" stuff, and also improving the logic to decide when to run the next test - we were sometimes doing this before the current one had finished running its javascript code.

Writing of the TestExpectations file continues, I have enough entries in it now to filter some of the tests out and make the final report useable. WebKit reports tests that didn't run as expected (crashes, things that failed, and things that passed when we told in the TestExpectations that they should fail). With 15000 tests not passing, the WebPage is quite huge, but now small enough Web+ manages to render it. I'm exploring the results and either adding rules to TestExpectations, or fixing the bugs when I've also seen them create issues on some web pages.

One of the fixes I did is support for gradients (in CSS and SVG). We had some code written, but apparently not too well tested. It was using the wrong range for the gradient stops (our API expects 0 to 255, and the code used 0 to 1). Also, the gradient semantics are a bit different, in our gradients, if there is no stop at the start or end of the gradient, they will draw random colors. CSS expect the first and last stops to extend to the gradient edge.

I already identified some other problems in both the testsuite and WebKit code itself. I'm not going to fix all the issues, so my goal is to write as much of the TestExpectations as I can, trying to describe why each of the tests fail and grouping them together. The goal is to get everything to run "as expected" (even if that means actually failing). This way, the impact of any change can easily be identified by the number of tests fixed or broken.

I'll still have a look at the crashing tests. If we can get those fixed, the testsuite will run more smoothly and we can expect the actual browser to be more stable as well.

WebKit weekly report #10

Blog post by PulkoMandy on Fri, 2013-12-06 08:24

Hello world!
You may have heard I'm in for another month!

So, this week I made some more progress with the testsuite system. The major improvements are finished support for the image comparison system I was talking about in last week report. The DumpRenderTree tool now dumps a PNG image of the webpage (this was quite easy to do thanks to the offscreen BView to BBitmap rendering and the translation kit) when requested (not all tests use the feature). Then, another tool called ImageDiff reads two images, and computes the difference. If the difference is big enough, it generates a new image highlighting areas that don't match.

I had to re-add the x86 translators to the x86_gcc2hybrid build so I could make use of them in the gcc4-only WebKit. Fortunately this wasn't too hard either, with just some jamfile tweaks required to make it work again.

The other thing I worked on on DRT side is support for saving crash reports. The problem is, several tests are crashing the DumpRenderTree tool. Without further handling, this results on the testsuite waiting on the debug_server alert asking wether to save a debug report, kill the team, or open Debugger. It is quite annoying to click "kill" an hundred of times as the testsuite runs. So, I'm now catching the signals (SIGSEGV and SIGABRT) from DRT, and from the signal handler I ask Debugger to save a crash report for the current thread. I also modified the python side of the testsuite so it reads reads these reports and move them to the right place. They are now stored with the test results, renamed to the name of the test that made things crash. This is quite useful to see what happened, and execute the crashing tests manually. There's still one problem: Debugger will currently not save any stacktrace, because it does that only for stopped/crashed threads. And since I'm catching the signals, the threads aren't in stopped state anymore when using this trick.

With these fixes out of the way, we're now passing over 52% of the tests. This may look quite bad at first but there are several factors to this, other than failing tests. Most of these come from features that we don't implement yet, such as WebGL, video and audio support, accessibility, battery status (yes, there is a javascript API allowing webpages to query that), web notifications, and so on. There are also a few minor problems like off-by-one size errors that make a lot of the CSS tests fail. And there are strage things like some colors not matching what was requested (a 0,255,0 green gets drawn as 0,254,0 for some reason, making most of the canvas test fail).

The next part of the process is reviewing these results and completing the TestExpectation file. This involves saying things like 'yes, we expect "battery status" tests to fail'. And there are some more fine-grained indications, either "we don't plan to implement it" or "here's the bug in our bugtracker". The testsuite tool can then notify people that some tests passed when they shouldn't have (meaning a bug got fixed somewhere), or on the reverse, some test doesn't pass anymore (so something was broken by your changes). People can update the TestExpectations in the first case, or fix the problems in the second one. There is also a way to tell that we expect some tests to crash the test engine, or to skip them altogether (I use this to skip the one test that crashes app_server).

While reviewing the test results, I also catched and fixed one bug in our WebKit drawing code. This was a corner case of canvas drawing functions, probably not very important, but it would have not been easy to catch without the testsuite. This set of simple pages focusing on a single feature at a time are great to hint you as to where to look for problems.

So, on to analysing the results and maybe fixing some more issues.

On a more interesting note, I finished the support for data: URIs, and added cookie storage for file:// ones. I will provide updated haikuwebkit packages next week including these changes and fixing the easy-to-trigger crashes in the current Web+.

WebKit weekly report #8

Blog post by PulkoMandy on Fri, 2013-11-22 08:19

Hello there!

So, this wasn't a fun week. Last week I had finished merging all commits from WebKit main repo. I got Web+ to run fairly well with these changes and I wanted to merge this into Haiku. However, while this works for the gcc2-hybrid version, I was not able to build a package for gcc4 and gcc4h. As a result, there were no nightlies published (the script wants all the architectures to work for a given revision before it gets published).

When trying to build the gcc4 package, I ran into several different problems: haikuporter triggering a KDL when running webkit "make install", the CMake script I used to build the gcc2h packages not working as it should on gcc4, and a few more. Since I don't usually run a gcc4 install, I had to do all this from a temporary install on an SD card, which is very slow at compiling, or, actually, at doing anything. Even a simple rm -r can take a minute or so to complete.

Meanwhile, I'm making some (slow) progress on getting the testsuite to run. For this to run I have to fix a tool called DumpRenderTree. This is a small executable wrapped around WebKit that renders a web page, and dumps to the console either a tree structure of it, or a plain-text dump of the page contents. It should also generate PNGs of the pages in some case. The testsuite runs DumpRenderTree on a set of pages and compares the results with "expected" files, and makes sure everything matches.

The problem I have is DumpRenderTree is always dumping the render tree instead of the plain text dump the "expected" files have. To switch between the modes, DumpRenderTree expects the page to call Javascript code to configure the output. The javascript interface can also do some other useful things, like clearing cookies, or otherwise interacting with the tool.

Since the WebKit API is specific to each port, the DumpRenderTree tool must be rewritten to use each of these APIs. I updated our version, modelling it on the EFL one. I now have something that should work, but somehow the JS callbacks aren't working.

I tried building WebKit with all assertions enabled to mak sure everything was right. I had to fix some minor harmless things to get HaikuLauncher to run, but DumpRenderTree would hit asserts over and over again. I tried disabling one, only to hit the next. Something looked strange, with methods as basic as isMainThread() failing (this is just comparing a TLS value with a global). I added some traces, and noticed the TLS was ok, but the global was changing value. I spent some time trying to look for something overwriting it, but then I noticed it was also changing address. This shouldn't be possible. Well, that hinted me to the actual cause for the problem: some files are getting linked twice into DumpRenderTree. what happens is we first build a big "libWebKit" library. This exports only our WebKit API, and inside it, there is also all the core code (WTF, JavaScriptCore and WebCore), the cross-platform parts of WebKit. The idea is we want the library to only export our official API, and not let apps mess with the internals.

However, DumpRenderTree does need to mess with the internals. It adds custom JS APIs for the app to access, reads internal data to extract frame content as text, and takes some shotcuts calling WebCore directly to do things in a way that works on all platforms. So, DumpRenderTree links not only our libWebKit, but also the WTF, JSC and WebCore libs again. Since we build these as static libs, we end up with code both in DRT itself and in libWebKit. Things then start breaking apart because these can't see each other. We get duplicated static variables, initializations called on one side that set variables the other side can't see, etc.

So, I moved back to the shared mode. This is what we used to do when WebKit was built with Jam. In this mode, WTF, JSC and WebCore are built as shared libraries, and libwebKit only links against them instead of embedding them. This allows things to exist only once in the running executable and DRT to access all the internal as it should.

But... that didn't work. I hit a command-line length limitation.

In the Jam build, we used to build multiple small static libraries (webcore1.a, webcore2.a, etc) and then link them together in a big shared library. This allows us to send a smaller set of files to ar to create the static libs and stay under the maximal command-line length. CMake uses a different trick to keep command lines short: it uses response files. These were added to gcc and the binutils to solve the very same problem on Windows. The idea is to put the command-line args into a file, and give it to the tool on the command line using te @file syntax. This works well with ar (when building the static libs), but our gcc will unfold the thing before calling some of the internal tools, and we hit the problem again when invoking collect2 (this is an internal executable used by gcc) or ld. It seems this is because gcc doesn't know that the linker it is calling is compatible with response files, so it tries unfolding the command line to be more compatible.

I'm now rebuilding gcc with the --with-gnu-ld configure option. This should force it into using response files when invoking the linker, and finally get our libraries built. Not this will change the ABI again and the haikuwebkit packages for both gcc2hybrid and gcc4 will have to be rebuilt. I'm not sure yet if we should do this only for the tests, or if we should also do it for the distributed libraries. It makes sense to distribute the ones we have tested, but the single-library solution should be faster (less symbol lookup) and use less memory at runtime.

If the gcc rebuild doesn't work, I'll have to use the same trick we did with Jam, making invasive changes to the cmake build. This would make it more difficult to merge with the official WebKit sources in the future.

Also, some sidenotes:

  • My patches to CD, IM, and IUP have been upstreamed. We now have a portable UI toolkit that uses native widgets
  • Kallisti5 is working on providing a server that runs Haiku. We will use this once he is done setting up to run the CMake testsuite/nightly builds, and run the WebKit tests. This is a step towards getting parts of our port merged upstream again.
Syndicate content