Blogs

New scheduler merged

Blog post by Paweł Dziepak on Tue, 2014-02-18 03:47

As you undoubtedly know, my scheduler branch has been merged a month ago. Also, some important changes has been made since, including bug fixes and performance improvements. It is now time to sum up what already has been done, and show some long promised benchmark results. There are still some issues that need to be addressed, but I do not think that any of them is a major one.

WebKit weekly report #19

Blog post by PulkoMandy on Fri, 2014-02-14 07:37

Hello everyone!

This week I worked on stabilization and small improvements of WebKit. There are a few new features, as well.

The crash with cursors I mentionned last week is fixed. I had forgotten to copy an object in the copy constructor, leading to a double delete. I continued working on the clipping code, and fixed the issues with www.haiku-os.org and a few other websites. But, I can't get it to work with haikuports, Trac, and now gmail is also broken. I don't want to do a release until we have a fix for that.

I also did some more work on supporting css shadows. With the updated clipping code, the ugly black box that was sometimes visible is now gone (it's clipped out). However, we also must draw the shadow itself. I did most of the implementation on WebKit side, but it needs support for a drawing mode (SourceIn) that app_server doesn't handle yet. So, I left that disabled for now, until we can get it to render the expected way.

An easy fix was adding the support for HTML5 "file" API. I just had to turn a compile flag on, and this gives us 10 more points on html5test.com. It also gets imgur.com image upload working.

I merged some changes from WebKit, without much problems. This brings the usual small fixes and cleanup, without too much code breakage this time. There are a lot of code cleanups going on at WebKit, making the codebase simpler and also faster.

I also did some work on the Network Kit HTTP backend. We now support gzip/deflate compression of HTTP data. This make some web pages load much faster, and also fixes issues with some websites serving compressed data even though the browser doesn't advertise support for it. While working on this, another problem in the HTTP code was discovered, there was a possible stack overflow because we were using a gcc extension to C++. This is now fixed, hopefully improving stability of the web browser.

With all those small issues fixed, it's time for another testsuite run. I hope I can find some test that fail because of our remaining clipping problems, as this would help me identify the issues more easily than with a complex web page such as gmail. I'd really like to squash at least the new issues introduced by this clipping change, so we can have a release that I'd qualify as stable. I'm trying to not start work on too much other features until we get this sorted out. Once that release is done, we can resume way on features that need more changes.

Next up on the TODO list: support for affine transformations (stippi added this to the app_server), which will improve SVG rendering a lot and possibly fix some other issues. Support for shadows, and the missing SourceIn composite drawing mode. When we get these two out, we should have much better rendering. There are probably other missing features, but they are yet to be identified.

See you next week!

WebKit weekly report #18

Blog post by PulkoMandy on Fri, 2014-02-07 10:17

Hi there!
As you can read on the frontpage, I'll continue working for Haiku in february.This will be the 5th month of this contract. Thanks to everyone who donated to Haiku, Inc for making this possible!

So, I've sorted out my filesystem issues over the week-end (no important data was lost), and I'm back to full-speed work. As I was saying last week, we had a problem with gcc4.7 not compiling the most recent WebKit code. I expected an update to gcc4.8 to solve this, but it didn't. What was needed is an extra configure option to enable C++11 threads support, as WebKit started using that and gcc doesn't autodetect the required OS support.

So, I rebuilt gcc with the needed option, and could get WebKit updated again, merging the work done at WebKit in december and january. As usual, not much crazy new features, but a lot of refactoring and cleanup. The API to manage the mouse cursor was changed to a simpler one that wastes less time and memory allocating stuff, some compile-time options were removed as all ports used the same value, and some custom classes were replaced with C++11 standard equivalents. This is made possible because of the release of a new version of Visual Studio, which still lacked some of these features. Of interest to us is the use of C++11 override. This comes from Java and allows to tell the compiler that a given method in a class should replace one from a base class. If it doesn't, you get a compile error. This is very useful in WebKit, as it allows detecting when the base class API changed (method removed or renamed, parameters added or removed, types changed, ...). I started adding the "override" keyword to some of the Haiku specific classes, and could remove a dozen of useless methods. This is one little change that will make further upgrades much easier. Another change is the deprecation of the history API we were using. This was the occasion to clean up our old code for this and get the back/forward buttons to work more reliably.

So, I started testing the new WebKit and noticed it was very crashy, with testsuite results as low as 4000 passing tests out of 32000 (ouch!). At least part of this was found to be caused by stricter stack alignment requirements on gcc side. gcc4.8 started using more x86 instructions that need 16-byte stack alignment. Before this only happened in some well-defined parts of the code, and I could fix this on a function-by-function basis. Now, all the code using floating point numbers is potentially affected. I can work around this by compiling all of WebKit with the -mstackrealign option, however this is something that should be fixed on Haiku side. Fortunately, the fix shouldn't be too complicated, it's just a change of the alignment constraint we have to do when creatign a new thread. If the initial alignment is correct, gcc generates code that always preserves it, unless you have some non-gcc compiled code in your function call stack (hand-written assembly code is one possible case of this).

There is apparently another crash related to the new mouse cursor code, which I haven't investigated yet. With these two out of the way, I'll have to run the testsuite again and see if there are other problems. With so few passing tests, the result html page (which list everything that failed) is too slow to browse and barely useable.

On to the new features now: it was more than time we fix the drawing glitches known as "border bleeding". You probably have noticed this problem on the side menu of this very website. There are some other places affected by this. Anyway, stippi did an amazing job of implementing ClipToPicture the right way. We now have a very fast implementation that also supports antialiasing. Bridging the gap between old and new apps, this improves the situation for both Gobe Productive (one of the few apps to use this API in BeOS days) and WebKit.

I had the code using ClipToPicture mostly ready in WebKit, waiting for the working implementation in Haiku. I could finally test it this morning and... well it doesn't work perfectly, yet. While it fixes the border bleeding, and we get our gradients where they should be again, when scrolling the page too fast (with pageup/pagedown buttons for example), the text above the gradients isn't drawn at all. Other websites also get new drawing problems in similar situations. I'm not sure what happens yet.

While stippi was working in app_server internals again, he also started implementing arbitrary view transforms. We had most of the API ready, with the BAffineTransform class available but only used to transform BPolygons. You can now set a transformation on a view and arbitrary rotate, scale, translate and otherwise distort all the drawing. This is all new and not yet completely tested (and in fact, there are some known bugs). But, it will allow a huge improvement of WebKit SVG rendering once it gets plugged in WebKit's GraphicsContext class.

I'm also trying to get some other devs into WebKit development, as working alone isn't fun. I've opened one "easy" issue on our bugtracker. It's about implementing Web Sockets support. There are some other things I would like to see done by others, for example support for web notifications using the BNotification API. Wouldn't it be nice to have a pop-up showing messages from gmail web page show next to the deskbar? Send in your patches!

WebKit weekly report #17

Blog post by PulkoMandy on Fri, 2014-01-31 08:50

Hello everyone!

The work started last week on ClipToPicture made some progress this week. We discussed this further with Stippi and now have a solution that doesn't involve rewriting half of app_server code, and is also a bit simpler and faster than what I tried to do first. I wrote a test application and some boilerplate code, then Stippi jumped in and implemented the missing bits. There are still some missing features like the ability to stack multiple clippings using PushState/PopState, and some problems when scaling and translating the view, as expected. We also met a drawing glitch when moving or resizing the window, however, we're not sure what's happening yet.

With Haiku switched to gcc4.8, I tried updating our WebKit to a newer version again. But, this doesn't work yet, and it seems the problem is a missing option in our gcc configure script invocation. I wanted to rebuilt gcc with the proper options, but I hit some filesystem corruption on my data partition. I'm now trying to backup everything, but a bug in Haiku makes this incredibly slow. Of course, I paused my contract since wednesday, and until I can get this issue sorted out and resume working. No data was lost, but touching some files on that partition triggers a KDL. So, it's time I reformat it and put the data back on it.

As a result of these FS problems, I haven't got much work done this week. So, this report is short.

WebKit weekly report #16

Blog post by PulkoMandy on Wed, 2014-01-22 11:33

Hello world!

As I said last week, the remaining drawing glitches are because of BView limitations. Well, it's time to solve those as well!

I'll start with what is now known as the "border bleeding" bug. You have encountered it if you tried opening the Haiku website, or the bugtracker, in Web+. You will easily notice that some items are completely filled with the border color, instead of the expected background one. To understand what's going on, let's have a look at the way WebKit draws things.

Google Code-In 2013 wrap up report

Blog post by scottmc on Tue, 2014-01-21 05:48

Google has now announced the 20 winners for Google Code-In 2013, with Freeman Lou and Puck Meerburg being the two winners from Haiku.

This was the fourth year of Google's Code-In, and the fourth for Haiku. This contest came at a good point this year for Haiku as the package management merge happened just a few weeks prior to the start of the contest and thus gave us plenty of ideas for tasks. Nearly half of our tasks were somehow related to writing recipes for packages to be built into .hpkg files. We also opened up the coverity scan results for students to try their hand at fixing some of those issues for the first time. Along with these tasks there were several others which ranged from fixing specific bugs from Haiku's trac tickets, to writing new programs such as a blogging program and a spider solitaire game, and even a few for the artistic type students who created a new flyer and some new icons.

This year we had 5 students who completed 20 or more tasks, this is more than any of our students completed during GCI 2012. We had 42 students who completed a total of 245 tasks for Haiku which is more than have been completed in any previous year for Haiku, so it was a very good year for us. Of the 42 students, 19 of them completed 3 or more tasks which qualified them to receive a Google Code-In 2013 t-shirt, this was also the most students we've have that completed 3 or more tasks.

I'd like to thank the 19 Haiku mentors, which included 3 former Google Code-In students, and all 42 students who completed at least one task for Haiku this year. Also a special thanks to those who were on irc to help handle the flood of students during the contest, for their patience in handling of all the questions and such that the students were asking. And lastly, a special thanks to Stephanie Taylor and the rest of the team at Google for having Google Code-In, and for picking Haiku to participate again. It was another very productive and fun Code-In.

WebKit weekly report #15

Blog post by PulkoMandy on Fri, 2014-01-17 08:46

Hello again!

No big new changes this week, but a lot of small fixes and improvements.

I reviewed the growing issues list for Web+ on the bug tracker, and fixed several of them. Most of these were small and rather easy to fix bugs (I kept all the harder ones for later). Here is a list with comments, not that the issues were hard to track, but this is also a way to learn a bit more about the WebKit codebase.

Web+ crashed when trying to upload a file to GMail. This was a bug in our BFormDataIO code we use for serializing the form data into the HTTP stream. It missed the case where the first element in the form was a file, and tried to read from it without initializing it first. The FormDataIO class is used so we don't have to put the whole form data in memory in order to send it. It handles each form element one at a time, with special case for files, which are streamed from disk in small chunks, rather than loaded into memory.

Web+ also crashed when trying to decode a huge image. The test case for this is a 93MB JPEG file that expands to 700+MB of pixel data. ShowImage manages to display that, however Web+ tries to do incremental decoding, showing an incomplete view of the image as it is loaded. Our implementation of this is not optimal, as there are at least two copies of the data, one in a BBitmap and one as a raw byte array. For now I fixed the crash, but we abort the decoding and just show a blank page instead. We may want to review the image code to lower the memory use.

I implemented listing local directories in the Services Kit. WebKit has support for rendering directories as part of their FTP handling. Returning files list in one of the formats FTP listings use (there is no standard for this, but a few common formats in use) makes WebKit parse it and generate an HTML page for listing. There are still some problems with encoding (WebKit doesn't seem to expect UTF-8 filenames in those listings), but things should be working now. I also fixed some problems with symlinks in the file:// protocol handler.

Some drawing glitches were fixed (again). We're now in a state where all improvements will require adding support to BView.

There was also a problem with opening links in a new tab from inside a frame. I also added shift+middle click as a way to open a link in a new tab and immediately switch to it (middle click alone opens the tab in the background).

I did several fixes to Cookie management. The most important one is there was a bug in the code for getting cookies for a specific website. A misuse of our StringHash class (this is a simple class that allows using a string as a hash for a hash map) led to memory corruption. We were trying to set the key for the map to a substring of the previous key to implement domain exploration (so a site at www.example.com can access cookies set at example.com - but not for just 'com'). Basically, the HashString freed the old key, then tried to copy characters from it to the new key, using memcpy. This is a classic use-after-free problem, that didn't always create problems in normal run, but was very obvious when running the browser with libroot_debug. Another fix was the proper implementation of CookiesForDOM. This is one of the two methods for accessing cookies. We used the same code as for the access from Javascript, but CookiesForDOM must also include "HttpOnly" cookies. Finally, a third bug was wrongly parsing the expiration date for cookies using the local time zone, whereas they need GMT dates. Depending on your timezone, this lead to cookies expiring too late (you probably didn't notice) or too early, sometimes right in the past. For example, some banking website use short-lived cookies (1 hour or less) as a timeout system. In my GMT+1 timezone, the cookie was expired immediately and I couldn't even access the login screen.

Some fixes went into the SSL support. One case of crashing was fixed, we were deleting the OpenSSL connection context before the network thread had a chance to exit, leading to a crash when leaving an https page before it finished loading. I also started work towards proper support for certificate checking. SSL connections didn't do any checking for certificates, and actually didn't even load the certificate store, making the SSL host authentication useless (you still get the encryption, but you can't make sure you're sending things to the right server). I implemented the Network Kit side of things, but now I must get this exposed in the Services Kit, then in WebKit, and finally add a nice dialog in Web+ asking what to do. Then, I must get the answer back to the network kit and continue or stop the connection with the unsafe host.

On WebKit side, I did a lot of small - but useful! - usability enhancements.

We got the error reporting for non-http connections working again. When trying to open a non-existing file:// URL, you now get a "file not found" message instead of a blank page.

The URL bar now always has an icon (the default is a little globe), to avoid the URL jumping to the right when the favicon gets shown. I also fixed some glitch pixels below the text in that bar, when using small font sizes.

I reworked some of the bookmark loading code. Now, bookmarks load in the current window, instead of the first window they can find in the workspace. If you open several of them at the same time, it works as expected. There was a race condition leading Web+ to try opening several bookmarks in the same tab, with of course only the last one showing up. Another problem was it was not possible to use symlinks in the bookmarks folder, as the BNavMenu we use for bookmarks wasn't traversing them. This now works as expected.

The search page in Web+ is now configurable. This means you can switch to goodsearch.com and help raise some money for Haiku while searching! Or, you can use the local version of Google or whatever search engine you prefer. The bug that made us unable to search for UTF-8 strings was also fixed, so you don't have to search in english anymore. And, there was also some progress with IDN domains, but the complete fix for this will have to wait for the next update to the WebKit package.

So, what's next? I will continue working on better SSL support, as this is currently set as an alpha blocker. I also plan to have a look at doing a bookmark bar. I tried doing this as a BMenuBar + BNavMenu, but these classes aren't designed for multiple inheritance, so I have some refactoring to do there. Or maybe I should go with another approach.

The "network lock-up" bug and missing BView features are also still fairly high on the TODO list, but these will need more time as I'm not as familiar with the code in these areas.

I didn't do this for some time, but let's also talk about non-working-hours time I also spent on Haiku. I did some Haikuporter recipes for XRick, OpenTTD, and a few other games. I also finally made a recipe for libusb, and others have used this to compile libftdi and avrdude. This isn't quite working yet, but I hope someone gets it going so I can finally do some hardware hacking on Haiku (did I hear blinkenlights?).

See you next week!