Diving into WebKit

Blog post by stippi on Tue, 2010-02-23 19:11

First of all, I want to thank Haiku, Inc. for giving me the opportunity to concentrate fully for a while on the WebKit port and browser! This is an awesome chance that I intend to make full use of.

At the moment, I have mixed feelings. Not about writing blogs. Not about working on WebKit. But about using the new WebKit browser to write the blog entry, haha! I've seen it crash, although in the last days, it has become pretty stable. After we upgraded to a newer WebKit version as the basis for the port, the frequent random crashes have almost disappeared and I saw only one crash in three days. Compared to one every few minutes before.

But still, I am of course pushing it a bit by eating my own dog food so early on. Lucky for me, I have integration with the system clipboard already working, so I can at least copy & paste my writing into Pe from time to time.

I'll try to give an overview of how I came to work on the WebKit port and what I've done so far. Basically, I woke up one day and thought Haiku really needs a modern, efficient and well integrated browser - now. In the beginning it wasn't clear to me yet, what the best approach for this would be. I started out with NetSurf and looked into that port. Francois has done some solid work there, but the flickering annoyed me and I've looked a bit into how this could be fixed. At the same time, I tried a few websites and it just wasn't convincing to me that NetSurf would be a full featured browser soon. It's certainly an awesome project, but it looked to me like it's not in the same advanced state as other engines. So I've started to consider Firefox and WebKit. For Firefox, I've heard that the problem is the Cairo port for Haiku. So I've downloaded the Cairo source tree and pretty much rewrote the BeOS port under the presumption that the existing Cairo software rendering backend could be used to draw into BBitmaps to get an instantly fully compliant implementation. Then however I failed to get the Makefiles set up correctly for building the testing framework. At the same time, it became clear through talks with Fredrik Holmqvist, that porting a current Firefox would be much more work than just porting Cairo to Haiku.

So WebKit began looking more and more attractive. I've quickly learned that Maxime Simone was keeping the WebKit port updated since his work on it during the Google Summer of Code 2009. I checked out the repo and began working on getting it to compile. Maxime was stuck at some linking problems, but after a while I figured out what those were and then Maxime helped me get the rest of the port into compilable state again. After I finally looked at the HaikuLauncher, which is the testing application to embed a WebKit view into an application, render a website for the first time, I've been working on the port and the browser ever since.

I think the infrastructure for ports in the WebKit source tree has improved a lot since Ryan Leavengood did the initial port in 2007. One of the last things that Maxime did was enabling the pthread threading backend. When I ran the HaikuLauncher for the first few times, I ran into all kinds of asserts, because WebCore code was executed on the wrong thread. My first work was therefor to design a proper threading concept for the WebKit code. Maybe it's a good time to mention that WebCore and JavaScriptCore are the core frameworks, the actual web engine. WebKit is a thin layer on top of WebCore that provides an API for embedding and controlling WebCore on each respective platform. So WebKit is where our BWebView and BWebPage will live, while WebCore is the actual backend. The Haiku port is thus separated into these parts. Three parts actually: There are all the files in WebCore which provide platform native implementations for various aspects of WebCore.There is for example GraphicsContextHaiku, which implements the WebCore rendering backend API via BView drawing functions. In WebKit, there is two sets of files. One part is the actual WebKit public API that the Haiku port provides, the other part are the so called WebCore support classes which bridge between WebCore and WebKit.

Work had and has to be done in all three parts. Some stuff was already finished from when Ryan worked on the port, other things were finished by Maxime, but a lot of the code was still stubbed out. Lucky for me, the port also consist of it's own jam based build system, which I presume Ryan wrote. It's nice not having to deal with that as well. After I fixed and implemented a great deal more code, HaikuLauncher became somewhat usable for surfing, and more people began to be excited about the WebKit browser. Michael Lotz has been crucial in tracking down some hard to fix memory corruption bugs and in extending the CURL network layer backend in WebCore to support htdigest authentication. Rene Gollent has implemented FileChoser support and tabbed browsing in HaikuLauncher. It was of course very encouraging to get support from so many people. Each day, the browser became more usable. And sometimes fixing one little thing, even if it was very hard to track down, can suddenly enable a whole bunch of websites.

So far I think my choice to invest into WebKit was a good one. It's possible to make a very native feeling browser, although some compromises have to be made. While WebCore is essentially single threaded, at least the native drawing happens in one app_server thread per page. And this will of course improve over time, as WebCore itself will make more use of threading. In terms of feeling native, I believe HaikuLauncher has left the Firefox port far behind. It even renders faster in many situations, sometimes drastically faster.

Today, I've finally found the problem with frames and iframes not being loaded and you can thank Ingo Weinhold and praise his source level debugger for that. This too made a great deal of sites work suddenly, like maps.google.com, the full version of mail.google.com, www.gmx.net finally works... and I've found the cause of some annoying clipping problems. If you want to give the HaikuLauncher a spin, you need to run a GCC4 or hybrid Haiku system. And it needs to be pretty recent, since we've also fixed bugs and added features in Haiku to support stuff for WebKit and HaikuLauncher. Have fun! But keep in mind it can still crash. Just because I managed to finish this blog doesn't mean you should trust it, yet. :-D

Download WebPositive (SVN revision: 444)

P.S. I've stolen the awesome icon from Meanwhile's great BB icons package.

Comments

Re: Diving into WebKit

A little confused, the package has two libraries in it...?

Re: Diving into WebKit

kvdman wrote:

A little confused, the package has two libraries in it...?

When you view the content of the archive, all the files seem to be there; but when you unzip it, all you get is the lib folder with libwebcore.so and libwebkit.so in it. It looks like the ZIP is corrupt.

Re: Diving into WebKit

There's something nasty wrong with Haiku's zipper. I filed this a while ago too. Seeing a lot of zips Haiku can't unzip properly.

http://dev.haiku-os.org/ticket/5396

*update: as in the ticket I submitted, the file unzips properly under Windows and OS X, but not under Haiku.

Re: Diving into WebKit

In this case that has nothing to do with Haiku's zip command. That zipfile registers as corrupt on every OS I have here.

Re: Diving into WebKit

Yes, sometimes the zips have corrupt structures. The fact is, that with the file I added in that ticket, and Stippi's file, they do unzip on OS X and Windows, and the files within them are usable. With Haiku, you just get either nothing or an incomplete unzip.

*didn't unzip under Windows, but OS X.

Re: Diving into WebKit

unzipped under OS X but don't run under Haiku gcc2hybrid rev. 35528

Welcome to the Haiku shell.

~> /boot/home/Desktop/HaikuLauncher/HaikuLauncher
runtime_loader: /boot/home/Desktop/HaikuLauncher/HaikuLauncher: Could not resolve symbol '_ZN8BTabView9SetBorderE12border_style'
resolve symbol "_ZN8BTabView9SetBorderE12border_style" returned: -2147478780
runtime_loader: /boot/home/Desktop/HaikuLauncher/HaikuLauncher: Troubles relocating: Symbol not found
~>

Re: Diving into WebKit

Maybe he also uploaded the file with the new browser ;) I'm sure it will be fixed soon!

Great to see that you're given more opportunity to continue what Ryan and Maxime have already done so far. I'm setting up a GCC4 hybrid just for this ;)

Re: Diving into WebKit

Don't extracts under Haiku, Ubuntu 9.10 nor Win7.. and I don't have OSX

Re: Diving into WebKit

Sorry for the corrupt .zip. I think I tried to test the link while the file was still uploading. Somehow that must have confused the server. To Karl: I highly doubt that this file unzipped on any other OS.

Re: Diving into WebKit

Unziped under OS X work with Haiku gcc2hybrid rev.35580

Thank you Stippi!

Re: Diving into WebKit

stippi wrote:

To Karl: I highly doubt that this file unzipped on any other OS.

I hate when people basically call you a liar. Beck could also unzip the first zip under OS X.

http://img62.imageshack.us/img62/826/osx.jpg

Also, I tested this on a nightly hybrid. Not a big deal, but it asks for Curl. Perhaps that could be included under the /lib folder so it works out of the box.

Re: Diving into WebKit

Ok sorry. What may have happened is that you downloaded the .zip on OSX before I messed it up. Or you downloaded it after I fixed it in place. In any case, the same broken .zip cannot have unpacked correclty on OSX. Or maybe it created all files from the listing, as some people mentioned the listing was complete. But it cannot have extracted them with full contents. The .zip was simply broken. That's why I assumed you concluded from your earlier findings with some .zip files not extracting correctly on Haiku, that this was another instant of the bug without having it tested for real. Sorry about that.

As for CURL, I don't want to include more libs. CURL is supposed to be installed on a real Haiku image.

Re: Diving into WebKit

stippi wrote:

As for CURL, I don't want to include more libs. CURL is supposed to be installed on a real Haiku image.

Thanks, but I've downloaded R35580 right from haiku-files.org and libcurl wasn't on the image.

You can see here it's an optional package:

http://haiku-files.org/files/optional-packages/

It was the only dependency missing and is ~235kb.

btw, I've tested the original 'corrupt' zip extracted under OS X in Haiku, and it works fine. You also have Mac, try it if you don't believe me ;)

Re: Diving into WebKit

I know CURL isn't in the nightlies. Those are supposed to be bare minimum images. CURL itself requires OpenSSL, so CURL in fact isn't the only dependency. I meant that one should use a proper Haiku image. The nightlies can be used to overwrite the system folder of an Alpha 1 installation, and for testing regressions. But I wouldn't regard them as proper Haiku installations by themselves.

Re: Diving into WebKit

I see thanks for the explanation regarding Curl.

I thought most people would be using the nightlies, since it's itself recommended by the devs to use a recent build not the alpha (since changes were enabled since then to allow HaikuLauncher to work), and most people don't build their own images. So, I don't really see why adding one small file makes that much of difference to make it work out of the box. Just my two cents.

Re: Diving into WebKit

Alpha 1 is very outdated now it's useless.

Re: Diving into WebKit

Very, very cool. I will install shortly.

Re: Diving into WebKit

Ok. By popular request, I've promoted the article to the front page.

Also, I've updated the package with todays achievements and mentioned the SVN revision in the link. Since HaikuLauncher doesn't have versioning, you will need to remember which version you downloaded and if the link points to a newer package. I keep uploading them in place.

CURL is now included and since OpenSSL comes in the nightlies, it should now work on virgin nightly images.

Todays version fixes rendering text with alpha channel and fixes more problems with the download window. Downloads can now also be canceled (but restarting is not yet implemented). I've also fixed some oddities with clicking links with the primary versus tertiary mouse button. Tertiary clicks will now always open new tabs in the background. When clicking links with the primary button which opens new windows (i.e. tabs), the new tab is automatically selected. The rest of the changes were internal refactoring and renaming to change the Haiku WebKit API into the official Haiku style and do a better job at hiding WebCore internals.

Have fun!

Re: Diving into WebKit

Ah.. Now it's ok!!

Wow Stippi!!! It's very functional already!!

THANKS FOR SHARING!! KEEP THE GOOD WORK

Re: Diving into WebKit

http://www.haiku-os.it/?q=node/695

Thank you Stippi, Ryan, Maxim, Rene and Ingo :)

Re: Diving into WebKit

Please reupload working zip. Can't wait to test this one. :)

Re: Diving into WebKit

WOW! This thing is fast! Amazing, it started in an instant, I guess not even in a second. Nice work!

(tested on gcc4hybrid from 24/02 - had to install Curl from optional packages)

Big Thanks and congratulations!

Re: Diving into WebKit

I already uploaded a new one after I saw the comments. Sorry again.

But I've just uploaded another version with a bunch of stuff fixed:

  • The URL text control going out of focues does not cause a page reload anymore.
  • The Download window pops open also for other tabs than the first one.
  • New tabs show immediately, not only when the title of the page becomes known.
  • A whole lot of internal refactoring and cleanup has been done.

Re: Diving into WebKit

You could include it into optional packages already

Re: Diving into WebKit

If any one wonders I didn't really promote Firefox. There is a lot of work to do to get it back into shape, not having any work done since January 2007. So I'm very happy that stippi chose to work on WebKit, IMO it is a much better fit than Firefox.

Firefox will probably come later on as soon as we find some coders that likes hard work and challanges.

/Fredrik Holmqvist, TQH

Re: Diving into WebKit

Perhaps the article can be promoted to the front page, not everybody catches the Blog-O-Sphere links I think. And a screenshot might be nice.
Also I agree it would be a nice idea to make it an optional package, I'm sure it will be a good way to get exposure. Maybe at first launch you can have a popup stating this is a work in progress.

I just tried it, it is very fast already! I'm impressed! It is a crown on the work of all HaikuLauncher contributers.

I'm not sure you're taking feature requests, but in case you are I have some. Of course you can choose which ones appeal to you, but it is just a list of things I miss from other browsers.

  • Usually alt-d brings you to the url bar, haikulauncher goes to the downloads window
  • Double clicking next to the open tabs usually brings up a new tab. Or maybe also just add a plus button to open a new tab.
  • cmd-(shift)-tab to scroll forwards (and backwards) through the tabs?
  • A mouse cursor that indicates when a link is clickable would be nice.
  • Ctrl-enter (or cmd-enter for Haiku I suppose) when you type an url usually adds .com.

How about the name? Have you talked to Ryan and Maxime about this yet?

Wow I'll shut up now, thanks for your efforts so far!

Re: Diving into WebKit

Any rational behind alt-d? I cannot make the connection. I'd understand alt-o. As for the other feature requests, I am basically aware that it still needs a lot of work to be more comfortable to use. :-)

About the name... it's pretty much decided that it will be WebPositive. The plan is to move the browser code into the Haiku source tree, along with a stubbed out libwebkit.so (Axel's idea). That way, the browser can be built at the time of creating a Haiku image, and the real libwebkit.so can be added as optional package. Once the move is done, I expect the browser to be named properly. HaikuLauncher in the WebKit source tree can then be simplified again for just running tests and serving as a simple demo.

Re: Diving into WebKit

stippi wrote:

About the name... it's pretty much decided that it will be WebPositive.

Woo hoo! That's just great, my favorite one! :-)

Re: Diving into WebKit

stippi wrote:

Any rational behind alt-d? I cannot make the connection.

Way back, browsers (IE) had the word Address: in front of the URL-box. As alt-a was already given away to select-all (edit: no, that would be ctrl-a; don't remember what alt-a was for), the next letter would become the shortcut for the URL-box: alt-d.

Re: Diving into WebKit

stippi wrote:

Any rational behind alt-d? I cannot make the connection. I'd understand alt-o. As for the other feature requests, I am basically aware that it still needs a lot of work to be more comfortable to use. :-)

Yeah, like Idefix said, probably aDdress. Thanks and good luck.

Re: Diving into WebKit

stippi wrote:

About the name... it's pretty much decided that it will be WebPositive.

Somehow I expected "Web-O-Matic". ;-)

stippi wrote:

The plan is to move the browser code into the Haiku source tree, along with a stubbed out libwebkit.so (Axel's idea). That way, the browser can be built at the time of creating a Haiku image, and the real libwebkit.so can be added as optional package.

I don't see the advantage of the dummy libwebkit.so. I'd simply make the WebKit an optional build feature like the SSL support.

Re: Diving into WebKit

Usually alt-d brings you to the url bar, haikulauncher goes to the downloads window

Actually, alt-l/ctrl-l/cmd-l is the standard shortcut for focusing the location bar in Firefox, Chrome and Safari. Internet Explorer uses the same shortcut but opens a stupid window where you have to write or paste the url.

Alt-d/ctrl-d/cmd-d is the shortcut for add bookmark. Some pages even say something like "Press ctrl-d to bookmark this page now."

Re: Diving into WebKit

BTW, is there somewhere a public commit list where one could keep track of what happening with WebPositive.

Re: Diving into WebKit

Stippi, will you pay homage to the original BeOS web browser NetPositive look-n-feel wise (button icons, etc...)?

http://www.tunetrackersystems.com/bedocs/documentation/User's%20Guide/03_network/Network07_NetPositive.html

http://www.libpng.org/pub/png/img_png/pngs-img-netpos.png

Re: Diving into WebKit

Hi,
i like the Net+ Navigations icons (2D) and the Net+ IconsToolBar.
The Tabview handle like Terminal App and Navigation hide and show in the Mediaplayer + fullscreen support.
The name i vote for WebLauncher (WebPositiv and WebSurfer a other good name)

Ralf Schülke aka stargater

PS: Good Job Stippi ( i hope you add a haiku error code :-) )

name..

It's not too late to make a vote. "Hawaii" was a best option than WebPositive. :')
Anyway, i guess that's not important as long as "it work the way it is suppose to do" © ;)

Re: Diving into WebKit

Yes, there is a Trac instance running on the server where we currently host the SVN:

WebKit timeline

Note that the server is a bit flaky, and HaikuLauncher will currently show an error, but FireFox manages to display the page most of the times.

Re: Diving into WebKit

Great to see WebPositiv working...

Very good work.

One reason less to use a Qt-Application (Aurora).

Re: Diving into WebKit

I'll have to look into how OpenSSL is included as an optional build dependency, but I am pretty sure that however that's done I definitely don't want to build WebKit when I build a Haiku image. ;-)

Re: Diving into WebKit

Just updated & rebuild latest Haiku from svn.
Then replaced installation files (booted from CD).
It say : Could not open "HaikuLauncher" (Missing symbol: __pure_virtual).
:(

Re: Diving into WebKit

Sounds like you're using gcc2 Haiku...WebKit requires gcc4 and as such so does the browser, so you'll need either a gcc4 build or a hybrid.

Re: Diving into WebKit

10x, you are right :)
This post is from WebKit/HaikuLauncher :)
Great job :)
It will be good if you can add some load indicator somewhere.
Now if you click on a link, nothing happends and you can't tell if browser is loading something or not.
It could be a mouse cursor change too for first versions :)

Re: Diving into WebKit

Loading indicator was like the first thing I added to the HaikuLauncher interface. Look closer! :-) Hint: It's in the same spot as the BeZillaBrowser loading indicator... :-D

Re: Diving into WebKit

It's pretty amazing what Stephan, Michael and Rene have been able to do in the past month on this project. I knew for a long time that WebKit was the solution for a nice native Haiku browser, and it is no surprise that Stephan came to the same conclusion.

I wish I could have done more to bring the WebKit port and the browser to this current level, but it seems a lot of this was just out of my league. Some of the bugs were really subtle and took the low-level skills of Michael to solve, and Stephan definitely understands the Haiku drawing and threading system better than I do. Nonetheless I guess I did do a lot to get things as far as they were for Maxime to update in the GSoC, and for this continuing work.

And I certainly plan to contribute myself when I have more time, but my work lately has been really hectic. I have at least downloaded the code, built it, and played with the browser and it is already a pretty pleasant experience as everyone else has discovered.

As for the name I think WebPositive is a good choice. I had my own idea for a name, and I'm sure I'll still use that name for something, but for the default Haiku browser keeping it simple and within the heritage of BeOS makes sense.

In regards to the shortcut for selecting the address bar, in all the browsers I've used recently they used Alt-L (for Location.) I'd vote strongly to use that over Alt-D, which could either be for the Downloads window as it is now or for bookmarking (which is what it seems to be used for on other browsers.)

- Ryan Leavengood

Re: Diving into WebKit

Great work Stippi.

Where I can download new revision of WebPositive?

WebPositive don't pass peacekeeper test, you can try it by yourself.

Re: Diving into WebKit

Here a other http://www.aadmm.de/

Re: Diving into WebKit

Hi all,

I've uploaded a new package. You can get detailed information on the changes from the timeline. This is an overview:

  • MIME types of local files are detected without the need for a file extension.
  • Not currently visible pages don't consume render time.
  • Co