Blogs

How To Permanently Blacklist a Package File

Blog post by Barrett on Sun, 2013-12-15 19:17

With the advent of package management and hrev46391, it has become possible to prevent a package from being extracted at boot time.From a suggestion of Matt, and with the contribution of Luroh (thanks!), i would like to explain you how to blacklist a package file in Haiku.

In Haiku's boot menu , there is a 'Blacklist entries' option available. This method will only let you disable system packages, and only until the next time you reboot.

You may know, since the package manager has been added to Haiku, some directories are read only, so unlike the past, isn't possible to just delete the driver. The Blacklist functionality addresses the issue that it may be necessary to remove a problematic file such as a library or a bugged driver, which would otherwise require editing the containing package file.

So let's go to the very simple steps to do that :

  1. Figure out which file in which package you want to blacklist, especially keep in mind if the file is contained in a system package or in a user one.
  2. The second step is to create a text file named 'packages' in /boot/system/settings or in /boot/home/config/settings/global, respectively the first directory is used to blacklist system packages the second is used for user packages.

The final step is to fill the packages file with something like that :

Package 'packagename' {
	EntryBlacklist {
		'entrypath'
		...
	}
}

'packagename' is the name of the package without version, for example 'haiku'.

'entrypath' is an installation location relative path, e.g. "add-ons/Translators/FooTranslator".

This way, blacklisted entries will be ignored by the package_fs. So that they won't appear in the file system.

Let me give a pratical example.

In my case i had the broadcom570x driver to blacklist, so i created the packages file under /boot/system/settings with this content :

Package haiku {
	EntryBlacklist {
		add-ons/kernel/drivers/bin/broadcom570x
	}
}

Then i saved it and rebooted. Once Haiku start the package is re-mounted and the file is ignored by the packagefs.

Hope it was interesting!

WebKit weekly report #11

Blog post by PulkoMandy on Fri, 2013-12-13 07:41

Hello everyone.

Some progress again this week.

First of all, I finally updated the haikuwebkit package, for both gcc2hybrid and gcc4. This means you can download things with WebPositive again, and all the bugfixes since the last release are in as well.

On the testsuite side, I got the crash-report feature to more or less work thanks to help from Rene and Ingo. There was one improvement to Debugger to allow extracting the stack trace when using the --save-report option on a thread that's not (yet) crashed. I tried various ways of plugging this into the testsuite, but it turned out catching the signals is the right thing to do. I also had to override debugger() in the testsuite system to call abort, and raise the expected signal, instead of dropping directly into debugger.

Most of the crashes reports are now working, but sometimes Debugger fails exiting and stays waiting for something. I have to manually kill it to get the testsuite to continue. Well, the cases where it works are already quite helpful, helping me to catch some bugs in my image saving code for the "pixel test" stuff, and also improving the logic to decide when to run the next test - we were sometimes doing this before the current one had finished running its javascript code.

Writing of the TestExpectations file continues, I have enough entries in it now to filter some of the tests out and make the final report useable. WebKit reports tests that didn't run as expected (crashes, things that failed, and things that passed when we told in the TestExpectations that they should fail). With 15000 tests not passing, the WebPage is quite huge, but now small enough Web+ manages to render it. I'm exploring the results and either adding rules to TestExpectations, or fixing the bugs when I've also seen them create issues on some web pages.

One of the fixes I did is support for gradients (in CSS and SVG). We had some code written, but apparently not too well tested. It was using the wrong range for the gradient stops (our API expects 0 to 255, and the code used 0 to 1). Also, the gradient semantics are a bit different, in our gradients, if there is no stop at the start or end of the gradient, they will draw random colors. CSS expect the first and last stops to extend to the gradient edge.

I already identified some other problems in both the testsuite and WebKit code itself. I'm not going to fix all the issues, so my goal is to write as much of the TestExpectations as I can, trying to describe why each of the tests fail and grouping them together. The goal is to get everything to run "as expected" (even if that means actually failing). This way, the impact of any change can easily be identified by the number of tests fixed or broken.

I'll still have a look at the crashing tests. If we can get those fixed, the testsuite will run more smoothly and we can expect the actual browser to be more stable as well.

WebKit weekly report #10

Blog post by PulkoMandy on Fri, 2013-12-06 08:24

Hello world!
You may have heard I'm in for another month!

So, this week I made some more progress with the testsuite system. The major improvements are finished support for the image comparison system I was talking about in last week report. The DumpRenderTree tool now dumps a PNG image of the webpage (this was quite easy to do thanks to the offscreen BView to BBitmap rendering and the translation kit) when requested (not all tests use the feature). Then, another tool called ImageDiff reads two images, and computes the difference. If the difference is big enough, it generates a new image highlighting areas that don't match.

I had to re-add the x86 translators to the x86_gcc2hybrid build so I could make use of them in the gcc4-only WebKit. Fortunately this wasn't too hard either, with just some jamfile tweaks required to make it work again.

The other thing I worked on on DRT side is support for saving crash reports. The problem is, several tests are crashing the DumpRenderTree tool. Without further handling, this results on the testsuite waiting on the debug_server alert asking wether to save a debug report, kill the team, or open Debugger. It is quite annoying to click "kill" an hundred of times as the testsuite runs. So, I'm now catching the signals (SIGSEGV and SIGABRT) from DRT, and from the signal handler I ask Debugger to save a crash report for the current thread. I also modified the python side of the testsuite so it reads reads these reports and move them to the right place. They are now stored with the test results, renamed to the name of the test that made things crash. This is quite useful to see what happened, and execute the crashing tests manually. There's still one problem: Debugger will currently not save any stacktrace, because it does that only for stopped/crashed threads. And since I'm catching the signals, the threads aren't in stopped state anymore when using this trick.

With these fixes out of the way, we're now passing over 52% of the tests. This may look quite bad at first but there are several factors to this, other than failing tests. Most of these come from features that we don't implement yet, such as WebGL, video and audio support, accessibility, battery status (yes, there is a javascript API allowing webpages to query that), web notifications, and so on. There are also a few minor problems like off-by-one size errors that make a lot of the CSS tests fail. And there are strage things like some colors not matching what was requested (a 0,255,0 green gets drawn as 0,254,0 for some reason, making most of the canvas test fail).

The next part of the process is reviewing these results and completing the TestExpectation file. This involves saying things like 'yes, we expect "battery status" tests to fail'. And there are some more fine-grained indications, either "we don't plan to implement it" or "here's the bug in our bugtracker". The testsuite tool can then notify people that some tests passed when they shouldn't have (meaning a bug got fixed somewhere), or on the reverse, some test doesn't pass anymore (so something was broken by your changes). People can update the TestExpectations in the first case, or fix the problems in the second one. There is also a way to tell that we expect some tests to crash the test engine, or to skip them altogether (I use this to skip the one test that crashes app_server).

While reviewing the test results, I also catched and fixed one bug in our WebKit drawing code. This was a corner case of canvas drawing functions, probably not very important, but it would have not been easy to catch without the testsuite. This set of simple pages focusing on a single feature at a time are great to hint you as to where to look for problems.

So, on to analysing the results and maybe fixing some more issues.

On a more interesting note, I finished the support for data: URIs, and added cookie storage for file:// ones. I will provide updated haikuwebkit packages next week including these changes and fixing the easy-to-trigger crashes in the current Web+.

WebKit weekly report #9

Blog post by PulkoMandy on Fri, 2013-11-29 09:26

Hello world!

This week I got the testsuite running. The fixed build of gcc I was talking about last week worked, and I can now link a working DumpRenderTree executable. The first test run wasn't very good, with DRT crashing quite often, some javascript alerts popping on my screen, and ended after running about 9000 tests in an app_server freeze.

I'm adding some of the missing features to DumpRenderTree: alerts are now dumped to the standard output like other ports do, so they can be compared as part of the testing process. I skipped the test that crashes app_server (the testsuite system has support for skipping tests, and a few other things). I also did some smaller changes like setting the default font size to 16px for the tests (the browser still defaults to 14px, which is a bit unusual, but nothing in the css spec enforces a default size of 16px) ; and changing the JS console messages format for DRT so they match other ports (the main change is to not include the file name in the message, this way the test gives the same result if run from a different directory).

I'm now working on support for PNG dump of web pages in DRT. Some tests are testing things that aren't easily seen in a text dump, such as CSS gradients. To test this, there are two systems used in WebKit testsuite: the first one is comparing the rendering with a reference PNG. This works well, but there are often minor changes to the rendering, making it hard to keep the test result up to date with the browser. The PNG files are also often platform specific, due to different antialiasing or drawing algorithms, font rasterizers, and the like. So, a better system called "reftests" is used. The idea is the same, but instead of comparing with a fixed PNG image, DRT generate PNGs of two different pages, and checks that they render exactly the same. When there are two ways of rendering the same thing, this works, and follows the browser changes automatically.

Our results are currently not very good, with a lot of failing tests. I expect the support for pixel-tests and reftests to improve this a bit. Note that the goal isn't to reach 100% success: none of the other platforms do that. Instead, there is a "TestExpectations" file which tells which test should fail or succeed. The actual results are compared with the file. When fixing a bug, you can then easily verify that you don't introduce new test failures, and possibly that you make existing tests pass when they failed before.

The WebKit project requires each commit to either introduce a new test, or fix an existing one, unless youhave very good reasons (sometimes it isn't possible to test something, or the test would take too long to run).

In order to run the complete testsuite, I will also need a running HTTP server. The current scripts support either lighttpd or apache. One option is porting one of these, the other is modifying the scripts to accept something else, such as PoorMan or Cherokee, for which we already have an Haikuports recipe.

With the testsuite running, we will now have a way to check that future work on our WebKit port don't break things that were already working, like it happened with the version bundled in current nightlies - I bet you noticed the crashes.

I didn't work only on the testsuite, and also did some small improvements on the network layer. I'm adding support for data: URIs, and improving the MIME type identification system, which is currently restricted to HTTP. I also started looking at support for gzip compression of HTTP data. This is suppoed to be optional, but some websites actually require it. It shouldn't be too hard to implement and could bring us an even faster browsing experience, anyway.

WebKit weekly report #8

Blog post by PulkoMandy on Fri, 2013-11-22 08:19

Hello there!

So, this wasn't a fun week. Last week I had finished merging all commits from WebKit main repo. I got Web+ to run fairly well with these changes and I wanted to merge this into Haiku. However, while this works for the gcc2-hybrid version, I was not able to build a package for gcc4 and gcc4h. As a result, there were no nightlies published (the script wants all the architectures to work for a given revision before it gets published).

When trying to build the gcc4 package, I ran into several different problems: haikuporter triggering a KDL when running webkit "make install", the CMake script I used to build the gcc2h packages not working as it should on gcc4, and a few more. Since I don't usually run a gcc4 install, I had to do all this from a temporary install on an SD card, which is very slow at compiling, or, actually, at doing anything. Even a simple rm -r can take a minute or so to complete.

Meanwhile, I'm making some (slow) progress on getting the testsuite to run. For this to run I have to fix a tool called DumpRenderTree. This is a small executable wrapped around WebKit that renders a web page, and dumps to the console either a tree structure of it, or a plain-text dump of the page contents. It should also generate PNGs of the pages in some case. The testsuite runs DumpRenderTree on a set of pages and compares the results with "expected" files, and makes sure everything matches.

The problem I have is DumpRenderTree is always dumping the render tree instead of the plain text dump the "expected" files have. To switch between the modes, DumpRenderTree expects the page to call Javascript code to configure the output. The javascript interface can also do some other useful things, like clearing cookies, or otherwise interacting with the tool.

Since the WebKit API is specific to each port, the DumpRenderTree tool must be rewritten to use each of these APIs. I updated our version, modelling it on the EFL one. I now have something that should work, but somehow the JS callbacks aren't working.

I tried building WebKit with all assertions enabled to mak sure everything was right. I had to fix some minor harmless things to get HaikuLauncher to run, but DumpRenderTree would hit asserts over and over again. I tried disabling one, only to hit the next. Something looked strange, with methods as basic as isMainThread() failing (this is just comparing a TLS value with a global). I added some traces, and noticed the TLS was ok, but the global was changing value. I spent some time trying to look for something overwriting it, but then I noticed it was also changing address. This shouldn't be possible. Well, that hinted me to the actual cause for the problem: some files are getting linked twice into DumpRenderTree. what happens is we first build a big "libWebKit" library. This exports only our WebKit API, and inside it, there is also all the core code (WTF, JavaScriptCore and WebCore), the cross-platform parts of WebKit. The idea is we want the library to only export our official API, and not let apps mess with the internals.

However, DumpRenderTree does need to mess with the internals. It adds custom JS APIs for the app to access, reads internal data to extract frame content as text, and takes some shotcuts calling WebCore directly to do things in a way that works on all platforms. So, DumpRenderTree links not only our libWebKit, but also the WTF, JSC and WebCore libs again. Since we build these as static libs, we end up with code both in DRT itself and in libWebKit. Things then start breaking apart because these can't see each other. We get duplicated static variables, initializations called on one side that set variables the other side can't see, etc.

So, I moved back to the shared mode. This is what we used to do when WebKit was built with Jam. In this mode, WTF, JSC and WebCore are built as shared libraries, and libwebKit only links against them instead of embedding them. This allows things to exist only once in the running executable and DRT to access all the internal as it should.

But... that didn't work. I hit a command-line length limitation.

In the Jam build, we used to build multiple small static libraries (webcore1.a, webcore2.a, etc) and then link them together in a big shared library. This allows us to send a smaller set of files to ar to create the static libs and stay under the maximal command-line length. CMake uses a different trick to keep command lines short: it uses response files. These were added to gcc and the binutils to solve the very same problem on Windows. The idea is to put the command-line args into a file, and give it to the tool on the command line using te @file syntax. This works well with ar (when building the static libs), but our gcc will unfold the thing before calling some of the internal tools, and we hit the problem again when invoking collect2 (this is an internal executable used by gcc) or ld. It seems this is because gcc doesn't know that the linker it is calling is compatible with response files, so it tries unfolding the command line to be more compatible.

I'm now rebuilding gcc with the --with-gnu-ld configure option. This should force it into using response files when invoking the linker, and finally get our libraries built. Not this will change the ABI again and the haikuwebkit packages for both gcc2hybrid and gcc4 will have to be rebuilt. I'm not sure yet if we should do this only for the tests, or if we should also do it for the distributed libraries. It makes sense to distribute the ones we have tested, but the single-library solution should be faster (less symbol lookup) and use less memory at runtime.

If the gcc rebuild doesn't work, I'll have to use the same trick we did with Jam, making invasive changes to the cmake build. This would make it more difficult to merge with the official WebKit sources in the future.

Also, some sidenotes:

  • My patches to CD, IM, and IUP have been upstreamed. We now have a portable UI toolkit that uses native widgets
  • Kallisti5 is working on providing a server that runs Haiku. We will use this once he is done setting up to run the CMake testsuite/nightly builds, and run the WebKit tests. This is a step towards getting parts of our port merged upstream again.

WebKit weekly report #7

Blog post by PulkoMandy on Fri, 2013-11-15 08:39

Hello world!

This week I reached a major milestone in my work, as I'm done merging all WebKit commits all the way to november 2013. HaikuLauncher is still running fine, and I will make the small required changes in WebPositive so it works fine again. Expect an updated WebPositive in trunk very soon now.

This week merges were as boring as the previous week ones: addition of the NiX port, removal of the Qt one (next version of Qt will go with Blink), replacement of a lot of never-null pointers to use references instead, rename of the KURL class to URL, and some API changes. WebKit has started using some C++11 features to help with cleaner and faster code.

I've also started looking at the test system. WebKit uses a tool called DumpRenderTree that will produce a text representation of the render tree. This allows running tests that are not dependant on any platform. There are also 'pixel tests', where WebKit renders a web page, and compares the results with a PNG file. And finally, there are some javascript-only tests for JavaScriptCore.

I've updated our version of DumpRenderTree (since WebKit has no common API on different platforms, each platform must provide an implementation for it). I've got it to run in the mode where you give it a single test on the command line, which is good for manual testing, but the testsuite instead feeds test names on the standard input, and this somehow crashes our version currently. I'll debug that and run the 33000+ tests to get a better idea of where we are.

There will be some more work to do before we can run the complete testsuite, for example tests involving HTTP require an apache web server port (anyone has a recipe for this ?), and many methods in the test frameworks aren't implemented yet. These allow the tests to access some browser settings from Javascript, for example they can disable cookies, change cache policies, and so on. This allows testing of WebKit with all possible settings.

I want to get the testsuite running to help with future work on adding new features and the merges we'll have to do. This will help getting a better idea of the status of our port, and make sure we don't break things when merging new versions of WebKit in. Also, the tests are usually simple and the failing one will point to the features we are missing, making the port much easier to fix and improve.

End of package management contract

Blog post by zooey on Sun, 2013-11-10 16:47

From mid-June to early November, I have spent 320 more hours working on package management. After having worked on the bootstrap support in HaikuPorter (which Ingo has already mentioned in his blog entry), I have spent some time reworking the way how Perl and Pythong organize their modules: Scripting languages with module support usually expect to be able to build modules from source and install them somewhere into the hierarchy of the scripting language. In our case, they can no longer just put the built (site-specific) modules into the "standard" folder, as that one is read-only. So the configuration of Perl and Python had to be changed, such that modules built from source are being installed correctly into a writable folder (in the non-packaged hierarchy). Perl already supports the notion of a vendor-modules folder, where packaged Perl modules will be installed (an example of such a beast are the perl modules contained in the git package). That idea has been extended to Python (such that it will automatically pick up the python modules provided by the mercurial package). Separating packaged modules from local (built from source) modules is an idea that should be followed by all (not only scripting) languages ported to Haiku.
The updated perl, python, git, mercurial and scons packages that are the result from that work have not yet been published, but I will do that during the next couple of days.

The package management branches of both haikuporter and haikuports have been merged into the respective master, i.e. all porting work done in there will produce packages. Several new 'testing-' branches will be added to the haikuports repository soon in order to be able to separate work on the bleeding edge from the work that tries to stabilize a given set of packages for a release.

Finally, I have started with work on the infrastructure required for automatic building of repositories and (at a later stage) packages. The first effect of this is that whenever a developer changes the version of a package used by Haiku's build system, the respective repository will be created automatically. Those repositories are being published to http://packages.haiku-os.org/haikuports/master, from where pkgman will download packages.

Summarizing those months of work: I think we have managed to put most of the work behind us, but we haven't reached the goal just yet. There's still more work to do and I personally will return to continuing that work in my spare time.

Syndicate content