The Informal Summer Gathering - USB HID, Filesystem bug-squashing and Media Kit encoding

Blog post by stippi on Sun, 2009-08-02 18:49

After I didn't write the promised report on the last Coding Sprint which took place after BeGeistert in April, I am now trying not build up an even bigger lag. Last week, Axel and his girlfriend Claudia hosted Michael, Ingo and myself at their nice home in Hannover, Germany. Oliver could sadly not attend our small, relatively spontaneous and informal Coding Sprint due to sickness, although he seemed to be with us in spirit considering all his ICU commits. (The ICU libraries are an important foundation of the forthcomming Haiku Locale Kit.)

Hm... where do I start? Basically, I could chose to report on three things: What we ate, how we slept and what we coded. Not much else took place, although I can tell you that we did have nice weather from my short trips outside to fetch bread rolls for breakfast (at 2 pm). The first couple days I wasn't fully up to speed with the coding and concentrated mainly on bug fixing and polishing. Ingo had brought me the forth in a series of books he introduced me to and I was a little occupied. Michael jumped right into coding on the HID driver, which is a generic driver for all sorts of USB input devices. I am happy to report that my wireless USB mouse/keyboard combo is now working (mostly) with his new driver. Obviously a driver supporting many more input devices is something that is very critical for Haiku.

One thing that we really wanted to focus on during this sprint was Haiku's stability and most importantly any remaining file system problems. So I brought my hard drive which showed two of such problems. Given reproducible test cases, Axel was able to fix them pretty quickly. One bug was a problem with double indirect blocks, which are needed for larger files when your volume is somewhat fragmented. The second problem was actually in the VFS layer and had to do with directory iteration and error handling. The actual corruption on my volume, which triggered the VFS problem, could have been caused by one of the bugs which were already fixed sometime earlier.

At the same time, Michael's notebook served as a great resource for triggering all sorts of kernel problems while he tried to get some work done. Then he would hand it over to Axel and Ingo for them to debug the problem while he was trying to continue on his EeePC. One of the problems he kept running into were spontaneous reboots, which happen after a so called triple fault. A triple fault is something that is not supposed to happen and hints at a buggy double fault handler in our kernel. A double fault handler is basically a mechanism to deal with the situation that a second fault happened while you were already trying to handle the first. It's certainly tricky stuff and Ingo, Michael and Axel found a number of problems in the kernel when they reviewed code and the respective hardware documentation. Ingo already made the first commit which works towards fixing the situation for good.

When Axel didn't have bugs to fix, he sort of worked all over the place, for example he improved workspace handling and configuration and also converted the Screen preflet into using the layout management. When Axel and Ingo were looking at the "system freezes for short moments" problem, which could be witnessed when Haiku performs a lot of disk activity, they had a nice idea for improving Ingo's DebugAnalyzer application, and Ingo worked on implementing it for a while. It's basically a page where you analyze the scheduling recording data and see all the threads which were running at the time along a zoom-able time-line. Threads show up green when they have been running, orange when they would have been running, but were preempted by another thread and red when they wanted to run but there was scheduling latency. This helps to see problems where for example the real-time thread which feeds audio buffers to the hardware was supposed to run but had to wait for some resource, which then caused audio hick-ups. It's really neat stuff, although a lot more information is still available from the scheduling recording data, which is not yet presented in the interface. Once done, this will give a very precise overview of what each thread in the system was doing at any given time, and how they affected other threads.

One of the nights, after having had lunch, we didn't hurry back to the keyboards quite so quickly, and kept talking a bit about the Alpha release and Haiku's future. We came up with a list of features we definitely want to see at some point, and so we created a Wiki page FutureHaikuFeatures. It was fun to sit in this round and talk about the future and how things could develop. But after this lasted for a short while, everyone's fingers began twitching and so we hurried back to our keyboards.

The last two days of the Coding Sprint, I had built up enough courage to finally dive into Media Kit encoding support. In the end, it was quite a bit less work than I anticipated all the while, especially with such a powerful foundation as the FFmpeg libraries. The code that Marcus and others had already written was in good shape and I could find my way into it fairly quickly. I took most inspiration for the Encoder API from the BMediaEncoder class, which seems to be a pretty direct front-end to the actual Encoder plug-ins. The Writer API was a bit of guess work, but nothing very hard at all. I only had to figure out how to best support all the existing API functions, which meant that I needed to decide which data the media_server needs to maintain in order to match suitable plug-ins for applications to use for a given media format. It's still not perfect, but good enough for tools like MediaConverter, Clockwerk and eXposer. I couldn't help but keep smiling when I finally rendered my first video in Clockwerk. On the train ride back, I also implemented audio encoding, although there are some remaining issues to take care of. If Berlios wouldn't be down right now, I would have added MediaConverter to the image already and made an optional package for Clockwerk.

Of course... you want to know about the Alpha. We have talked about it a lot and I am happy to report that we don't consider any remaining issue serious enough to hold it back any longer. We just don't know what to do with the IDE versus ATA stack. On the one hand, the ATA stack doesn't have some problems of the older IDE stack, but Michael himself said that the new ATA stack is actually slower. And then it has some new problems of it's own. Tough call, but as soon as we have made up our mind, I think we are good to go! Exciting times!

I want to close with thanking Axel and Claudia for their hospitality! We had a great and very productive time. Also I want to apologize for reporting on Michael's, Axel's and Ingo's work perhaps in less detail than it certainly deserves! For example I forgot to mention Michael's great work on Haiku's BMessage implementation and there were a lot more things we worked on during the week. It's great to be part of this team, striving towards a common goal. I hope these coding sprints encourage more developers to help polish Haiku and code the necessary features to make it the best OS out there! I know that many such efforts are under way as we speak. Keep having fun, everyone!

Comments

Re: The Informal Summer Gathering - USB HID, Filesystem ...

I must say, you guys did some awesome work this week :) Regarding IDE vs ATA, I would probably say to go with ATA since at least from anecdotal observation it seems to more reliably boot/work on a greater variety of controllers, even if it might be slower for now.

Re: The Informal Summer Gathering - USB HID, Filesystem ...

anevilyak wrote:

I must say, you guys did some awesome work this week :) Regarding IDE vs ATA, I would probably say to go with ATA since at least from anecdotal observation it seems to more reliably boot/work on a greater variety of controllers, even if it might be slower for now.

Yes, great stuff guys!

I have to agree with Rene - on several occasions we have guided visitors in the IRC channel to switch over to ATA and it has solved their problems - most recently the IDE stack was conflicting with the EHCI controller for an individual and hanging at boot, but when he switched to ATA, it worked flawlessly.

Until we make the switch official, we won't know how many incompatibilities we'll run into - I suspect it will be few.

- Urias

Re: The Informal Summer Gathering - USB HID, Filesystem ...

I agree. Reliability is more important than speed at this moment.

It's great to see you made so much progress on Haiku. Thanks a lot.

Re: The Informal Summer Gathering - USB HID, Filesystem ...

Keep up the awesome work guys - I love reading about the coding sprints, reminds me of similar (unfortunately non-Haiku) coding sessions while I was at uni.

Just remember, everybody very much appreciates the effort you put in!

Re: The Informal Summer Gathering - USB HID, Filesystem ...

stippi wrote:

Of course... you want to know about the Alpha. We have talked about it a lot and I am happy to report that we don't consider any remaining issue serious enough to hold it back any longer. We just don't know what to do with the IDE versus ATA stack. On the one hand, the ATA stack doesn't have some problems of the older IDE stack, but Michael himself said that the new ATA stack is actually slower. And then it has some new problems of it's own. Tough call, but as soon as we have made up our mind, I think we are good to go! Exciting times!

Well done, gentlemen! I always look forward to news from the coding sprints because it's always really encouraging. You guys are, as always, amazing!

Re: The Informal Summer Gathering - USB HID, Filesystem ...

Thx for good information but does it fix ticket #3150?

Re: The Informal Summer Gathering - USB HID, Filesystem ...

IIRC, it's not clear if #3150 was perhaps fixed already. Before the Coding Sprint, Axel fixed a bug that could have caused all sorts of problems. But Axel himself would have a better idea. The file system bugs that were fixed during the week would not have caused #3150 in any case, but they could cause problems with already corrupted volumes during normal use, or when trying to fix them with checkfs.

Re: The Informal Summer Gathering - USB HID, Filesystem ...

Terrific work guys. It's really appreciated.

Re: The Informal Summer Gathering - USB HID, Filesystem ...

very good wok guys ! alpha is on the road, almost avalaible, it's a dream which become real :)

Re: The Informal Summer Gathering - USB HID, Filesystem ...

Fantastic work, guys! Awesome!
Now, could you just post some comments from Axel's Claudia? I can only imagine what she thought of 4 guys sitting in front of their screens for more or less a straight week. I envision her pulling back the curtains after 4 days and poking all of you with a broom into the shower while you cringe and hiss B_DONT_DO_THAT. :D

Re: The Informal Summer Gathering - USB HID, Filesystem ...

Well, thanks for a 10 year old operating system. And now? Within five years the beta? And then - after another 5 years: HAIKU IS RELEASED. Nobody is going to use operating systems then, but hey, who gives a damn ;)

PS: Yeah, somehow it's great - but somehow it's so useless ...

Re: The Informal Summer Gathering - USB HID, Filesystem ...

deb2006 wrote:

PS: Yeah, somehow it's great - but somehow it's so useless ...

Usefulness is in the eyes of the beholder.