UVC Driver: Final Report GSoC 2011

Blog post by gabriel.hartmann on Wed, 2011-08-24 21:53

The Google Summer of Code for 2011 is over now for me. The final state of the UVC driver project while very far from perfect is at least at a point where incremental improvement can be made. Literally the day (maybe 2 days, depending which timezone you're in) before the "firm pencils down" date I finally managed to get data all the way from the camera to the screen. The decoding of that information is totally wrong at this point, but coloured pixels show up on the screen and they appear to react when things move in front of the camera. Success?!

Beyond the relatively simple task of writing a colour decoder that actually matches the format of the data being provided by the camera, there remains a whole lot of work to be done. The work comes in three major categories: the application layer, the driver layer, and the USB layer.

By the application layer, I mean the interface between the driver and applications. The two major webcam applications for Haiku Cortex and Codycam both primarily request and accept frames at 320x240 resolution. Allowances have been made in the driver for this behaviour. The current code should allow other resolution requests to be negotiated successfully, but without application testing, it's likely that resolution negotiation will have some bugs. Also this tendency on the part of applications ignores camera preferences for default resolutions.

At the driver layer the UVC driver does not even attempt to support all the features of the specification yet. In particular no attempt has even been made to deal with compressed video frames. The MJPEG frames that uvc cameras normally provide haven't yet been touched. The specification also allows for other proprietary formats. Again this is unexplored. Finally, the driver to camera resolution negotiation is not particularly robust and assumes to some degree that applications are going to request reasonable resolutions (from the camera's perpsective). This portion needs to be looked at again. This will be a time consuming process because camera stalls in negotiation lead to unrecoverable crashing of the media_addon_server. A camera stall occurs on the slightest provocation. Learn to love rebooting.

At the USB layer, ehci isochronous transfers tend to crash Haiku into KDL. There are probably a series of reasons for this, but at least one known issue is that ehci doesn't deal well with a large number of queued transfer requests. For this reason, there is currently artificial waiting time ( snooze(for a for a little while) ) in the generic CamDevice so that the queue doesn't get to backed up. Of course this makes a decent frame rate impossible. At least the more or less random pixels now displayed don't appear to have a very quick frame rate. Even with the artificial delay you only have about 20 seconds of video frames (if you're lucky) before crashing into KDL. Again, learn to love rebooting. There is a newer version of ehci available which deals with this issue, but I couldn't get it to play nice with my system.

In short, to use a transportation metaphor, a trail through the jungle has been beaten from camera to display. Technically one can get from point A to point B. However if you want to do this easily, quickly, and reliably you're going to need a super-highway. Expansion and improvement of the route is needed at all levels.

I had a great time with this, my first GSoC and learned a lot about USB which was something I didn't really know a lot about before. I'd like to thank my mentors Jerome and Philippe for their time and patience.

UVC Driver -- GSoC Three-quarter-term Report

Blog post by gabriel.hartmann on Mon, 2011-08-01 20:25

Not so long ago, at the half way mark of the GSoc, I was optimistic that I was near to actually interpretting data from the camera in such a way as to produce images on screen. I was successfully grabbing payload data from the camera, the camera's in-use light was on, things were looking good. Since that point, progress has been repeatedly stalled by strange and difficult to debug behaviour.

The first problem which I encountered was that the vast majority of the data received from the camera consisted only of headers with very little actual image data ever attached. In a few seconds of contact with the camera as an example, only 6 out of 7714 packets (0.000778%) contained content. That was worrying. Furthermore the bit indicating an error was often set in the headers, although not perpetually. Investigation of that problem has revealed one of two possible scenarios.

1) Protected content -- This situation occurs if the data source device detects that the video or still-image data is protected and cannot be transmitted. In this case, empty packets containing only headers will be sent for the duration of the protected content.

2) Input buffer underrun -- If the data source device is not able to supply data at the requested rate, it will transmit empty packets containing only headers for the duration of the buffer underrun.

There are other possible error modes, but these are the only two described in the specification which indicate the empty packets with only headers behaviour. Both I and my mentors believe that the second scenario is most likely. However we do not know for certain, which leads to another problem.

Requesting information about what error mode exactly is being encountered of the camera, should be very nearly in exactly the same format as the probe / commit routine which directly precedes the initiation of data transfer. However, whenever the camera is asked for further information about the error mode it is indicating, both the media_addon_server and the camera lock up. Now, it is far from abnormal for the media_addon_server to lock. This has happened countless times during development. But the camera has never locked up befor. Even after a restart, the camera remains locked. A full power cycle is required to bring things back to relative normal. So that's one major intractable issue at the moment, but it is not the only one.

Another major issue was an essentially infinite loop in the VideoProducer's Datapump thread. A division by zero was causing a very large negative number to be included in a sum. The result of this sum was supposed to be a time in the future for which a semaphore grabbing loop should wait. However since the time was hugely negative, at no point could a constantly positive and climbing system time reach this "future" date. Investigations into this issue were going on at the same time as I was investigating the empty packets error. Because my camera was locking up, I had been unplugging and replugging my camera repeatedly. In order to make this easier I had started using the usb plug on the front of my machine. This movement has, I believe, solved the division by zero problem. I haven't done rigorous testing yet, but it was a consistent problem before, and now that I've switched usb plugs, it has gone away. Do you see what I mean about strange and difficult to debug behaviour?

Finally, in an effort to see if the empty packets problem was confined to the beginning of data transfer, and perhaps worked itself out later, I began watching printouts of actually full packets being received. After about 20 seconds of this, Haiku reliably crashes into Kernel Debug Land (KDL). I'm now looking into kernel debugging advice my mentors have given me. Just judging by the kernel debug messages, it looks like an inappropriate NULL pointer problem in the ehci isochronous transfer code.

So, while it would be nice to be writing a deframer or some code for negotiating resolution between the user applications and the driver, with a view towards getting images on the screen at user requested qualities, I think it's probably more important to get the back end working properly. There are only about 3 weeks until the firm pencils down date and I haven't entirely given up hope of one day seeing images, but at the very least I'll hope to have made thorough informative investigations into a series of problematic behaviours that would certainly not be acceptable for end-users to encounter.

UVC Driver -- GSoC Midterm Report

Blog post by gabriel.hartmann on Sun, 2011-07-10 23:31

Since my last blog entry a lot of progress has been made. Currently I'm right on the cusp of actually producing images on the screen that have been captured by my camera. Successful communication is occurring between the driver and the camera in at least two different forms.

The first form of communication to be successfully implemented involves the setting of values within the camera which affect image capture. These are the familiar brightness, contrast, sharpness etc. settings which most cameras support. Nearly all of the options available to my camera are now presented for manipulation by end-users and successfully communicated to the camera. These values are maintained within the camera between power cycles and this fact is successfully communicated to the user via the available controls. The controls can be viewed and modified in the media preferences application or the cortex demo application. ParameterWeb control documentation indicates a range of different style controls within the continuous or discrete parameter varieties. However it appears to me that the only discrete value input method currently supported with an appropriate gui interface is the binary on/off option. This is suitable for features like the auto value for white balance which can only be either on or off. However the powerline frequency setting which has three possible values was unable to be represented with the appropriate discrete control of the B_RECSTATE type which has three possible states. To simulate this capability a continuous control was modified to only allow three values which are indicated by placing the sliding control within a +/-10 range of the desired value. The slider snaps to the available values to indicate this behaviour. One future feature which would perhaps be desirable is controls with auto settings which indicate in real time by their movement what values the camera is using in its auto mode. Right now sliders are simply frozen in their last position when the auto mode is in effect. I had some brief discussion with my mentors about this feature, but it was deemed to be unnecessary at this stage as a lot of work is left to be done in actual image capture.

UVC Driver -- GSoC Quarter-term Report

Blog post by gabriel.hartmann on Mon, 2011-06-13 20:45

On June 7, I turned in my dissertation and my semester ended. On June 10, I had my first final exam. Now it's time to produce a progress report for Haiku. Almost miraculously, I've actually managed to squeeze some Haiku development time in and am making progress of a kind.

The most tangible progress is mostly in the form of debug messages and crashes into Kernel debug land, but I consider anything that I do which has a measurable effect to be progress. So far all my efforts are aimed at trying to understand how all the different pieces of a userland driver component come together. I've had a lot of help with this from my mentors and from people in IRC and on the haiku-development mailing list. Michael Lotz and Anevilyak (IRC nick, I don't know a real name) were particularly helpful.

My current understanding of the task is this:
1. I should detect the connection of a camera.
2. I should detect its capabilities.
3. I should expose those capabilities in a MediaNode.

In reality two and a half out of three of these have already been done for me. The USBKit is successfully detecting my camera. Furthermore listusb lists all the relevant interfaces, configurations, and endpoints the camera provides (there are many). In a similar way, a first draft of the UVCCamera driver (written by Jerome Duval) is also parsing this information and placing it in appropriate data structures. These data structures even make their way to encapsulation in a MediaNode which is visible to the Media server...sometimes. No capabilities are presented to the media server and CodyCam still claims that no video device is available, but I have seen something named "USB USB[sic] Video Device" in the Media preferences menu. However the behaviour of the node is a little strange. It's appearance is not stable at all. It tends to appear on an initial media server restart, then never appear again. This isn't always the case, so it's not exactly a reproducable bug which is irritating. Also attempting to test code by plugging and unplugging my camera or even a flash drive is a bad idea as it leads to Kernel panics and trips to a frozen Kernel debug land.

I do not yet understand the precise journey information takes from its extraction in the current UVC driver back to a Video Provider MediaNode. I know that a VideoProvider constructor takes a CamDevice. I know that a UVCCamDevice extends CamDevice. So presumably passing a UVCCamDevice to the VideoProvider constructor is how a MediaNode is created. I even see code which could do this in the USB webcam add-on directory, but I'm a little fuzzy on the exact set of operations which lead from the current UVCCamdevice code back to MediaNode creation. I'd like to understand exactly how that works.

Now that I've got a clearer if still hazy view of what's going on and what's needed I have some short term goals. First of all I'd like to understand exactly what a VideoProvider node needs in order to expose capabilities to interested applications. So far all my reading of code and documentation has focused on USB, but I really need to understand what the ultimate goal is before extracting data from the camera. Second (or perhaps in a shared first position), I'd like to not only detect camera capabilities but actually invoke a command which returns some data. I understand that Jerome Duval has done this recently and I'd like to either reproduce what he has done or extract some new kind of data. Third I'd like to actually expose whatever capability I am able to extract to the media server. My thinking is that if I can expose a single feature from end-to-end then the repitition of this task for more features should be fairly straightforward.

I'll be spending all day today working on the first two goals. Tomorrow I have to study for an exam as it takes place on the day after tomorrow. Then I've got a long break before my last exam in which to focus on the UVC driver. I feel like I am starting to get a handle on the structure of the problem and am able to poke and prod the current code to learn new things. That makes the problem solvable. I suppose my biggest medium-term worry is how to deal with compressed video streams. I don't know what decompression facilities are available or if they're applicable to this situation, but that's a problem for later.

A USB Video Driver for High-end Webcams (GSoC Proposal)

Blog post by gabriel.hartmann on Mon, 2011-05-02 03:04

As part of the Google Summer of Code I'll be working on developing a driver for Haiku that allows for the use of high-end webcams. By high-end webcams I mean in this case those which adhere to the USB video device class (UVC) specification. Preliminary work will involve bringing Haiku's support for the Enhanced Host Controller Interface (EHCI) to a point where UVC driver development proper can begin. Understanding the state of EHCI support and what work needs to be done in order to begin UVC development is my major goal for the community bonding period.

UVC development will entail the detection and exposure of camera features via Haiku's media kit. This will require (if I understand correctly) the production of a node with an attendant ParameterWeb which will hold the actual feature definitions. Then ideally any interested application will be able to issue commands to a UVC compliant camera and receive back appropriate responses in the form of image frames or video streams in various formats and resolutions, or status reports depending on the camera. The primary test camera will be a Logitech Quickcam Pro 9000 which supports a fairly wide range of resolutions, contains a microphone, and has a hardware button (presumably for taking still photographs). I have also noticed during the course of some computer vision research with the camera that it has what appears to be a hardware driven exposure compensation feature. There is also a similar feature exposed through the Windows Logitech driver software, but when this is turned off some exposure compensation still occurs. It will be intersting to see whether this feature is genuinely rooted in the hardware or is a result of hidden propietary Logitech software.

Syndicate content