Professional Sound API

don't know if you have seen this write up of cortex but you might be interested in the VST plug-ins information near the bottom and how the vst plug-ins and media kit can work together

http://betips.net/1997/09/09/fun-with-cortex/

I suspected as much. TT isn't actually the kind of audio application I had in mind. TT is more of an automation software that schedules the playback of audio files with some live mixing. Cool stuff, but not in the same boat as iZ/Tascam with their DAW or any other audio manipulating app.
TT has been running BeOS for the past 10 years. Migrating to Haiku should be relatively easy and is a sensible step to stay in business. Others would have to resurrect code that hasn't been touched for 10 years, while their money making apps have evolved on other OSs.

Regards,
Humdinger

oops. missed the "not"...
is there a reason not to name the company?

What about latency issues with the MediaServer?

Anyway, I'm in the process of porting (S)LV2, LADSPA, DSSI and other sound processing related stuff to Haiku, probably going to release binaries in June or something (LADSPA and DSSI SDKs and some plugins already working, but I have no time right now - damn university).

He is probably porting just the APIs for now, and will probably make media node adapters that allow you to use the plugins using this API in the media kit, this does not really make anything messy but allows the use of other plugins in Haiku thus avoiding rewriting them unless they are really desired... would be nice to have a open source audio processing workstation on haiku but that will take time and need someone to be dedicated.

Exactly.

To be more precise, I'm porting the LADSPA and DSSI SDKs (DSSI needs also libdssialsacompat, and that one is already done too), SLV2, some plugins and other stuff I wrote, including a compiler for a DSP language (compiles to LV2 plugin source code), a LADSPA-to-LV2 and a DSSI-to-LV2 bridge (a VST-to-LV2 bridge will come too sooner or later), so that there only needs to be a LV2 media kit addon (and this will probably be done by another guy from the Italian Haiku user group).

Regarding creating a Haiku-specific API, I would rather suggest the use of LV2: it's simple, decentralized, powerful and, above all, not UNIX-specific and extremely extensible.

Your analysis seems a bit off...jack is like a single cycle CPU with low throughput where to the user/programmer an instruction takes once cycle but the cycle must be much longer yes it has zero cycle latency but the cycles are much longer thus the low throughput really no CPUs are designed this way anymore because its stupid.

If Haiku operates as you describe it has an audio pipeline which spreads out the latency between each step instead of delaying the whole pipe while one chunk goes through as you described Jack

A pipelined audio system would IMO have better throughput with minimal degradation in latency especially on SMP processors which are widespread at this point and were obtainable back in the BeOS days though a bit more expensive. In fact the improved throughput would probably have the effect of reducing system load thereby reducing latencies...assuming high load audio processing if you assume lower load Jack should be able to keep up but will reach its limits faster from this perspective I see a pipelined audio kit as more robust.

I have no idea how these are actually implemented but your description doesn't make much sense to me. Maybe you can set *my* thinking straight if I have missed something or other.

t=1: input -> A
t=2: A -> B
t=3: B -> C
t=4: C -> output

while in JACK it would be:

t=1: input -> A -> B -> C -> output

While your diagram is OK you have to remember that T isn't the same in both of those most likely that jack's t will be very large and haiku's t very small

PS: I think the system refered to earlier is RADAR V ... which I believe can still run BeOS as an option (going from what I have read never seen one myself)

t is a unit of time as the DSP system sees it.
What I believe it means is that in JACK the whole chain is processed in one step then it happens again and again over time but the time step is larger.
In haiku, it passes the buffers through each node and then they give them back to be passed to the next node and so on, this means that each "time step" is smaller and the delay of moving the data from one place to another is spread out over the whole chain.
At least that's what I understand :/

Oh my, what a mess! I try to be as clear as possible this time, but this is the last time I try to explain things, since I was only interested in knowing what the developers think about this issue and I hope they know what I am talking about at least.

So, let's say that we are reading and writing audio buffers from/to the soundcard and each buffer represents x milliseconds of audio.

Now, let's consider we have a program which reads stuff from the microphone input, processes it and writes it back to the soundcard: each x milliseconds the sound card driver gives you the data which corresponds to the last x milliseconds of microphone input (so your input data goes from (now - x milliseconds) to (now)), you process it (let's suppose instantly for simplicity) and you write it back to the soundcard - ignoring other latencies, the soundcard can only start to reproduce the the processed sound from the moment you sent it, so from (now) to (now + x milliseconds). This means that the output sample corresponding to the input sample (now - x milliseconds) is (now), so we have x milliseconds of latency - which in this case is only related to the buffer size and the sample rate.

Now we move to the case of inter-application audio data routing: we have three independent programs, A, B and C, and a signal flow like: input -> A -> B -> C -> output, which means A takes its input from the sound card and outputs to B, B outputs to C, C outputs to the soundcard.

Once a system like JACK or the Media Server receive data, they let all nodes process some data, the difference is in which data is actually processed.

AFAIK, the Media Server does not care about processing chains, it just "calls" nodes in random order giving them as input the output of the previous node at the previous cycle, and keeping in memory their output for the next cycle.

If you do your own math, you will discover that each node in a chain introduces a latency which is equal to the amount of time corresponding to the buffer size (input of B is the past output of A, input of C is the past output of B, which correpsonds to the second past output of A).

In our case, since we have 3 nodes in a chain, the Media Server introduces an additional latency which is three times the time corresponding to the buffer size (so our software latency, excluding algorithmic latencies, is now 3 or 4 times the buffer size in time - I don't know whether the output to the sound card is actually the current or the past output of C).

JACK, instead, analyses the connection graph and does basically a topological sort, so that at each cycle, it can give to a certain node the data produced by the previous node in the chain corresponding to the "global" output which is going to be produced in this whole cycle.

So, it will execute, in order, A, B and C in our case, using the current output of A as input of B and the current output of B as the input of C, which means it introduces no latency at all (total software latency, excluding algorithmic latencies, is just the buffer size in time).

Now let's make some considerations:
1. JACK is probably more resource intensive that Media Server, so it steals some more throughput, but I have no measurements so I can't really tell.
2. The number of nodes executed at each cycle by both systems is probably the same (all nodes), unless they have some mechanisms which exclude from execution nodes which are not useful to generate the total output (theoretically there are ways to do that in both systems, but in practice it might even be harmful).
3. I/O buffer sizes are the same or at least comparable in both systems, since whatever systems, smaller buffer sizes correspond to less latency but also to less throughput (more interrupts or read/writes and more code executed in practice).
4. The threshold to hear separate sounds is somewhere around 10 ms, which corresponds to buffer sizes of 441 samples at 44.1 kHz, 480 samples at 48 kHz, 960 samples at 96 kHz, etc. In pratice the hardware (A/D, D/A, and other stuff) introduce even more latency, so you should typically stay at least below 5 ms of software latency. In JACK you can just stick to that, with the Media Server you should further divide your buffer size by the maximum number of chained nodes, which gives you worse throughput.
5. A nasty side-effect of introducing latency for each node is synchronization of different streams, and that IS important if you want graph-like connections, because the human hearing can perceive differences between sounds no matter how small, in time, they are (well, almost, but for practical purposes that's what it is).

Ok, let's see if I can make things more clear.

Ok.

Yes.

Well, I said I have no experience with the media kit itself, apart from resampling; my source of information is: http://www.haiku-os.org/legacy-docs/bebook/TheMediaKit_Overview_Introduc...

I don't know if media rosters can be chained themselves (doesn't Cortex do that?), but even inside one roster, connecting nodes in series creates latency.

Real world example: let's say I have 3 ms buffering latency and I'm using a guitar rack program which uses nodes for each effect. I want to apply 5 effects in this order: wah, distortion, phaser, flanger, FIR convolver - if each node introduces latency I have a total of at least 12 ms of latency. If I used JACK to connect individual application each doing one of these effect I would have had only the 3 ms of latency.

Why? Because JACK is capable of executing the effects in the "correct" order, while the media kit does not take the order of effects into account. So the media kit has to use past buffers for each effect to avoid processing bad data, and each buffer is 3 ms long.

Result: 12 ms is hearable, 3 ms instead is not.

I hope this is clear enough.

The overhead for JACK is really tiny, I'd be very surprised if it's as much as the Media Server in a like-for-like comparison, let alone more. Where possible JACK uses FIFO scheduling, so the OS scheduler is automatically running the next needed JACK thread(s) whenever there's work to be done, and it uses carefully aligned and wired shared memory so that everything is in RAM and there's zero copies, probably the biggest source of "overhead" is the choice of single precision floating point for PCM representation, but if you're doing anything really serious you have to pay that price anyway, so better to pay it only once inside JACK itself where they have highly tuned conversion code.

You might think the graph ordering problem would add overhead, but JACK's design ensures this can be done outside the tight audio loop where it isn't time critical - and without locking too.

The FIFO scheduler is potentially dangerous (nothing else will be scheduled while a FIFO process is ready to run), but modern Linux lets processes be given a "limited" privilege to run with this scheduler for a finite fraction of time. Because JACK is all about low latency a process that abuses FIFO scheduling to stay running too long is undesirable anyway, so the interests of JACK performance and system stability are aligned.

All this is great - so long as you're doing pro audio. The moment you've got people who don't know about audio writing the software, you lose. Such people aren't going to spend hours analysing their work in cachegrind and they certainly aren't going to worry if their code sometimes incurs a disk seek (typically 10ms each) when it's supposed to be emitting audio. Their approach is fine for listening to music while reading your email, but it's not pro audio, and in JACK what will happen is the xrun counter will start climbing and the program gets kicked out of the graph.

The consensus has long been that we aren't going to get general application developers to "pay their taxes" (to use Raymond Chen's preferred phrase) on pro audio APIs, so you will always want a separate "no compromise" pro audio system for doing real work because the system used by general application developers will have to accept relatively long latencies and unpredictable latency jitter from less than excellent software. From what I have read and understood about it, the Media Kit is the latter type of system.

Well it's always possible to stretch a point. After all, a ZX Spectrum is not what we'd generally consider a pro audio system, but a crowd of paying guests dancing for an hour to sounds coming (in part at least) from a ZX Spectrum is hard to argue with. I'd say if you care about low latency, the Media Kit is probably not a good choice.

But to be fair the primary criterion for most users, whether they do pro audio or just blog about skateboarding is always application availability. If you like Reason then an OS without Reason is no good to you, even if it has really low latency.

I DO think that using the Media kit for pro audio (extreme low latency) is not appropriate because it will inevitably introduce jitter and also because it does not allow direct hardware acceleration which is trivial because it is hidden by the Media kit API (please correct me if I am wrong).

I have some interestes in audio myself and I tried to think about this previously, and after reading the Be Book I found these conclusions

- Using Media Kit is only for non-time critical (no pro-audio) software, such like media players because of the overhead of the context switching, non optimsed sound routing calls (again correct me if I am wrong)

- Pro audio is similar in some ways to 3D graphics, it needs the fastest access to the device and the best hardware acceleration, so it needs some direct access API for sound, something like DirectSound, ASIO, etc .... So I found some answers not in the MediaKIT section but in the GameKit instead so let's see what it says here :
BPushGameSounds

Are used to let you fill buffers flowing to the speakers (or headphones, or whatever audio output device is being used). Their missions are the same, but their methods are different; BPushGameSound also provides a way to play sound in a cyclic loop, keeping ahead of the playback point in the buffer. This is the extreme in high-performance, low-latency audio, but does require some extra work on the programmer's part.

- My personal approach for a Pro audio system would be an implementation of Jack API in top of the GameKit which provides higher level API instead of using directly the BPushGameSounds and the BStreamingSound and it would be for the "careful and know-what-to-do programmers" and for the not so critical missions, there's the MediaKit.

My 2 cents

Hi all. First time poster here. I was a developer back in the BeOS days (Harrison used BeOS extensively in our products, along with the iZ, Tascam, and LCS guys). In all of these cases, BeOS was the control/automation computer and the heavy-lifting was done by DSP hardware. I don't believe any professional product ever used the Media Kit.

The Media Kit is very well designed (overdesigned?) for media playback and simple music production. It is not ideally suited for professional use because of the latency issues discussed before, the relative complexity of some simple tasks (such as plugin hosting and parameters) and the lack of features such as time sync, video sync pulse, netjack, etc. Of course all of these issues _could_ be incorporated into the Media Kit, and probably very neatly.

On Linux it has become fairly common for systems to use PulseAudio for desktop sounds and JACK for professional audio. I'm a big fan of the "cleanliness" of BeOS/Haiku so I'd prefer that Haiku didn't have this split. However, practically speaking, Haiku has the opportunity to gain a lot of developers and a lot of existing code if they adopt JACK. If they don't do this, then there is going to be a long and hard road to re-implement everything that has been happening in Linux Audio for the last 5 years.

DavieB, regarding your comment about a separate mixer: there have been many examples of this over the years on all platforms. BeOS had a simple software mixer that allowed you to adjust the level of apps that were playing back on the system. There have been mixer-only apps for CoreAudio, ReWire, and JACK. Ardour/Mixbus has often been used as a mixer-only system with no track playback. I'm even aware of people using Ardour and Mixbus as live mixing consoles(!) However in most cases, the inconvenience of saving the mixer state in addition to the various other programs makes it impractical. There is an effort to incorporate "session management" into JACK to provide just this sort of inter-operability. (yet another thing that the Media Kit will want to re-implement in the future...)

I could imagine a cool software mixer that automatically adds an input strip(s) every time you launch a new app. But you'd want to separate the "pro" apps (whose settings could be saved/recalled and automated) from the desktop apps such as the web browser.

Best Regards,
Ben Loftis

edit by admin: removed the URLs.

I was thinking about the issue a bit lately and I think I came to the conclusion (which anyway would be better discussed on the development mailing list with the developers) that an easy and clean way to get something working for professional audio on Haiku could be to have JACK as a node/roster/whatever in the Media server, so that simple applications may just use the Media Kit and professional apps could use JACK - it would be also preferable to be able to connect normal Media server nodes/rosters/whatever to the JACK's inputs/outputs (and maybe have some setting, maybe even dynamic, to specify how many JACK I/O ports one wants to have).

This as far as audio data goes and in the hope that direct connections in the Media server to soundcard I/O do not introduce latency too (if that's not the case, that could probably be changed without breaking APIs, I hope) and that Haiku is capable of makiing JACK work decently.

When it comes to MIDI, I don't really know; I've been told that the Media Kit has problems for professional MIDI usage too, but I actually have no idea.

Buffers are not passed in and out nodes by MediaRoster (no xxx returns to mr / mr feeds yyy). Nodes being connected and started, they perfectly knows their output(s). They simply hands the buffer to their connected output(s), which trigger a BufferReceived() event on the corresponding node(s) in the graph, which run in a different thread.
The BMediaRoster allows you to query, instanciate and control nodes, but is not involved in buffer(s) processing.

There is no main/central/looping processing thread managing the pipeline(s). As with many BeOS/Haiku areas, it's an asynchronous multithreaded model, where each node works in parallel (actually, on SMP hardware) and cooperatively via messaging to produce buffers in time to perform them when due.
It's very visible in ProcessControler: the media_server doesn't do that much work, instead the load is spread in several threads, in media_server_addon's ones for system media nodes like mixer, physical nodes, in client apps (Media Player, CortexAddOnHost, Clockwerk...) for others.

The timing design is kind of reversed compared to JACK: it's a performance time driven data flow, not a process-driven cycle. Also, JACK design focus mostly on live audio, while Media Kit design don't/can't.

I dunno if Media Kit design is up for Pro Audio, but when I consider that JACK is not an end-user audio-video framework (GStreamer is more in par with Media Kit here), that its design is optimized for live audio but had to evolve from single RT processing thread model to a dual thread model to actually benefit from SMP system, had memory locking and RT memory alloc support to avoid VM swapping, I find more similarities in both frameworks than one could though at first.

The fact remains that nobody will knows what worth Be's Media Kit design and Haiku re-implementation until one actually try to stress it as a Pro Audio solution will.

Any volunteer?
:-)

Dude I have done research with this latency doesn't even get to perceptual till 150+ msec.

But belive what you want.

12msec is

0.00012 seconds