cortex

Cortex is basically a front end to the BMediaRoster, it allows you to connect BMediaNodes, so what you can do is up to you. In theory every ported program could be interfaced with the media_kit.

http://betips.net/1997/09/09/fun-with-cortex/
http://birdhouse.org/beos/byte/22-media_kit/

There are poor informations just because no one is writing about it, you can find various media_addons in haikuware...so go ahead and document it!

wouldn't it be up to the nodes' developers to handle such cases and document them?

cortex could use some more functionality, though, i agree -- such as making an attempt to open nodes that aren't already open when nodegraphs are reloaded, so it could be like loading patches in puredata; or something more like an interface with haiku's scripting, so that loading a graph wouldn't require actually opening cortex. modular collections of software could be distributed and loaded, and connections between each segment could be altered by users -- so, if someone enjoys an editing software but hates the sequencer in it, they could patch in a different one.

near as i can tell, the one reason it never quite caught on is that proprietary software isn't conducive to modification, but with as much as has been going on in the realms of creative coding and open media, it's probably time.

i think of cortex as akin to puredata and nodebox, which are graphical representations of relationships between objects which are themselves written in c and python, respectively. here, we can write them in c and c++. by "we" i mean third party developers -- there's no operating system on earth where it'd be considered fair to expect the os devs themselves to build media software beyond basic media players.

Not true. The Media Kit offer no help to handle 1:N, N:1 or whatever N:M connections, but it doesn't forbid it either. The default mixer node is the best proof that it's perfectly possible. What missing in the kit is some support class(es) to make multiple (even better, dynamic)inputs / outputs handling easier to implement.
The Media Kit "connection" defines a link between an output source and an input destination. Aka actual working ports, not a pair of "logical ports".

.
Which won't mind anyhow, because this stuff is clearly a developer level issue, not an user one. See above why I agree that, from developer point of view, Media Kit is kind of too raw on this topic.

But Media Kit is not a digital audio framework. It's a multimedia framework.
This make a lot of difference, in particular in the shortcuts your data flow system can take or not, in forced synchrocity or not.

True. It was never intended to be more than that.
Actual multimedia applications does the media kit nodes graph assembly all by themselves way better and way error-proofer. Check Media Player source code.

It's disingenuous to assert that the Media Kit doesn't "forbid" M:N connections. It does, the connections defined by the Media Kit are specifically 1:1. The Mixer "gets around" that by spontaneously creating new inputs for each connection and redirecting the connection to the new input, a crude hack that's no replacement for actually being able to connect things together arbitrarily.

The other side of the mixer doesn't have the same trick, so you can't connect the output more than once. If you need the output of (anything) in more than one place, too bad, write yet another node to duplicate the signal.

The audio adapter smashes everything into the least common denominator. I'm talking about e.g. taking a stereo input and turning it into a left output and a right output, or a centre output and a difference output. Or vice versa. For which today you would need to write another node.

Well yes, but what I was really getting at is that graph tools (like Cortex) to manage them are a small niche. Most Jack users don't spend much time fiddling with its graph tools, they may only run one Jack program most of the time so the graph would be rather empty.

I don't see how the Media Kit could do more than what it already does with regard to multiple inputs. What would you expect it to do ?

Maybe you want a variable number of inputs, all with the same purpose, like the mixer node.
Maybe you can handle just 2, all with the same purpose (for example in a ring modulator, which multiplies two sound waves), or maybe two very different ones (let's say something that overlays subtitles on a video, would get one text and one video input).

The same thing happens on the output side, you may want all the consumers to get the same data, or you may want them to receive different parts of it. The system and the media kit don't know, so, they leave it up to you to manage this the way you need, in your own node.

I may want a mono output connected to two different nodes with a mono input for example. Or i may want two output mixed into an input automatically, i don't think the system should force you to use mixer nodes to do that. Additionally, one may want to separate a stereo input, to process the left with a node and the right with another, then i would like the output to be returned to the audio card as a stereo connection. Those are facts, limits of the media_kit. Actually it's not clear for me which solution should be take to go over those limitations, but i can think of a more specialized set of nodes making more easy for the programmer to handle basic situations. Or alternatively the concept of media_input and media_output might be extended a bit to handle such situations.

The main problem is that as said the media_kit is a lot generic, but video and audio have different needs. I'll take the previous example, connecting two video outputs to one input should not have the same behaviour of the audio case. So that's why i was thinking to more specialized nodes, it leave anyway the possibility to inherit the node from BBufferProducer/Consumer, but it would make more easy for plugin developers, for example, to have more complex but common behaviours...

What make you think i'm not playing?

But what do you mean by "sum them" ? it's a bit simplistic ! Do you mean buffer1+buffer2 ? This doesn't make sense.

You can't mix sounds the same way you mix videos or MIDI events. You have to tell the media kit how you mix the 2 buffers, and that's the role of the node.

Maybe jack don't show you this node behind the UI but something has to do the job.

In PCM audio, for each frame you simply add together the relevant sample from all the inputs. If you're using a signed integer representation you should use saturating arithmetic, like a DSP, not the wrapping arithmetic provided by default in most CPUs. For floating point it's unimportant.

Over the history of personal computing there have been many non-audio people who thought like you, they came up with some very amusing algorithms in this area, all wrong. I particularly like the solution chosen for the Amiga in the 1990s where if you play some music and then also play silence, the music gets quieter. Wrong, but they were trying so very hard.

You can read the JACK source code for yourself, it's a simple addition and it's performed automatically by JACK itself. JACK made a bunch of design choices which make this (and other important things) easier to get right, but the fundamental concept isn't hard.

our mixer is not just blindly adding streams together. What if they have different bitrates? The mixer node takes care of this, it will also resample (and there are two algorithms implemented to do that with different levels of quality).

You are ignoring what I said: what if you want to implement a ring modulator ? Getting the sum of two signals, when what you wanted is to multiply them, is utterly useless. So your suggestion is restricting what the media kit could do. And I'm still talking about sound streams here.

The same apply on the output side: what if the nodes you plugged as outputs don't expect the same format ? There's negociation between nodes to agree on a common format. While this happens with pairs of nodes, it's easy to find a solution. When it happens with multiple nodes on each end, you really need some kind of mixer or splitter node in the middle to handle the format conversions and make sure every node gets input in a format it can process.

Note this doesn't increase latency more than what you suggest: in your scheme "something" would still have to do the mixing and copying. Why not use nodes for that?

There's already a mechanism which provide format negotiation between nodes, so actually from this point of view the selection of which format will be accepted may be leaved to the node. Or alternatively i imagine node X which is connected with Y using floating point format, when you attempt to connect the node J to the same input as X, the format should be the same. This is actually a problem resolved from the begin in jack, since it's using the same format for all clients.

About the ring modulator, from the user perspective if you connect two nodes to the SAME input, they will be mixed up and will come to the destination node as a single mixed stream, if you connect two nodes to two different inputs of the same destination node, it will take care of what to do. The system mixer may mix them up, your node may sum/multiply etc. Some other node may just deny multiple connections to the same input and multiple inputs. The power is untouched. This is a explicitly behavior which i think in the audio is something natural and expected by the user. This also obsolete my thesis of having a multiple input open every time.

From the latency perspective, and also assuming that devs make well-fined rules on how this shoud be handled by the media_kit, it will add just the latency to mix the frames if the format is restricted. It will also remove the latency of the mixer / adapter node, which should be not underestimated. If you have 3 nodes it is ok, but what if i have considerable number of nodes and a considerable complexity in the graph? I have to run a lot of unneeded nodes.

But consider the advantage from the user perspective, it will remove the annoying thing of opening nodes, connecting them etc. This also leave some room to think about a little but very flexible set of BMediaNodes provided by the api to manage different situations. The fact that in video mixing up isn't the same as audio, is basically limiting the potentially power of media_kit in the audio level. The same apply for interleaved stereo streams.

I'm actually running a project, that i hope to release at the end of the month, which is strongly related to those things. So i maybe wrong on some things, in any case i want to try myself.

No. Sounds will actually "mix" together as described without a special mixer. You can try it for yourself, stamp your foot and clap your hands. The sounds are added together, you hear a combination of both and you don't need to buy any fancy equipment, isn't that amazing? It turns out that what's going on here is a physical property of sound waves, and in fact it's the foundation of the PCM encoding that makes your PC audio work.

Closer. One Mixer node per input, then the Actual Media Node, then one Splitter node per output.

Goodness no, as we saw at the start of this reply this simple "mixing" doesn't inherently incur any latency at all. Now, on a personal computer the addition step does take (minute amounts of) CPU time, and it can happen rarely that this is enough to push us to use a longer period (buffer size) and that will increase the latency, but most of the time it's a drop in the ocean.

Now, if we need to wake up a third party process to do the addition, then that certainly incurs more overhead, making it more likely that we'd use a longer period, so that is a reason to avoid this approach.

What happens when connecting two nodes with unconmpatible formats actually? The connection can't be made. The same is for multiple connections, no difference. This is resolved by imposing more restrictions since the case of multiple connections is particular. Where is the limitation? You still have the actual behavior, i.e. 1:1, but plus there's the possibility to automatically mix frames or split when doing multiple connections, N:N.

I don't think it should be necessarily a hidden magic thing, there are a lot of ways to expose controls and settings for the user.

This is hilarious, where i wrote that mixing is zero latency? : )
Are you really reading what i wrote? There's not anything which is really zero latency in the world, except quantum correlation, maybe.

Anyway, can you demonstrate that media_nodes are not increasing the latency? And the resulting overhead? And the throughput? If you can do that, i'll retire my idea and follow your thesis. At some point i'll come out with benchmarks, and we'll see the truth, before this moment you can't be sure of what you are saying, neither me.

Anyway, with nodes or not, the situation should be cleared.

No if it's managed by the media_kit in a smart way. Why the opposite? What you can do actually with the media_kit is preserved at all.

No if you need another mixer you instantiate it, connect the two nodes to the mixer, and the mixer output to the final node. I don't see the point really.

The conversion can be lossy in that case (say you convert from float to uint8 because the consumer node only accepts that). I'd rather know this before I notice the output of the node chain is completely garbled and aliased. Same thing when you need to resample because two nodes operate on different frequency ranges. Making this happen "automagically" has consequences deep inside the Media Kit concepts. The most obvious one is it makes the Media Kit knows about the data formats. It has to know that the streams are audio, video, or something else, which format they use, and so on. It means the Media Kit somehow makes the decision for you, to pick a format on each side and instanciate some kind of converter. How does that work? Pretty much exactly like our existing mixer node. It takes buffers in one given format on one side, and outputs them in another format on the other side. Making this a separate node or some invisible part of each node is a minor design decision at this point. With the current Media Kit solution, you get a more complex node chain, but a very simple API for each node. When you integrate the mixer into each node, whenever you instanciate a node, you have to ask yourself several questions. Is it a "sound" node? If so, I have to configure the mixer settings. Do I have a single input so I can bypass the mixer? Wan I agree on a common format with said input? Will I use interpolation to do the resampling? Is some kind of filter needed after or before resampling to avoid or limit aliasing?

You see this would make the API and the process for writing and instanciating/configuring a node a lot more complex. Jack avoid all this complexity by picking a single common format, which makes things a lot more simple, but restricted to a single bitrate and sample format for all the nodes. Forget about using their system for anything else than audio in the system-defined format. Now that's a big restriction to what you can do.

NoHaikuForMe keeps saying mixing is for free, in the message just above yours. I'm sure you have better understanding than he does, having actually written some code and played with Jack and/or the Media Kit.

So, here's why mixing does add latency: while adding sounds in the real, physical world isn't much of a problem, we only have one (or two) speaker and only one DAC connected to each of them. This is a hardware limitation. So we can't add waves in the physical world,which would indeed be the simplest and fastest way of mixing them. Instead, we have to do this in the digital side of things, before the DAC. We want to do this at arbitrary places in the signal processing chain - or, in your suggested model, at the input of every node. The only way to do this is to rely on plain old software to do the mixing. The way it works (assuming everyone works in the same bitrate and sample format to keep things simple) is to read one buffer from each of the inputs, and for each position in the buffer, sum up the samples from each input, and place the result at the same position in the output buffer.

There is a problem with that: you can't keep blindly summing things up or you'll quickly encounter some saturation. If your inputs are in the range (0..4095) (it'scommon to have 12-it DACs, 24-bit or 48-bit is also possible but let's take 12-bit as an example). So, if your inputs are in the range 0..4095, and you sum 4 of them, the result can be in the range 0..16383. It now needs a 14-bit value to store it. In the end, if the DAC on your hardware is 12-bits, you have two choices: either you ignore the problem and you get very bad saturation, or, you divide this by the number of outputs to keep everything in the same range. This latter solution is probably what the software you used on Amiga did: (some wave + silence) / 2 = some wave / 2.

In any case, mixing involves manipulating buffers line any other signal processing, and things that come for free in the physical world are often not so easy to replicate in the software one.

I don't like the idea of making the Media Kit aware that some of the streams may be audio and these may have implicit mixing. I think this can be done either with explicit instanciation of mixer nodes (our current solution), or inside the "audio" nodes themselves, without having to modify the Media Kit at all. There could be a MultiInputAudioNode class or something like that, which would handle this input connect/disconnect, output broadcasting, format conversion and so on, with an API to configure it all. Nodes would then subclass this and implement the core signal processing relevant to them. This is just one possible specialization of what Media Kit can already do, and can be used to build something more dedicated to audio processing above the Media Kit, such as a Jack-compatible API, or something completely different. Meanwhile, the "core" MediaKit stuff can still be used for video, or some other arbitrary signal processing.

[/quote]

This is possible, but you're back to the way it works currently. This means you have to keep two kind of mixers around, an explicit standalone mixer node, and an implicit one bundled into every node. There's the case of different mixer algorithm, but also the case where you want to post-process the mixer output. Maybe you want to normalize it, or filter it, or something else, before going on with the sound processing. The Media Kit is nice for this because it is very low level, and allows you to master the exact signal path. This is powerful but maybe a bit complicated, however, I don't expect people to use Cortex to define their processing chains. The same graph-like paradigm could be used in a higher-level application, working only on MultiInputAudioNodes, or instanciating 3 nodes for each graph element (mixer > actuall processing > splitter), or maybe a mix of the two (to handle 'legacy' nodes).

Anyway, I agree we're talking about void so far, and I'm waiting for your code so there is some actual testing and numbers. Good luck with the work :)

Yeah, i have to admit that everything will become more confused, it's the engineers trap to solve things with too much complexity where in a lot of cases the simplest solution is the better.

Thanks for pointing out, i have a lot to learn already

: )

Yeah, it's like what i was thinking, and i'm building an internal API for myself, something you may have already see around from Be. I got inspiration about this, from the Titan source code, which include some nodes developed by Be inc.

We should have things like this, for example the BSoundCounsumer node planned to become something like BSoundPlayer. I think it's partially a mistake, in my idea there should be something like a BAudioProducer and BSoundPlayer should be based on this.

Similarly we may have a BAudioConsumer node and a BSoundConsumer easy-to-use class.

I think this, just because i see there's a lot of code duplication in nodes, a various things done by nodes are equals. That's just an idea right now, i'm sure it's good for an application but not so sure about an OS API which have a lot of problems, and i may not see them right now...so i don't exclude i may convince myself that the generic approach is the best.

This discussion has cleared a lot of things for me, so basically in the first way an hypotetic audio routing app, should instantiate a mixer automatically in those situations.

As you said we may just instantiate those node by default, but the main problem is the same as before, we need something close to zero latency (i mean when doing 1:1 connections with a mixer and a splitter in the middle). Anyway, i would add that, in any case we should have minimum-latency nodes, with multiple inputs or not. So maybe, as said, i'm using a too complex approach...

I'm also curious if the situation will be improved by the current scheduler contract : )

Thanks..i think we need it for a lot of reasons, if we want to seriously develop the media_kit we should have some reference to detect regressions and improvements.

It is important to keep in mind that PCM data is granular. This is true inside a DSP-based system, but it's even more true for something like the Media Kit that deals not with individual samples or frames but with whole buffers amounting to several milliseconds of audio. As a result the system latency acquires the same granularity, if buffers containing 4ms of audio are used then it's quite meaningless to claim one additional millisecond of latency from some change, do you see? Either the 4ms buffers are fine, or you need larger buffers, or more of them, but no way can the latency now be 5ms. Be's design makes this a lot harder to understand than it should be, but that's really not anyone else's fault.

In a system like Haiku the DAC really isn't your concern, you are responsible only for handing over the PCM samples to some sort of hardware, it might be the digital-only HDA implementation on a modern graphics card or it might be a cheap 1990s AC97 implementation, but either way it's the same for Haiku. Likewise the word "normalize" is wrong in this context, and we will see why floating point is relevant later.

You could do either of these things but they'll sound terrible, so don't. Actually the second option might end up sounding like a third rate implementation of a really vicious compression-type distortion which is sort-of cool for a certain type of project but not what you were looking for. If you did want that you shouldn't implement it this way FWIW.

Ah, no. Although managing levels sensibly is a useful skill for a human it's not an appropriate thing for the system to try to guess on its own. However such shenanigans are exactly why you should choose float. You get a sign bit, an implied leading one and 23-bits of "fraction" across the entire normal range of the 8-bit exponent. Even if you divide by, say, four billion, you still keep all the precision anyway, and the same applies for multiplying too, which leads to this behaviour of floating point being sometimes called "infinite headroom" (in fact you get about +/- 750dB to play with). It was very easy to goof during processing in analog and in 16-bit or even to a lesser extent 24-bit integer PCM systems, making something too hot and distorted beyond repair, or else so quiet there's nothing left to amplify. This sort of goof doesn't happen in floating point.

Saturating addition. If you're determined not to learn enough theory to see why it's the Right Thing™ then why not just try it?

And how does any of this contradict what I wrote above? How does your pipeline manage to chain together buffers each four milliseconds long such that the total is five milliseconds or indeed anything but a multiple of four milliseconds? Your example is 20ms, which is a multiple.

By "system" latency I wanted from the outset to distinguish what we're discussing from the audio latency, which is arbitrary in this context. A tape delay simulator, for example, can introduce colossal audio latency but that doesn't mean anything for the system latency.

A quite useful thought experiment to run in your head when thinking about this stuff is, how do larger or smaller buffers impact my CPU? Remember, the actual calculations to be done are the same whatever the buffer sizes. If your CPU isn't powerful enough to run a cabinet simulator in real time you can't usually fix that by just making some buffers a bit bigger, you'll either need a more powerful CPU or else go without the cabinet simulator. Changing buffer sizes my help you a little with CPU caches to squeeze out a last few percentage points of available performance, but it cannot make the sort of dramatic differences that Be's design seems to anticipate.

And this is where we come back to adding numbers together. A floating point add is a very cheap operation on a modern CPU. Compared to almost any interesting effect that you might incorporate into a Media Kit node it is inconsequentially small overhead. Unless you insist on doing it as an entirely separate node and thus add the node overhead on top. That's why it's silly to talk about this increasing latency, the granularity of latency is a whole buffer, several milliseconds long, there's no way any sensible topology will cause you to spend a noticeable fraction of that time doing the addition, so you'll keep using the same buffers as before and the system latency remains the same.

Indeed?

First let's try some other numbers that obey your constraint, how about the filter has 0.5ms latency. Total is 2.5ms of latency according to your thinking. But our buffers are still 4ms long. No matter what strategy the Media Kit uses, when a particular buffer starts playing for the next four milliseconds the audio played won't be influenced in any way by what your filter or producer are doing. Your approach is working out what the Media Kit calls "processing latency" and which generally doesn't get a name because it's not a very useful thing to know as we'll see shortly.

Now, let's get back to your numbers and try a worked example

T=0 producer begins working on buffer A, the first 4ms audio buffer
T=1 producer hands buffer A over to filter
T=4 filter hands buffer A' (modified by filter) over to consumer
T=5 consumer has sent A' to PCM hardware, back now to producer to work on buffer B, the second 4ms audio buffer
T=6 producer hands buffer B to over to filter
T=9 a PCM buffer underrun occurs, probably audible as a loud click or stutter

So we see this doesn't work. Why not? Because as we discussed previously the workload is too high, we need to do 5ms of work every 4ms of time, and that's not possible.

It seems you've contradicted yourself. Does the buffer duration have to be at least as large as the 200ms latency of the media player, which you "wouldn't want" or not? Earlier "just because a single application request it" was enough reason, now it isn't.

I can't restate this often enough - when you're not sure, go try it and see what happens for real. If what you find experimentally doesn't match up with how you thought the system works, it means you were wrong.

Sure, I'm using "system latency" here to mean the time taken to play all the frames of audio in every buffer used. Typically a synchronous system will use 2 buffers, but more may be necessary, the system you've described could need considerably more. You might wonder why this amount of time is important, it's because whatever input the user is supplying cannot reliably affect the output until all these frames have been played. If you try to cheat (and some people have, including by accident) then there is noticeable jitter which is far worse than higher latency as far as musicians are concerned.

The reason to worry about this measure is that it's perceptible to the user but you have the opportunity to do something about it unlike the physical latency of the DAC, or the unavoidable audio latency of a "Cathedral effect" convolution filter the user insists on running.

I'd finished my example, I was back to examining yours as I wrote in the text. They illustrate different problems with your theory.

But OK, let's back up to my example, with the filter taking just 0.5ms to run, you'd call the latency 2.5ms, but in fact the sound coming out of the system now is from more than 4ms ago, not 2.5ms. Every buffer takes 4ms to play, there's no getting around that. Let me spell that timeline out since I didn't previously.

T=0 producer begins working on buffer A, the first 4ms audio buffer
T=1 producer hands buffer A over to filter
T=1.5 filter hands buffer A' (modified by filter) over to consumer
T=2.5 consumer has sent A' to PCM hardware, and by Be's design we can rest until T=4
T=4 producer begins working on buffer B, the second 4ms audio buffer
T=5 producer hands buffer B over to filter
T=5.5 filter hands buffer B' (modified by filter) over to consumer
T=6 audio frame playing is still from buffer A, now over 5ms old, our latency is clearly not 2.5ms

No, the workload is too high when you can't do one period of audio in one period of time. You've tricked yourself by dividing the workload up into pieces, but you have to do all the pieces or else you'll stall out. Since this is the Be Media Kit the usual next resort is to say that you'll run some of the workload on another CPU, but this means additional overhead and buffers and it only buys you the ability to split the workload N times for N cores, not "arbitrarily larger".

I hoped to get jua to understand how this works, and perhaps PulkoMandy too.

It doesn't make any sense to talk about "latency" for a system that doesn't operate in real time. Note that in Haiku "playing audio from disk" is in ordinary practice a real time operation, because the software volume controls sit between the media player and the soundcard, and also the supplied media player has local controls too. When you increase the volume in the mixer GUI, that's a real time change to the audio being performed.

Ah, the subtle stuff. I could really use a whiteboard here, but we'll have to do without. We've seen why it can't be less than one buffer length, but the reason it can't be less than two is a bit trickier.

Haiku's Media Kit, like ASIO or JACK, lets the workload vary over time. You can drop in another synth and it'll just keep going. Don't need to stop everything - black out the stage, spend a couple of minutes re-wiring everything - no you can just get on with it. But there's a price: This means that even if in fact the framework chooses to schedule your node to run close to the end of a period you can't rely on that. If you did the result would be either drop outs or nasty jitter when things change. So you always have to act as though you might run at the very start of the period, the latency will be two buffer lengths (or three, four, etc.)

Today most of Haiku isn't built to this exacting standard, there are plenty of other things that need fixing more urgently, but it would be a mistake to take the lesson that Haiku's doing better here rather than worse.

The entire workload, not just one node, exactly as korli and I have explained. Forget about the nodes here.

Avoiding jitter is a basic requirement. I mentioned this quite a lot earlier, you can't take seriously a system that will introduce jitter in this way. We are not talking about the more or less imaginary sub-frame jitter that audiophiles mumble about when reviewing cables, and which most likely vanishes during reconstruction - this is jitter of the order of milliseconds, which is all too audible.

That's not how agreeing works. I've been trying to get you to understand all the way through how it has to work, we're not trying to reach a compromise here, the question is only whether I get you to understand or not.

Aside from more prosaic considerations the value of possible changes during rewiring. If your scheduling moves earlier you will be running too early to deliver your promised latency, and you have no control over that, so you have to always assume the worst case or you'll introduce drop-outs or jitter. That worst case is two (or more if you introduce more) buffers.

We already saw exactly this, using your own example numbers, in http://www.haiku-os.org/community/forum/cortex#comment-27582 and you simply snipped it out completely from all further conversation as if it did not exist.

Your insistence on trying to think only of the workload of individual nodes leads you astray. All of the nodes must share the same resources. Multitasking isn't fairy magic, the computer doesn't become more powerful when you run more stuff, it just parcels the same resources out more thinly.

No, to do this correctly the latency will be 8ms. If you assume you'll be scheduled later you risk drop-outs or jitter so that's the wrong decision.

If you want to do twice as many things at once you end up doing them twice as slowly, that's how timeslicing works. I'll show the number of simultaneous execution contexts as X to help with following along, and assume that the above rule of thumb is sufficient, in reality the penalty for having multiple execution contexts is a little higher because of context switches, caches etc.

t=0: A begins filling buffer 1 (X=1)
t=1: Buffer 1 sent to B, B starts filtering (X=1)
t=4: Buffer 1 sent to C && A begins filling buffer 2 (X=2)
with two things at once to do, they take twice as long, two milliseconds to do both
t=6: Buffer 1 start being played to the user && buffer 2 sent to B for filtering (X=1)
back to just B running for a while at full speed
t=9: Buffer 2 sent to C && A begins filling buffer 3 (X=2)
slow mode again, two milliseconds until we're done
t=10: a PCM buffer underrun occurs

Huh. That didn't work after all. Well let's try to shuffle things around.

t=0: A begins filling buffer 1 (X=1)
t=1: Buffer 1 sent to B, B starts filtering (X=1)
t=4: Buffer 1 sent to C && A begins filling buffer 2 (X=2)
with two things at once to do, they take twice as long, two milliseconds to do both
t=6: Buffer 1 start being played to the user && buffer 2 sent to B for filtering (X=1)
t=9: Buffer 2 sent to C (X=1)
Yay, we cut some corners but now we'll make it in time!
t=10: Buffer 2 starts being played to the user && A begins filling buffer 3 (X=1)
t=11: Buffer 3 sent to B (X=1)
t=14: a PCM buffer underrun occurs

Still didn't work.

You can keep re-arranging things all day, but you're trying to fit 5ms of work into 4ms of time so it won't fit, no matter what you do.

No, only the illusion of concurrency is created, and only at a human timescale, the timeslices are not actually happening simultaneously. And on the scale we've been discussing these slices are quite thick, milliseconds like the buffer period. If one node's thread is pre-empted by another it most likely won't get another chance until the next period, that doesn't matter for your example, which is doomed anyway, but it's important to have it in mind for e.g. graphs where parallel execution is possible within a single buffer period.

Why not? Other than because it would be inconvenient for you?

So, explain how any node can ever reliably keep the "promise" to do finite work without any guarantee in return that it will get some amount of resources to do the work in. If that's not possible, why bother asking them to make such a "promise" at all?

OK, so again, how should they decide what to report? Multiply everything by two? By fifty? By a thousand?

This is hand-waving. A preflight for one node discovers nothing about its behaviour when competing with other nodes for resources. How shall a node correctly discover the latency you've been talking about which magically allows for an unlimited number of simultaneous execution contexts?

So, the way you make 1+1+3 less than 4 is by being less than truthful with all the numbers, the 1s are actually worth 0.5, the 3 is maybe worth 2, and so the total is only 3 after all. Pure theatre. A system that dispenses with this foolishness gets the same results, 3ms of real work can be executed synchronously in under 4ms too.

There can be a drop out or jitter every single time this adjustment is made, intolerable. Worse, these silly estimates and the pretence that it's OK for the entire workload to take more than one period leads to the system spiralling out of control with "late" messages when workload exceeds capacity rather than booting things from the graph.

This probe can only measure the state when it is performed, not the state it's being asked to predict. You haven't offered (and indeed the available Haiku software makes no attempt to offer) a solution for that. The entire premise (if we are to believe jua) of the Media Kit is undermined by these estimates.

Check for example the equalizer node provided with Haiku, its latency estimate is calculated by simply running its calculations over a test buffer at the moment of connection. This is exactly the sort of probe you're talking about, and jua would have us believe that it shouldn't do this because the results running alone now will of course not reflect what happens if later it is competing with another node for timeslices. But, what should it do instead?.

For audio on a PC type system? Do all the work and race to idle. This was the approach taken by JACK for good reason. Note that if you lose this race then by definition you could not have avoided a buffer underrun by any means, whereas you can't be sure with the Media Kit approach described above -- so JACK is justified in deciding to kick a node from the graph for running too slowly if it misses the deadline. In the Media Kit design you can at best rely on heuristics for managing lateness.