cortex

On the one hand, I really am mostly interested in explaining the theory. It's an opportunity for people to learn stuff that's useful outside of the present context, stuff that I wish I'd known twenty years ago before I wrote my first audio software.

But on the other hand, in practice yes the Media Kit design really is pretty terrible. Does it work? After a fashion sure, and so did the BeOS R5 netserver.

+1

The solution is simply to do this check some times programmatically and update the latencies accordingly. But for most nodes it's not needed.

I'll try to use different words for explaining what you aren't understanding (or refusing to). Before all it's false that latencies aren't not taken into consideration in jack. If you go through the documentation at some point this question is explained very well. Basically the latency callback is not declared by clients with only input or output ports, or when the input is completely unrelated to the output. In BeOS the situation is not different, for (pure) consumers the latency isn't something that have meaning at all.

Extending what i said before, let's dispel some myths out there. The approach taken by Haiku is performance driven, and it's true. But by itself this means only that you have more control on what will happen in the node. This is probably not needed in a audio system but the media_kit is not a such type of beast in the strict meaning. So, i'll show you what is the current approach which usually a BMediaEventLooper take :

1 - the new buffer event is scheduled at time x
2 - receive a new buffer event at time x
3 - create the buffer and send it

Theoretically there's nothing preventing you to do this :

1 - create the buffer
2 - the new buffer event is scheduled at time x
3 - receive a new buffer event at time x, retrieve the buffer and send it

From the latency perspective, i think jack use some method to estabilish how many time it will take to process the client (when the latency_callback is not declared), and a similar technique may be used on top of the media_kit. I don't think really those are the problems out there.

Please give a variety of examples of nodes for which no solution is needed. You should explain your working. If you don't have any examples to give, you should reconsider your argument that "most nodes" don't need to solve this problem. Note that for the overrun examples "updating the latencies accordingly" just results in latencies climbing forever, you still can only do finite work in finite time as before. This is why jua's clever examples don't work, they assume you can do infinite work in finite time just by slicing it up more, maybe a nice argument for a theoretical mathematician, but quite obviously wrong for a computer programmer.

OK?

Please try to rewrite this so that it doesn't contain a double negative with unclear valency. I recommend trying to write things in positive terms when you can, for example perhaps you meant here "It's true that JACK takes latency into consideration" ?

As the documentation explains the latency callback is only to be used by clients which introduce an algorithmic delay. This occurs when the algorithm unavoidably needs to use "future" input sample frames to calculate the value of "past" output sample frames. A "lookahead limiter" is a popular and useful example.

You should not confuse it with delays related to the audio hardware (which JACK tracks separately), or with the "latency" values used by the Media Kit for scheduling decisions.

The situation is quite different in BeOS because the latency reported is used for scheduling decisions. That would be nonsensical in JACK.

For a BMediaEventLooper implementing a trivial effect (say, doubling the linear amplitude of an audio stream) which of these steps is deriving the new values from the old? I suppose it is "create the buffer" ? In this case you are wrong, you are prevented from moving this step before "receive a new buffer" because you cannot proceed until you have the input data to be processed.

Of course you can sidestep this by incurring an entire period of additional latency for each such node. That's actually more or less what Be's engineers were recommending when the BMediaEventLooper was introduced. Their plan was to come up with a fix for the next release of BeOS. Alas, events overtook them and BeOS R5 missed quite a lot of things more obvious than this. The Media Kit you have today leaves the problem unsolved.

No. JACK doesn't need to "establish how many time it will take to process the client". It just runs the client, synchronously as described previously. If the client takes too long to run (causing xruns) it is dropped from the graph.

Because JACK does not have this imaginary "method" you cannot incorporate it "on top of the media_kit".

Any BBufferConsumer is an example of a node which doesn't have the need to have a complex latency handling, because it's usually fixed. The consumer latency (more like jack do with the driver latency), is used to estabilish how many time it will pass before the buffer is heard at the speakers.

Talking more generally, if the node processing cost is not variable (in the sense that it's every time a Ω(n) or Ω(n log n) and so on), it's latency may be perfectly calculated using a phantom cycle, just as i've seen Clockwerk do.

So at this point you may say (as before) "What happens when the load become too high?". Well the media_kit, notice the producer that it's producing buffers too late. At this point the producer should update it's latency accordingly. I would like also to show out how this way to do things is more reasonable for video rather than audio.

You've understand perfectly. No need to comment a typo of a non mother tongue speaker. This just show us your joking way of doing discussions.

NO. It's used to get the latency of a port which is not trivial. So for example if the internal signal path is not unique and jack can't trivially estabilish how to operate. The callback is needed for example in a client processing audio from two separate inputs to two separate outputs.

Absolutely not, scheduling is another story. BeOS use the latency to determine when the buffer should be played out.

The only difference with jack in this case, is that jack process everything in a unique cycle. Where the BeOS instead, use different threads to do that in a cooperative fashion. But if a jack client block out the results will be the same (an xrun).

That's something like a xrun.

Anyway, in sync mode jack always wait for the clients to end. In async he doesn't wait and go to the next cycle. If jack were not taking into account latencies, how it recognize xruns?

Well, i remember it doing all this way in the driver :

1 - the buffer is long 10ms
2 - process graph
3 - if the process has been more long than 10 ms we have an xrun

But this way to work is replicable also in Haiku. Jack is like a media_node itself and can be compared to it. It's clients aren't comparable to media_nodes but a little subset of it.

BBufferConsumers make up rather less than "most" nodes. In many cases the BBufferConsumer is paired with a BBufferProducer which does declare a latency.

It seems as if this recycles the same argument we demolished earlier in the thread, go back and read it.

You run into a tautology. If you can't get all the work done, you can't get all the work done. In general if you can't get 10ms of work done in less than 10ms, you won't have any more luck trying to do the remaining work and the next 10ms of work in the subsequent ten milliseconds. This is a conceptual error by Be engineers that we've been banging our heads against on and off for the last half of this thread.

I can't be sure whether I understand you when you write things that make no sense. Your claim here was wrong, as I explained, so it was also possible you didn't mean to claim that at all, I gave you the benefit of the doubt.

I don't think you've understood why this latency for most clients is "trivial". It's ZERO. Remember, this is strictly algorithmic latency. For all clients without algorithmic latency, regardless of how many inputs or outputs they have, no callback is needed.

I am concerned that you still haven't grasped what's going on here, because the Media Kit does such a lousy job of explaining the realities of real time audio. Algorithmic latency here is the latency that's unavoidably incurred by the mathematics of what we want to do. Not by whether you use an overclocked Xeon, or whether you used the latest high-speed DIMMs or whatever. So let's do a worked example.

We'll choose a lookahead limiter because although in practice they can be quite complicated both their purpose and the fundamental method of operation are fairly accessible. The purpose of the lookahead limiter is to ensure that, no matter how loud the input sounds are, the output is kept under a particular threshold, usually configured as a parameter, and further, the sound is distorted as little as possible. Imagine you're on stage and there is a sound guy controlling the mixing desk, but he is psychic, so that a fraction of a second before you play something very loud he adjusts the gain so that it doesn't deafen the audience. That's what this limiter is for.

Obviously computer software, unlike the imaginary sound guy, is not psychic. It applies gain based on the trend of samples, to give smooth changes and thus reduce distortion. So our limiter cannot set the gain correctly for a particular sample N until it knows what the next few samples N+1, N+2 ... N+M are. The value of M for a lookahead limiter might be something like 48.

If we're sending and receiving sample frames in periodic blocks (as in both the Media Kit and JACK) then we can't just use the received samples to calculate the samples we're sending, because we need the next M samples too. So instead we insert M samples of delay, always outputting samples that are M samples later than the ones we received, this is the algorithmic latency of our limiter and that's what JACK's callback is for.

Does that make sense now?

Exactly, this is a wrong-headed design, and the Media Kit pays dearly for it. It's hard to tell exactly when this muddle started, we do not have the source code history of BeOS. Algorithmic latency is largely irrelevant to real-time scheduling, so the BeOS "latency" measurements end up proxying as a way to schedule the processing work.

Yes, so your suggested "improvement" doesn't work, unless as explained below you're willing to incur a one period per node additional latency. This is the decision Be's developers didn't like the look of back in 1999, but we don't know if they could have come up with a solution (for example, a more rational way to schedule things in the Media Kit) because soon after the "focus shift" was announced, BeOS R5 PE was pushed out the door and developers who weren't focused on making cheap "Network Appliances" work with BeOS were fired.

No, it's additional latency, an entire period (ie some agreed number of frames) of additional latency, per node. This is a very high cost to pay, and one Haiku developers have previously claimed is not paid in Haiku. Feel free to either dispute this with them, or accept it as true.

You seem to have answered the question yourself. If a new cycle begins and we are still processing the previous period then we have an xrun.

Indeed? Well I suppose in a way you are almost right. The xrun is detected because the hardware driver (for example ALSA) reports it to userspace. JACK needn't try to guess. You might think "but it's not a guess" alas you would be wrong, the clock that matters for xruns is the sample clock on the hardware, which may or may not match wall clock time. The hardware driver has access to this clock (we hope) but the userspace application does not directly.

In what sense is JACK "like a media_node itself" ?

Two nodes doing the same work at the same time doubles the CPU load, not the processing time. I explained this in detail before, I won't do it again.

Estimating latency is tricky, but by no means impossible. Same as before, I won't go over it all again.
If you want to know more about it, read the sourcecode of actually existing nodes. You see, Media Kit nodes exist, and they work.

In practice all of the supplied nodes and example code have the same "way to work": They increase their internal latency estimate (up to an arbitrary limit) and press on regardless. Since most "lateness" reports are speculative they often get away with this.

As these internal latencies grow beyond one buffer period (for example, default Media Players alone can easily end up reporting a latency over 140ms, despite a buffer period of 40ms) the system starts to experience buffer bloat. Despite a requirement in the Be Book, Haiku's BufferGroup code does not block but instead always returns an error if no block is yet available, so sooner or later the system will trip over its own nose in this fashion, and indeed various bugs have been reported that amount to this problem.

It's important because this algorithmic latency must be tracked in some pro audio applications. As we'll see Haiku ends up not providing any way to do that. If you want to handle algorithmic latency in Haiku you'll need to abolish (or entirely re-think) the Media Kit.

This is an attractive option, and it seems (though we can't be sure at this distance of time) to be how Be's engineers conceived "event latency" in the Media Kit very early on. By the time it was actually shipping, however, they had switched to the present approach, in which reporting algorithmic latency here doesn't work.

The BMediaEventLooper's ControlLoop() dispatches B_HANDLE_BUFFER events (thus, the actual work of the node) based on the node's reported "latency" and its own estimate of scheduling latency. So yes, this is used to schedule work.

This is where (what I believe was) the original scheme that would have handled algorithmic latency fell apart. To get the Control Loop to behave in a desirable way Be's engineers began recommending that people use SetEventLatency such that the reported "latency" was almost an entire buffer period. By the time BeOS R4.5 shipped, all example code took as its assumption the fact that "latency" is really just a curious way to say "how long before performance time this node should be run" and all concept of actual algorithmic latency is gone.

Critically, in Haiku every consumer makes this decision independently. The idea seems to have been that this propagates lateness detection through the network, but in practice the effect is that reported latencies grow until they bump into the arbitrary limits hacked in by various developers over the years.

In JACK xrun detection is a problem solely for the driver, at the edge of the graph. So long as the graph executes in its entirety within a buffer period, we're OK.

BTimeSources are software, they estimate a relationship between real time tracked by the OS and, for the cases we're interested in, a frame counter increased by the Multi Audio Node. Whilst more useful than just relying on the real time reported by the OS directly, they are not the hardware, just a proxy.

Haiku actually ignores the behaviour dictated by the Be Book for these time sources in a bunch of places, but it hardly matters because in practice so did BeOS.

The Multi Audio Node's consumer does make its lateness decisions based on whether it was able to write buffers in time, as far as I can tell. But other consumers don't have that option, so most "lateness" reports in Haiku are speculative.

Do you understand what CPU load is? When you do more work, it takes more time. Perhaps this time does not feel perceptible to you, on human timescales, but the reason the CPU load goes up is that the CPU is spending more time working. Processing time has increased.

There are basically two "methods" used by existing nodes. One method is to pick a number out of their air, say, one millisecond. The node declares its latency will always be one millisecond no matter what. This "estimate" is worthless.

The other one - which we'd already addressed - is to measure once at startup how long it takes to run the function that generates the output, this by its nature cannot take any account of future changes.

Yes, you've said that before. And I even agreed with you, they do exist, they're just awful.

Haiku gets away with a lot because hardware has got a lot more capable since the days of BeOS. The poorly implemented resampler I mentioned earlier for example, doesn't sound as terrible on a PC with 192kHz audio, it's still bad, but that higher sample rate hides the mistakes better.

Good point, the System Mixer uses two different numbers for this calculation to determine its internal "latency", I had considered these to basically be picking a number from the air, but you could treat them more kindly if you were generous.

For a FormatChangeRequested it chooses either the buffer duration plus 4.5ms or 1.5 times the buffer duration, whichever is larger.

At Connect it chooses buffer duration plus 3.5ms or 1.5 times the buffer duration plus 1.5ms, whichever is larger.

As with many nodes if it receives a "late" notice it increases the internal "latency" by the reported lateness, unless it reaches an arbitrary limit (here 150ms) and then further late notices are ignored.

These choices mean that under normal circumstances after a particular buffer B has been created by the System Mixer, but before that buffer is handled by the Multi Audio Node, the System Mixer will calculate a further buffer B+1, and queue that too, so there's always an extra buffer "in the air". I have not found any commentary to explain this decision, if indeed there ever was a conscious decision to have it work this way.

Yes, but buffers are created spontaneously as the event time triggers. Offline mode is... vestigial at best at this point. Most stuff does not work properly in offline mode, ignore it.

Did you mean to say lateness rather than latency, here? I can't tell what you think you're telling me.

I said the limit is arbitrary. It was imposed by Axel back in 2010, it's set to 150 milliseconds, no further rationale was ever provided or asked for, it's an arbitrary choice.

The reported latency starts significant lower and grows. The buffer period is, as I already said, about 40ms, and yes, this leaves more than one buffer "in flight". The code actually acknowledges this, it ensures there are extra buffers allocated to allow for it, hence buffer bloat.

I thought I'd explained what algorithmic latency is. It's just a little further back up the page, maybe go back and read it again.

The event time just cranks monotonically here, the RealTimeFor() function basically converts the event time to a real time, then subtracts the declared "latency", and that's how long the BMediaEventLooper() will sleep. Thus, Media Kit latency controls when the work is actually scheduled to be done, increase the "latency" and the work will be scheduled sooner. It's a self-fulfilling prophecy.

That's actually the example I was looking at while writing. The card clock is not directly exposed as "DAC time source", instead, exactly as I wrote, the Multi Audio Node updates a frame counter which acts as the "clock" for this time source. It's a subtle difference, but it's worth knowing about.

There are plenty of bugs already filed about the Media Kit. Maybe somebody will fix some more of them this year. I am not speaking at all, any voices you're hearing are in your head, this is just text, as a bug report would be text.

Nothing was actually late in a sense that would matter to the user, but the producer is sent a late notification anyway. That's why I call it speculative. Remember that all the extant nodes respond to this by increasing their reported latency (so that, as we saw above, they will run "earlier").

I understand scheduling and load tracking well, no worries. Your description there isn't really correct though, at least not for Haiku (hint: a B_URGENT_PRIORITY thread still plays "the lottery" -- lower-priority threads still get time-slices as well, no matter how much CPU the urgent one demands).

Anyway, I didn't even want to get lured into this discussion again, I will stop now.

While I don't have time to look at the code in detail now, it seems you are right on this one. I somehow remembered it only applied to B_REAL_TIME_PRIORITY itself, not anything above 100 already.
Anyway, you might want to reread what I said about processing time.

As I said, I won't discuss things further in here.