Dev Log #69 - Unexpected engine refactor (Patreon)

I was getting pretty close to finishing the implementation of Macro blocks and came across it pretty big problem, so as is typical in software development my previous estimate of "a couple more weeks" I think is 100% wrong.

I'm now about to embark on a pretty big refactor of the audio engine, though I'm feeling positive about it, and I don't actually think it will take that long (maybe just add a couple more weeks to my previous estimate.) I feel like I know exactly what the problem is and what I'm doing, though maybe that will change once this red bull wears off.

The audio engine already went through several rewrites early on in Blockhead's development, at a time when I still didn't really understand what Blockhead was or how things were eventually going to be structured. The code I eventually ended up with has worked well until now, but it turns out there are some subtle problems introduced by Macro blocks which render things a bit broken.

It is possible for me to just continue adding Macro blocks into the engine as it currently is. The user-facing problems in doing this would basically be:

1. The auto-crossfade transitions that the engine generates when blocks are moved/modified might sound a bit weird in the case of macros (i.e. moments of silence might occur)

2. Glitches and pops will likely happen while dragging macro blocks around, which is exactly what the auto-crossfade transitions are supposed to eliminate

I would 100% want to fix these issues eventually anyway, so rather than continue hacking in Macro blocks and writing a bunch of code that will end up getting thrown away, I'm just going to sort everything out now.

The problem in some detail

When the user changes things in the UI thread, the audio thread doesn't see the changes right away. They need to be synchronized in a thread-safe way. Blockhead currently updates things on a per-object basis, so if one block is changed, only that block is synchronized with the audio engine.

When a synchronization event occurs, the old version of the block data is kept in memory for a moment. The audio thread continues rendering that old version of the block while crossfading it out, while at the same time crossfading in the new version of the block. This is the generic strategy that Blockhead uses to eliminate clicks and pops while changing things.

Changing a block's position counts as an update which needs to be synchronized just like everything else. If the user is dragging a block and moves their cursor a single pixel, it will generate a new version of the block data with the updated position.

Macros introduce some puzzling problems to this system. When a macro block is fully baked it can pretty much be treated as above, like any other block.

However, when it's unbaked, rendering a macro block involves the audio thread looking inside the macro at the internal workspace and rendering the internal blocks in real-time.

A macro going from "unbaked" to "baked" or vice versa counts as an update which might introduce a click, so just like any other state change, a crossfade would occur in this situation.

There we could have some strange situations where the engine is crossfading between two versions of a macro block. For example, if neither version of the macro has a baked output, the engine will be crossfading between two versions of the real-time rendering of the internal blocks.

But the internal blocks might also be transitioning between different versions of themselves. Those internal blocks are being referenced by both versions of the macro, so when the first version of the macro is rendered, it will render the transitions between the internal blocks, then when the second version of the macro is rendered, it might try to re-render those internal transitions, because the crossfading of individual blocks is decoupled from their parent macro block.

I don't expect that to make much sense. This is all already extremely stupid and what I've explained here is just the tip of the iceberg. It gets way worse when you really get into it.

The solution(?)

Stop individually synchronizing individual objects. Any time anything changes, create a copy of the entire workspace with the changes applied, synchronize the entire workspace in one go and then crossfade between the two versions of the workspace.

This would also fix the long-standing issue with the current engine where moving a block on/off a lane can still introduce a click, even though the entire purpose of the crossfading strategy is to eliminate clicks, it currently only works when a block stays on its current lane. In this new design this wouldn't be a problem because the entire workspace is being crossfaded.

It might sound a bit expensive to create entire copies of the workspace every time anything changes but it actually doesn't need to be, because there is this thing called "immutable data structures" which allows you to do this kind of thing efficiently.

Immutable data structures are essentially what I have already been doing in many parts of the engine, but the way I have set things up so far really doesn't take full advantage of the concept.

The idea is basically to have one big data structure which represents one version of the workspace. Instead of making changes to the workspace directly, changes are made by creating a copy of the entire workspace, but only a shallow copy. The part of the workspace that changed will point to some new memory whereas the rest of the structure will continue pointing at the old memory, so copying the entire workspace is much cheaper than it sounds.

There is a C++ library that I have been aware of for ages which provides some generic tools for doing this kind of thing:

https://github.com/arximboldi/immer 

I haven't used this library before but from what I can tell it's exactly suited to what I want to do, and to be honest it's probably what I would have been using from the start if knew how Blockhead was eventually going to be structured.

If all this works, then I think it will be a big step towards making Blockhead's audio engine much easier to work with, and a lot more flexible.