Switch to interleaved channel layout in AudioBuffer?

[dwuertz]

I noticed that AudioBuffer stores multi-channel sample data in a planar fashion, i.e. each channel is stored as a continuous slice. This leads to some processing overhead in practice that could be avoided if channels would be stored interleaved:
Most if not all file formats store their samples interleaved. The same holds for I/O APIs. As an example, when playing back an audio file, there is both a de-interleaving and interleaving operation involved.

I would be curious to hear a rationale for the current design, because honestly I can't find a good one myself. Would you consider interleaved sample storage in the future? I realize it would be breaking change with possibly large impact on the architecture, but it seems like a better design to me overall.

[pdeljanov]

OS APIs generally use the interleaved format because those are streaming interfaces and chunking data would introduce a baseline latency. However, it is not true that most formats store samples interleaved. It it actually the opposite. Therefore, there will be an interleaving step somewhere in your audio pipeline.

I recognize that Symphonia's current interface for doing this with (Raw)SampleBuffer is ugly. In the dev-0.6 branch I rewrote the audio APIs to provide simple functions on AudioBuffer for reading/writing from/to planar/interleaved buffers in bytes or samples.

Please have a look and let me know what you think.

---

On that note, many audio DSPs (e.g., the rubato crate) would need to deinterleave or be written specifically to handle interleaved audio. Interleaving audio early in the DSP chain could result in multiple interleave/deinterleave operations. The dev-0.6 branch allows the user to make the call when to perform the interleaving.

[kyr0]

Interleaved audio processing cannot EASIY be optimized using vector instructions (SIMD) like SSE, AVX and NEON.

In most DSP libs you process audio deinterleaved, because you cannot load LRLRLRLRLRLRLRLR efficiently in a vector register to process LLLLLLLL and then RRRRRRRR separately; you memcpy the data in a chunk into a register or you deinterleave first and then interleave thereafter. If the operation is implemented in a for loop with if/else's or index jumps, no optimizer can vectorize it and it will be slow.

Only if you run the exact same operation on data of all channels you can still vectorize arithmetics on interleaved buffers, but not with operations that rely on pairwise operations (adding up single values). That would effectively sum two channels instead of down sampling for example.