VBAP / Spatial PCM Sampling extension

[mkschulze]

Hello,

I'd like to leave this here as I know you started to work on an ambisonic extension, which is nice.

However, from what I read there is a 3D Audio approach, that is kind of more promising and transparent and would ideally also be possible. It is called Spatial PCM Sampling (SPS) and is a vector based approach.

This company does it and has some interesting things going on here: https://www.mach1.tech/faq

Also a great explanation can be found here: http://pcfarina.eng.unipr.it/SPS-conversion.htm

Just leaving this here to have somebody give it a quick thought and maybe see what would be required.
Likely #191 seems related to provide n discrete channels, which then could be used.

thx cya

[peterStitt]

I also think it would be great for CLAP to support n discrete channels in/out. VST2 does this (up to 64 channels) while VST3 is limited to specified layouts. This makes the defunct VST2 SDK a much better format for generate immersive audio than VST3. It would be great to see CLAP support arbitrary layouts.

The draft for ambisonics is a nice start but ambisonics can be decoded to any arbitrary speaker layout so support is also required for those. Then there are also all sorts of other spatialisation techniques like WFS, VBAP and the previously mentioned SPS.

I work a lot with Ambisonics and other surround approaches so I'd be happy to try help out on this if I can.

[hockinsk]

Would be great to have a native 3D audio multichannel approach in CLAP. I'm already using Mach1/SPS 3D Panners & Monitor under VST2 (VST3 not possible) in Bitwig with a kind of psuedo mutlichannel bus workaround which proves how flexible the SPS / Mach 1 format is even working in a Stereo 2.0 channel DAW like Bitwig. To have this in CLAP and hopefully one day multichannel support in Bitwig is a bit of a dream of mine. Bitwig does support multichannel .wav files and playback, just not at the plugin/audio routing level.

[peterStitt]

I've been looking at this again because I'd love to release CLAP versions of my plugins in the future. Any chance that we will be able to implement arbitrary multichannel formats? The ideal is that I can define that my plugin supports, say, X inputs and Y outputs.

At the minute, I guess the workaround would be to define a sufficiently large Ambisonic bus and just use that for arbitrary formats, since there is no remapping?

My offer to help out still stands :)

[abique]

I have no plan to spend time on designing an extension for this myself. Also I don't have a vision for it so I would probably not do a good job.

Though, if you make a PR with a well designed solution, we could take it.

[abique]

Also it isn't because the extension exists that hosts will support it right away. Everyone has its own schedule and works for its own interests.

[baconpaul]

I know nothing about Spatial Audio, but does the audio ports extension already let you define as many ports as you want, and the audio ports config extension allow you to present labeled bundles of them? I realize that may not be what you want semantically but isn’t it mechanically what you would need as a starting point?

the reason I say thst is because it makes me wonder if the extension here is a change the ports extension or if it is a semantic labeling one (which would be easier)

[hockinsk]

There shouldn't be anything spatial-specific needed other than how many inputs and how many outputs the plugin employs. Many plugins will need to update this. e.g you might choose from 5.1 or 7.1.4 or Dual Quad or 1OA or 3OA etc.

[baconpaul]

Can the audio ports config (stabilized since this issue was opened, here: https://github.com/free-audio/clap/blob/main/include/clap/ext/audio-ports-config.h) do what you need then?

Again I'm coming at this from a complete ignorance of anything other than 'bundles of stereo ports'. So please don't take my answers to be definitive!

[peterStitt]

Thanks for the replies @abique and @baconpaul.

I haven't built a CLAP plugin yet so I'm sorry if I make mistakes about the functionality. From what I can see, we could configure a plugin to support, say, X mono ports in a bundle and that could essentially be the same as saying I want a bus with X channels. Is that correct? Something like that would work mechanically, yes.

I'll try to expand on uses cases to make things clearer. It might be that the functionality is already in CLAP.

Let's say I have an ambisonic microphone that has 8 capsules. I have a plugin that converts these capsule signals to a choice of different formats: HOA, virtual microphones, ITU surround formats, custom surround formats. I need the input to the plugin to accept 8 channels and to ensure that it will not reorder them. So I would use a bundle called "MyAmbiMic" and include 8 mono ports.

For the output I need to be able to select from a set of outputs. HOA and ITU surround are already clearly covered in the CLAP spec. Virtual microphones could be mono, stereo or any arbitrary number of virtual microphones. Mono and stereo are clearly covered already. Say I want to output from 3 to15 virtual microphones, my ideal approach would be to specify that I want a port with 3 to 15 channels (probably just 15 is OK and output silence on unused channels). For non-standard surround formats I might want to support up to 100 channels and let the user tell the plugin how many speakers they have and where they are (for the DSP). These probably don't need to be labelled, as long as they are never reordered.

If I understand correctly, I could have a bundle called "Virtual Mics" that has 15 mono ports and another called "Custom Surround" with 100 mono ports.
This would actually be OK functionally. My main worry is that a host might pick some limit to the number of ports that it supports. Is that possible?

What I think of for this is JUCE's discreteChannels(). For VST2 you used to just be able to tell it how many channels you wanted. VST3 removed that possibility. If CLAP has/had some functionality that works like this then that would probably suit most plugin manufacturers who want to target arbitrary channel formats.