Audio improvements

[juanpuerto99]

Hello!!
I have come to KNI and I found it to be a quite interesting project for those people like me who come from monogame.

With monogame I found quite a few limitations regarding the audio, mainly for the songs. XACT generated several bugs for me when using it (especially with CUEs), the MediaPlayer class seemed a bit limited to me (for example, I couldn't loop songs correctly or apply effects) and I imagine the soundeffects would not be suitable for songs of considerable length. Reading the posts, they mostly come to the conclusion that the ideal thing to do is to use third-party libraries like NAudio or others.

What state is the audio section in KNI? Is there a document where I can read more about it? If I had to play a song on KNI, what would you recommend I use?

Thanks in advance!!!

[nkast]

The Audio and Media backend has been refactored with some parts rewritten and many of the issues are resolved/fixed.
Additionally the audio API in KNI is thread-safe.
For songs, I recommend importing it with the SongProcessor and using the MediaPlayer class.

As for documentation, the public API is basically XNA. However the audio behaves 'as should be'.
Audio on the original XNA was very strange. For example SoundEfectInstance.State would take an entire frame or a call to FrameworkDispatcher.Update() to get updated after a .Play() or a .Stop(). Apparently the implementation was buggy because the XNA documentation describes a more reasonable behavior.

About XACT, I am not going to pursue completing the Xact implementation.
We should perhaps collect a list of features that people use alternative libraries for, and implement a clean new API.

My understanding is that we need:

• wavebanks, a collection of sound effects stored in a single buffer.
• cues, individual SFX stored in the wavebank.
• events, when a cue stops, etc.
• some sort of a state machine to play a sequence of sfx, and transition with mixing and fading in/out.
• Streaming wavebanks from disk.
• Store wavebanks in compressed form. As an example, the implementation of MediaPlayer for DesktopGL uses a DynamicSoundEffectInstance and decodes Ogg/Vorbis on-the-fly. https://github.com/kniEngine/kni/blob/main/MonoGame.Framework/Media/DesktopGL/ConcreteMediaPlayer.cs#L227-L261

[damian-666]

that Concertermediaplayer looks looks good. DynamicSoundEffectInstance

i put two days research into it and most of the info are based from opiions of many people and the code , is in a long chatbot4o convo. i can share it before anyone spends time on it ii think got the best way at this point.

if it compiles. it does look right. it makes some kind of wild guesses but combined with existing code it should be < 1 week.

vlcapi might work for multiple songs ink. each needs a window.. i never used more than one.

the others sounds can an should be on a fast bkthread. with no locks ..

i spend yesterday and a few hours today generating code with the newest chatgpt4o and after its 3 days old its got pretty good.
ill see if some of it compiles,

much is based on this.. i think that could be the implementation moving forward from XNA .

**int[ DirectTK](https://github.com/microsoft/DirectXTK/wiki/Audio/0881aee2d2522b8faccc48639e3e4de5ebeaec72

This has an AudioEngine abstraction we can build over**. it has all you mentioned except for Cues.
the curve thing is in the guts of the xaudio2 COM but i think is overkill ( to smooth transitions) it used to pop if you turned volume down too faster so i dont think we need to implement all that and tweening or smoothing as in the XNA tool
it querying what filters are on device, and the rest is software, i think Cues is too abstract. the AudioEngine can just be a basic thing, i implemented something almost exactly like it before i found monogame..

they dropped the XACT format, and cues but keep the wave bank and voice and mixers.

i think OpenALSoft is used now instead of OpenAL.

there are some available hardware implementations of the various filters. i grilled the bot on that.

we took the latest chatgpt0 without even the codex and it did a decent job generating avalonia voice to text and hotword changing project thte digs all the way into the other deeper COM api that is newer.. IXAudio2 i t made wrappers or used silk.

o but asked it to mix 2 effects in bothe OpenAL and XAudio2 using silk wrapper and generating wrappers using the cswinapi official tools its all c# but unsafe. I had it try the DirectTK later, i would send the convo but its too long but ill have a try at this, or if someone else does ill send the convo... and tons of links threre a 8 + hours of reddits and other threads about DSPs and all thius.

ill see if tiis builds its SEEMS to be getting really useful... for a 3 day old LLM. im not using the codex either just getting it to code on the main chat. the thing even trained me a hotwork using NWave and then went deep to the guts of windows to change their horrible but almost good Voice Access but it is offline. wire some agents to it and its great. So im building a windows assistant on this , for hands free UI. on the same convo but i think all the basic good needed its given. at least mabye via a test rig. then we can put the two versions in Concretlayers...

[nkast]

Great find. They have a Wavebank implementation for xaudio that can do in-memory and streaming, as I described above.
And also an XACT3 dump sample in cpp, an WaveBankReader.cpp,
and a tool to pack wav into XWBs. All MIT.

[nkast]

Recently I ported the Ship sample and I had to disable all the xact code in it. It had this .xap (?) format to import sounds.
https://github.com/kniEngine/XNAGameStudio/blob/ce5b55311549733a65519bbcdeb42f358f5f60a9/Samples/Ship-Game/ShipGameContent/sounds/sounds.xap#L304

Posibly we could implement a wavebank importer to read from content xml file.
Allthough there are very little benefit from an in-memory wavebank. A small perf in loading and saving a few bytes? But it's the basis for building the other stuff...

[juanpuerto99]

I think some of the things Damian mentioned are a bit more technical than I can handle at the moment hahaha.
I was thinking about which engine could be good if I ever want to move to web development, and since I know a bit about Monogame, this project sounds tempting. But I've always hated how Monogame handled audio (with unfinished XACT stuff, issues with MediaPlayer) :P

For playing songs, I always saw recommendations to use MediaPlayer with Monogame, but it also seemed quite limited to me. Not only because you couldn't apply effects or things like that, but also because you couldn't transition between two songs (or play two at the same time), it looped a bit poorly (I didn't have that problem when playing with sound effects), and things like that led me not to want to use it at the time.

I also remember seeing the source code of games like Terraria or Stardew Valley, which had XACT implementations inside, although if I recall correctly, they were modified versions of XACT (with things I didn't have with the AudioEngine that came with Monogame). I'm not sure if this info is helpful at all xD

The idea of a new API sounds tempting, but it also sounds like a lot of work. I don't want to complicate your life over just a doubt I had hahaha.
As soon as I have a moment, I'll take a look at ShipGame to see how you make things work, and I'll check if I can get MediaPlayer to work properly.

Greetings!!

[damian-666]

to OPs question i think you can spawn multiple processes to play multiple songs. at the same time or try to mix them. worse way is spawn a process for each song. you can do that with chrome.. I don't see any other way. by googling. you can try VLC Player api, but these players Media players and Songs are limited to one . they need a Windows handle and they want to be on the UI thread. but lining them up , thats a trick. see it done at the bottom in webtech , its hard with JavaScript. untyped and slow.

we needed the filter stuff anyways, for a basic framework where it either kind of works or used to work, that's not for AAA games with tons on pre canned assets and little in-game movies, or super real expensive sound .. thats appearing in unity and steam.
they where better in the 1980 because they has to synth the lofi sounds. they is nothing as boring as repeated exactly same high fidelity stereo human "ugh".. when you get punched..

Ccombine and play mulplee looping songs by using DynamicsoundEffectinstance is fine. you can always play one sound and mabye at least mix it. in and see whats availabe now with new drivers and bindings. you could mix a song and then play it mabye... the reason is that the samples rates and the phases need to line up , for stereo and other or you ears will be confused. there is nothing worse than hearing two chrome windows playing different videos. , to me.

I think thisi is very doable and the bot already coded some demo for me .. the openALsoft has a license i dont understand but its part of a libary as a dependency so i dont think its bad. sink is using it . the bots code might not compile but it can help fix the existing.

To allow for interactive short sounds when you pull a trigger you want to hear the gunshot with the feel of the button. the speakers are almost no distance away. its harder than music.. to avoid latency gives a good feel. 5ms is even too much you can get as close as 1 with expensive stuff. or maybe some tricks. so 2hz or so. is good that the feel of ana touch screen.. . 5 -7 is tolerable, so it has to stay off the ui thread. so well see how effects apply i think they will apply in a preprocess.

best practice is to preload soundeffects, ( or banks or groups of them as Monogame had) for each level, create soundeffectinstance, apply some parms. if its playing you create a new one. effectinstanace. if not and you need the same variant just play it again. i key soundeffectinstance wiht the variations i put to tham.. pan zoom, whatever.

NAudio is good and has been evolving for a along time. but is basically windows only. some of the 400 issues /branches are porting it but in a adhoc way.NAudio it does seem hook into low leve apis. but they are windows WASM

NWave is floating point software only may and a basic DSP course, the use of floats32 bit might be an issue. chat used it as a means to an end to make a filter that might passed to hardware . to make a dsp filter that i can put in the chain and tell me if its running on hardware or software. the quaalcomms are all DSP and FPGA so this nneedss to be more open done.. especially since the ai is getting useful at coding it could run on device.

naudio/NAudio#903

hopefully i can get the bots code to run on android and windows sometime thins month. a or at least make a sample up let us update the wrappers to the newest ones. there are breaking changes that are not backwards compatible like use of 0 instead of enums. On reddit AI for coding some have agent chains my realize my dream of telling the bot what i want in plain English and keep building , trying, and call me if it gets stuck or if it works.

the latency of NWave since its CPU and MediaPlayer (worse) is said to be 50 ms thas not that great for tings like gun shots or fast sounds. MediaPlayer (worse)

https://stackoverflow.com/questions/26846342/is-it-acceptable-to-use-soundeffectinstance-for-background-music

another option is the web tech but there are lots of crackles and stuff. 16khz sampling is fast.. you cant allow cate or do GC and need solid time tested apis.

here is a web tech lets you batch songs. its so boring to watch the vids because his typos result in runtime errors. typescript might be ok but unless its blazor and comes for free its going to be slow . or unsafe. already blazor is a bit dodgy since anyone can put up a website. my favorite physics tech is in typescript. the answer is yes it can play a goup on songs together in the sample code. but she complains of crackles.

https://tonejs.github.io/
https://tonejs.github.io/examples/envelope

https://tonejs.github.io/docs/r11/MultiPlayer