Sound in screen share on Linux

[DeeBeeDouble]

MOD edit / notes

This issue ticket was previously a request to integrate with the existing solution, which I didn't want to work on – mostly as both JavaScript / TypeScript based client and native Node modules would better integrate with the code and build system (i.e. a such binary I would have to provide would likely has to be somehow rebuilt outside of Electron Forge and probably depend on more libraries / binaries which normally would not be needed for packaging of my application). However it quickly became one of the most popular topics in WebCord and became a bit more generic about sound screen sharing on Linux. It is also very special for me because of the large community interest about resolving it, making it much closer to be finally resolved once and for good. So thank you all for participating in it!

MOD EDIT2: This is now resolved in Chromium, awaiting to be merged to Chromium stable and/or Electron. Right now this is a feature behind the flags, hopefully it will be enabled by the default and will work just fine. See bugs.chromium.org#1143761 and Chromium commit 9a3d5ea for more details on it.

@SpacingBat3

Original message by @DeeBeeDouble:

Hey,

I was always annoyed by the fact that discord on Linux doesn't support audio sharing. Then I found webcord and another project that allows you to share your audio in a browser.

Then I thought: "Wouldn't it be absolutely awesome if webcord could integrate this?" Currently, it's rather complicated to >install the sound support (especially for newcomers). An integration in with webcord could make this process much easier.

I can not really code, so I can't make any suggestions on how to do that or if it is even possible, but I would just love to see it and thought I could at least give it a try.

Have a nice day, everyone ;)

[SpacingBat3]

I'll probably won't intergrate this project with mine, I think it's a bad practise to depend on third-party binaries. However, I might make a use of NPM and find a module (probably native ones) to directly manage the PulseAudio server. But since it would take me some time to work on that, don't expect it'll be available soon.

[DeeBeeDouble]

Yeah, that would also work. But it is also going to be compatible with pipewire? (Because you only mentioned PulseAudio)

[SpacingBat3]

But it is also going to be compatible with pipewire?

If PipeWire replaces it, then yes.

[DeeBeeDouble]

But it is also going to be compatible with pipewire?

If PipeWire replaces it, then yes.

Pipewire already is the default for many major distros today, like the newest Ubuntu, Pop-Os, Fedora and so on. So I'd say pulse already got replaced.

Anyway, shouldn't it also be fine just to support pulse audio and let pipewire-pulse do the rest?

In the end, it's your decision, of course.

[SpacingBat3]

Pipewire already is the default for many major distros today, like the newest Ubuntu, Pop-Os, Fedora and so on. So I'd say pulse already got replaced.

I thought about the PulseAudio server using PipeWire. On many distributions PipeWire can be either installed alongside PulseAudio (and probably used as a separate audio system) or replace PulseAudio when a separate package is installed (pipewire-pulse on Arch). So, it won't directly support the PipeWire, but rather use PulseAudio client implementation and connect to local PulseAudio server, regardless if it is a compatibility layer implemented by PipeWire or actual standalone PulseAudio server implementation.

I hope that clears things up.

[DeeBeeDouble]

Ohh okay, that makes sense, thanks.

[SpacingBat3]

And here's a few more details about the audio issue on Linux:

1. Upstream issue: [Linux] System loopback audio capture (#1143761).

• It might be useful / contains different workarounds and more details about this.
• It shows current progress about this issue.

2. The most recently updated PulseAudio client module for Node.js: @tmigone/pulseaudio.

• Unlike other modules, this one seems to be the most actively updated.
• This module supports TypeScript out-of-the-box (without DefinitelyTyped-provided type declaration files) as well.
• It seems to have JSDoc-based documentation.
• Does not work with sockets right now.

3. This comment from electron/electron#10515:

Then the corresponding deviceId can be used in the audio constraints and mixed with the video stream to get a video-with-audio stream.

• I'll probably use another module (more recently updated and designed for TypeScript).
• I'll take a look if there's any way to get directly a stream and somehow make it understandable for the renderer process. It's unlikely that's possible (Javascript+DOM is quite different than Javascript+node and sharing the types would probably require of manually converting them from one to another).

[snoweuph]

Hi, when work on this is started, will there be an extra branch for it, that can be watched to stay up to date?

[SpacingBat3]

Hi, when work on this is started, will there be an extra branch for it, that can be watched to stay up to date?

Probably not. I doubt I will do any changes without experimenting them first. And I think there's no much work at implementing it (only a lot of testing and understanding the module I will decide to use), so there won't be made a lot of commits. But if I feel unsure about its implementation and have concerns whenever it will work or mess up something within my code, then I'll definitely open a new branch.

I'll inform on this issue on any progress done. Right now I give it a lower priority due to its complexity, but it might be worked on once I deal with most bugs I can fix, probably finish a Flatpak maker for Reforged project and have a lot of free time. So please be patient or work on it by yourself – it's a bit shame there was no revolutionary code Pull Request on WebCord 😉.

[snoweuph]

I see that as a challange :)

it's a bit shame there was no revolutionary code Pull Request on WebCord wink.

ik some JS and TS and worked a little bit with electron before, maybe I can get something working, but I don't know, I will see :)

[snoweuph]

I've got a question:
Does normal Discord in the Browser support Screenshare audio theoreticly or was that striped of it?

[snoweuph]

Wait, I've understand that my Question doesn't make sense, In my Question I was thinking that audio and video have seperate endpoints, but thats prolly not the case, they are probably connected, so its just a matter of injecting the audio into the video stream

[DeeBeeDouble]

Wait, I've understand that my Question doesn't make sense, In my Question I was thinking that audio and video have seperate endpoints, but thats prolly not the case, they are probably connected, so its just a matter of injecting the audio into the video stream

Not quite sure, maybe take a look in the other project and see how they did it. I mean that project: https://github.com/edisionnano/Screenshare-with-audio-on-Discord-with-Linux

[snoweuph]

Okay, thx for the info 👍

[snoweuph]

I will also ask the creator of that if he maybe wants to help,m i kinda understnad how it works, but as someone who has made it himself he prolly will be much better at it

[snoweuph]

from understand droping Pulseaudio support will make this so much easier, the creator of that project is also saying the same.

How about making it a expermintal pipewire only feature for now that need to be enabled first?

[edisionnano]

The easiest way to get this working would be to drop pulseaudio support and make a native node addon with node-addon-api and cmake-js that uses rohrkabel capture audio from apps like my virtmic binary does. To match a pipewire node to an X11 window(s) you can use the PID key pipewire provides

[SpacingBat3]

The easiest way to get this working would be to drop pulseaudio support and make a native node addon with node-addon-api

I think it would be great to preserve PulseAudio and X11 compatibility, since PipeWire is still not or can't be used everywhere. And to not reinvent the wheel, the existing module can be used then.

And native module implementation might not be a requirement as long as there's a way to communicate with PipeWire/PulseAudio via UNIX socket. This could mean a reimplementation of protocol / client, but could work across different processes – native modules don't play well with Electron's renderer process from my experience.

[DeeBeeDouble]

from understand droping Pulseaudio support will make this so much easier, the creator of that project is also saying the same.

How about making it a expermintal pipewire only feature for now that need to be enabled first?

I would be totally fine with only supporting pipewire for now if it makes things easier. Later we could then start working to make it compatible with PulseAudio if this is still needed by then.

[DeeBeeDouble]

There is a new discord client that already supports audio share. Just wanted to let you guys know because I guess some inspiration would make it easier to implement. Also, there is a mention on Reddit that it should be implemented into webcord with over 90 upvotes, so it is definitely a requested feature.

[SpacingBat3]

@DeeBeeDouble it's not the issue to implement in any way. I want to implement it the proper way, at best to have a node module extension to record audio and transfer that data back to Node and at least implement a code to create a virtual sink for all audio output devices and use that as the stream input device. I may play with Rust to achieve this, I have a very little experience writting native Node modules and manipulating the Buffer data. I may also try to develop a code with TypeScript by trying to connect to PulseAudio Unix socket and try manage it using it. The better might be a fully TypeScript-made solution for faster compilation and cross-platform support, yet slower runtime execution (that might not be that slow to affect the overall app's performance).

[DeeBeeDouble]

@DeeBeeDouble it's not the issue to implement in any way. I want to implement it the proper way, at best to have a node module extension to record audio and transfer that data back to Node and at least implement a code to create a virtual sink for all audio output devices and use that as the stream input device. I may play with Rust to achieve this, I have a very little experience writting native Node modules and manipulating the Buffer data. I may also try to develop a code with TypeScript by trying to connect to PulseAudio Unix socket and try manage it using it. The better might be a fully TypeScript-made solution for faster compilation and cross-platform support, yet slower runtime execution (that might not be that slow to affect the overall app's performance).

Alright, even better

[SpacingBat3]

I've realised that actually it might not be the best idea to use any stream for the screen recording. You see, any modification is likely to be found by Discord if they want to. Using a MediaTrack that has different properties than expected would likely make WebCord easier to discover as a modified client. This is why I think it would be better to actually set the default device to monitor or virtual sink and rely on Chromium's code for recording audio rather than picking it directly by WebCord.

I may add a flag to forcefully enable audio in screen recording to mitigate this issue, so with the right configuration (monitor as default, microphone as selected input device) you should have both working microphone and system audio.

[SpacingBat3]

Also link to (much prettier) commit info on Github mirror: chromium/chromium@9a3d5ea.

You might also find there a tag list which contains versions of Chromium that contains this change, so I guess you might as well use that to estimate whenever this reaches stable Chromium/Electron.

[SpacingBat3]

Looks like Electron 29.x.y (the next version of Electron) is going to support it!

It's currently in beta, but testing this with WebCord (with 29.0.0-beta3), audio sharing seems to work flawlessly, at least when I capture stuff via DevTools (in order to play it via the speakers or stream that isn't captured by Discord). The only problem I've noticed is that audio from Discord is captured as well and given loopbackWithMute isn't explained as it would filter/mute the app audio from the stream, I suspect this could be the case for Windows as well.

But again if that's the case, with Linux the one could still make a sink in order to precisely select which streams to use for the recording and workaround this issue on their own. I don't think something like this can be made on Windows.

[ThatOneCalculator]

Awesome, can't wait to test this! Is installing from git good or should I switch to a different branch?

[SpacingBat3]

Awesome, can't wait to test this! Is installing from git good or should I switch to a different branch?

master should work fine, you just need to npm i electron@beta and you're good to go.

[Sid127]

how would one capture via devtools, for the sake of testing?

[SpacingBat3]

how would one capture via devtools, for the sake of testing?

I mean, I just use getDisplayMedia and put it in <video> tag (then I play it via secondary output device, which isn't captured via WebCord, so echo doesn't loop when happens), I guess this was more testing how it works in general rather than with Discord, I guess the might check it with alt account or sth if they want to test it in the practise. In general, beta builds of Electron are required and possibly also the use of the new API for providing getDisplayMedia().

[guest271314]

But again if that's the case, with Linux the one could still make a sink in order to precisely select which streams to use for the recording and workaround this issue on their own. I don't think something like this can be made on Windows.

I havn't used Windows in years. You are right. That's one way we can capture the output of window.speechSynthesis.speak(), or more specifically, Speech Dispatcher, where Web Speech API speak() does not output audio on the tab, rather at the system level on Linux, so getDisplayMedia() doesn't capture the audio, for how to capture specific (virtual) devices on Linux using PulseAudio guest271314/SpeechSynthesisRecorder#17 (comment). For more experiments, and working code on Linux see https://github.com/guest271314/captureSystemAudio.

Have you explored all options on Windows?

[guest271314]

I tested Chromium's implementation on Version 123.0.6284.0 (Developer Build) (64-bit), Linux.

Capturing system audio on the Tab with prefereCurrentTab: true constraint does not work at all.

For Screen share the systemAudio: "include" constraint has no effect; we still have to manually toggle the share audio option in the picker.

The monitor device volume decreases from 100% to ~8% for an unknown reason.

[guest271314]

Looks like we have to disable enable-webrtc-allow-input-volume-adjustment flag for the volume of the captured system audio to not decrease from 100% to 8%.

[SpacingBat3]

The monitor device volume decreases from 100% to ~8% for an unknown reason.

I think this did not occur for me when screen sharing on Discord via the Electron, at least both by looking at the pavucontrol and by playing the audio on my 2nd speakers, I haven't noticed that to happen.

[guest271314]

@SpacingBat3 Here's the code I ran to test

var stream = await navigator.mediaDevices.getDisplayMedia({
  video: true,
  audio: true,
  displaySurface: "screen",
  systemAudio: "include",
  preferCurrentTab: true
});

var recorder = new MediaRecorder(stream);

recorder.onstart = (e) => {
  var utterance = new SpeechSynthesisUtterance("Test");
  utterance.onend = (e) => {
    console.log(e);
    recorder.stop();
  }
  window.speechSynthesis.speak(utterance);
}
recorder.onstop = (e) => recorder.stream.getTracks().forEach((track) => track.stop());
recorder.ondataavailable = (e) => {
  console.log(URL.createObjectURL(e.data));
}

Here's the video demonstrating the volume decrease from 100% to 8% in PAVU as soon as the capture begins https://issues.chromium.org/action/issues/40155218/attachments/53517043?download=true. See https://issues.chromium.org/issues/40155218#comment27.

[guest271314]

@SpacingBat3 Pardon, capture system audio doesn't work at all for Tab capture. Comment out the preferCurrentTab: true constraint https://gist.github.com/guest271314/baaa0b8d4b034ff4e9352af4f2bbf42c.

[SpacingBat3]

@SpacingBat3 Pardon, capture system audio doesn't work at all for Tab capture. Comment out the preferCurrentTab: true constraint gist.github.com/guest271314/baaa0b8d4b034ff4e9352af4f2bbf42c.

Given getDisplayMedia implementation on my side is quite naive (it mostly ignores the arguments given by the browser – it always returns the video, audio is opt-in based on user input and Electron support), I doubt preferCurrentTab will even make it to choose tabs for capture.

And I've tested this right now, I see the volume of monitor device actually decreases by the default, I guess it might be worth to add a workaround for it before I switch to Electron 29. Unless this will be fixed at the upstream before Electron 29.0.0 will be released, of course.

[guest271314]

I'm on Chromium Version 123.0.6284.0 (Developer Build) (64-bit). The workaround I am using is in the linked gist; I launch with these switches set --enable-features=PulseaudioLoopbackForScreenShare --disable-features=WebRtcAllowInputVolumeAdjustment

[guest271314]

Of course you could skip all of that and just set the default input to the monitor device and use getUserMedia() to capture system audio https://github.com/guest271314/captureSystemAudio?tab=readme-ov-file#pulseaudio-module-remap-source

pactl load-module module-remap-source \
  master=@DEFAULT_MONITOR@ \
  source_name=virtmic source_properties=device.description=Virtual_Microphone

[SpacingBat3]

I'm on Chromium Version 123.0.6284.0 (Developer Build) (64-bit). The workaround I am using is in the linked gist; I launch with these switches set --enable-features=PulseaudioLoopbackForScreenShare --disable-features=WebRtcAllowInputVolumeAdjustment

As a side note, switch --enable-features=PulseaudioLoopbackForScreenShare isn't actually needed for me on Electron 29 beta to screen share with audio. Also I'm pretty sure 29 will still be at Chromium 122. Anyway, there's no need to discuss this bug there more IMO, as said I'll be mostly finishing the screen share support once Electron 29 will mature enough.

[guest271314]

I just commented here because I actually tested what you are talking about re a commit landing in Chromium. Electron uses Chromium source code. On Linux on Chromium capture of monitor devices has not been possible without workaround for around 5 years https://chromium.googlesource.com/chromium/src/+/4519c32f528e079f25cb2afc594ecf625f943782. So I don't think my comments are off-topic.

[SpacingBat3]

Given WebCord has received an update to Electron 29 that should support screen share on Linux, I suppose I can just finally close this issue.