Unable to deserialize webm from audio recorded in Chrome #16

FossaMalaEnt · 2023-07-25T12:59:49Z

Hi,

At first, I wanna thank you for the time you're putting into this project!

I'm trying to record the audio from Chrome, route the WebM opus stream to .NET, convert to another format and re-route the audio to another system, for capturing the audio i'm using something like this:

        const stream = await navigator.mediaDevices.getUserMedia({ audio: true });

        this.mediaRecorder = new MediaRecorder(stream, {
            mimeType: 'audio/webm; codecs=opus'
        });

        this.mediaRecorder.start();

        this.isRecording = true;

        console.log("Starting media recorder ", this.mediaRecorder);

        this.mediaRecorder.ondataavailable = (e) => {
            //Do things, convert bytes to base64 and send the audio to the .NET back-end
        }

Now, following the examples, I'm using this code in order to extract the opus audio from the WebM stream:

        string b64Audio = audioPkt.Audio;
        byte[] webmBytes = Convert.FromBase64String(b64Audio);
        MemoryStream webmStream = new MemoryStream(webmBytes);

        var oggStream = new MemoryStream();

        MatroskaDemuxer.ExtractOggOpusAudio(webmStream, oggStream);

I confirm the webm bytes are the same recorded from the js, so no change in data, but in the line MatroskaDemuxer.ExtractOggOpusAudio(webmStream, oggStream); I got the following error:

System.ArgumentNullException: 'Value cannot be null. Arg_ParamName_Name'

Stack trace:

in Matroska.Muxer.OggOpus.OggOpusAudioStreamDemuxer.CopyTo(MatroskaDocument doc, Stream outputStream, OggOpusAudioStreamDemuxerSettings settings)
in Matroska.Muxer.MatroskaDemuxer.ExtractOggOpusAudio(MatroskaDocument doc, Stream outputStream, OggOpusAudioStreamDemuxerSettings settings)
in PROJECT_NAME.API.Hubs.LiveDataHub.SendAudio(ReceiveAudioDTO audioPkt) in PATH_TO_PROJECT\PROJECT_NAME.API\Hubs\LiveDataHub.cs: riga 70
in Microsoft.Extensions.Internal.ObjectMethodExecutor.<>c__DisplayClass33_0.b__0(Object target, Object[] parameters)
in Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher`1.d__23.MoveNext()

I'm not able to understand what's the cause of the problem looking into the examples and in the various docs of the repo, Am I missing something? Any idea of what can be the issue?

The text was updated successfully, but these errors were encountered:

StefH · 2023-07-25T14:45:21Z

Did you try the same code using a valid example webm file?

FossaMalaEnt · 2023-07-25T15:13:45Z

Nope, if you need this test to confim that your package works on my machine I'll immediatly do it, but my requirement is to get the opus audio stream from the webm opus recorded audio from Chrome, It's you're package supposed to works only when reading webm files (FileStream)?

StefH · 2023-07-25T15:16:28Z

Can you try this file?
Estas Tonne - Internal Flight Experience (Live in Cluj Napoca).webm

FossaMalaEnt · 2023-07-25T15:40:16Z

Just tried the file you provided and with another file i've manually converted with an online converter and it looks working fine, the problem appear only when using audio data streamed from js using the MediaRecorder API.

If can be usefull, this is the MatroskaDocument of your webm file once deserialized:

{
    "Ebml": {
        "DocType": "webm",
        "DocTypeExtension": null,
        "DocTypeReadVersion": 2,
        "DocTypeVersion": 4,
        "EBMLMaxIDLength": 4,
        "EBMLMaxSizeLength": 8,
        "EBMLReadVersion": 1,
        "EBMLVersion": 1,
        "Void": null,
        "CRC32": null
    },
    "Segment": {
        "Attachments": null,
        "Chapters": null,
        "Clusters": [Too long
        ],
        "Cues": {
            "CuePoints": [Too long
            ],
            "Void": null,
            "CRC32": null
        },
        "Info": {
            "ChapterTranslate": null,
            "DateUTC": null,
            "Duration": 419941.0,
            "MuxingApp": "google/video-file",
            "NextFilename": null,
            "NextUID": null,
            "PrevFilename": null,
            "PrevUID": null,
            "SegmentFamily": null,
            "SegmentFilename": null,
            "SegmentUID": null,
            "TimestampScale": 1000000,
            "Title": null,
            "WritingApp": "google/video-file",
            "Void": null,
            "CRC32": null
        },
        "SeekHead": {
            "Seek": {
                "SeekID": "HFO7aw==",
                "SeekPosition": 218,
                "Void": null,
                "CRC32": null
            },
            "Void": null,
            "CRC32": null
        },
        "Tags": null,
        "Tracks": {
            "TrackEntries": [
                {
                    "AttachmentLink": null,
                    "Audio": {
                        "BitDepth": 16,
                        "ChannelPositions": null,
                        "Channels": 2,
                        "OutputSamplingFrequency": null,
                        "SamplingFrequency": 48000.0,
                        "Void": null,
                        "CRC32": null
                    },
                    "BlockAdditionMapping": null,
                    "CodecDecodeAll": 0,
                    "CodecDelay": 6500000,
                    "CodecDownloadURL": null,
                    "CodecID": "A_OPUS",
                    "CodecInfoURL": null,
                    "CodecName": null,
                    "CodecPrivate": "T3B1c0hlYWQBAjgBgLsAAAAAAA==",
                    "CodecSettings": null,
                    "ContentEncodings": null,
                    "DefaultDecodedFieldDuration": null,
                    "DefaultDuration": null,
                    "FlagDefault": 0,
                    "FlagEnabled": 0,
                    "FlagForced": 0,
                    "FlagLacing": 0,
                    "Language": "eng",
                    "LanguageIETF": null,
                    "MaxBlockAdditionID": 0,
                    "MaxCache": null,
                    "MinCache": 0,
                    "Name": null,
                    "SeekPreRoll": 80000000,
                    "TrackNumber": 1,
                    "TrackOffset": null,
                    "TrackOperation": null,
                    "TrackOverlay": null,
                    "TrackTimestampScale": 0.0,
                    "TrackTranslate": null,
                    "TrackType": 2,
                    "TrackUID": 27231307933070081,
                    "TrickMasterTrackSegmentUID": null,
                    "TrickMasterTrackUID": null,
                    "TrickTrackFlag": null,
                    "TrickTrackSegmentUID": null,
                    "TrickTrackUID": null,
                    "Video": null,
                    "Void": null,
                    "CRC32": null
                }
            ],
            "Void": null,
            "CRC32": null
        },
        "Void": null,
        "CRC32": null
    }
}

and this is the same obj but after a webm audio stream from Chrome:

{
    "Ebml": {
        "DocType": "webm",
        "DocTypeExtension": null,
        "DocTypeReadVersion": 2,
        "DocTypeVersion": 4,
        "EBMLMaxIDLength": 4,
        "EBMLMaxSizeLength": 8,
        "EBMLReadVersion": 1,
        "EBMLVersion": 1,
        "Void": null,
        "CRC32": null
    },
    "Segment": {
        "Attachments": null,
        "Chapters": null,
        "Clusters": null,
        "Cues": null,
        "Info": {
            "ChapterTranslate": null,
            "DateUTC": null,
            "Duration": null,
            "MuxingApp": "Chrome",
            "NextFilename": null,
            "NextUID": null,
            "PrevFilename": null,
            "PrevUID": null,
            "SegmentFamily": null,
            "SegmentFilename": null,
            "SegmentUID": null,
            "TimestampScale": 1000000,
            "Title": null,
            "WritingApp": "Chrome",
            "Void": null,
            "CRC32": null
        },
        "SeekHead": null,
        "Tags": null,
        "Tracks": {
            "TrackEntries": [
                {
                    "AttachmentLink": null,
                    "Audio": {
                        "BitDepth": 32,
                        "ChannelPositions": null,
                        "Channels": 1,
                        "OutputSamplingFrequency": null,
                        "SamplingFrequency": 48000.0,
                        "Void": null,
                        "CRC32": null
                    },
                    "BlockAdditionMapping": null,
                    "CodecDecodeAll": 0,
                    "CodecDelay": null,
                    "CodecDownloadURL": null,
                    "CodecID": "A_OPUS",
                    "CodecInfoURL": null,
                    "CodecName": null,
                    "CodecPrivate": "T3B1c0hlYWQBAQAAgLsAAAAAAA==",
                    "CodecSettings": null,
                    "ContentEncodings": null,
                    "DefaultDecodedFieldDuration": null,
                    "DefaultDuration": null,
                    "FlagDefault": 0,
                    "FlagEnabled": 0,
                    "FlagForced": 0,
                    "FlagLacing": 0,
                    "Language": null,
                    "LanguageIETF": null,
                    "MaxBlockAdditionID": 0,
                    "MaxCache": null,
                    "MinCache": 0,
                    "Name": null,
                    "SeekPreRoll": 0,
                    "TrackNumber": 1,
                    "TrackOffset": null,
                    "TrackOperation": null,
                    "TrackOverlay": null,
                    "TrackTimestampScale": 0.0,
                    "TrackTranslate": null,
                    "TrackType": 2,
                    "TrackUID": 8083526733988260,
                    "TrickMasterTrackSegmentUID": null,
                    "TrickMasterTrackUID": null,
                    "TrickTrackFlag": null,
                    "TrickTrackSegmentUID": null,
                    "TrickTrackUID": null,
                    "Video": null,
                    "Void": null,
                    "CRC32": null
                }
            ],
            "Void": null,
            "CRC32": null
        },
        "Void": null,
        "CRC32": null
    }
}

FossaMalaEnt · 2023-07-25T16:08:44Z

Just looked at the console and I noticed the following error (looks different from the debugging context, I don't know why):

ERROR: Segment at position 146 not mapped. Exception: NEbml.Core.EbmlDataFormatException: invalid element size value
at NEbml.Core.EbmlReader.ReadNext()
at Matroska.MatroskaSerializer.Deserialize(Type type, EbmlReader reader)
fail: Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher[8]
Failed to invoke hub method 'SendAudio'.
System.ArgumentNullException: Value cannot be null. (Parameter 'Clusters')
at Matroska.Muxer.OggOpus.OggOpusAudioStreamDemuxer.CopyTo(MatroskaDocument doc, Stream outputStream, OggOpusAudioStreamDemuxerSettings settings)
at Matroska.Muxer.MatroskaDemuxer.ExtractOggOpusAudio(MatroskaDocument doc, Stream outputStream, OggOpusAudioStreamDemuxerSettings settings)
at PROJECT_NAME.API.Hubs.LiveDataHub.SendAudio(ReceiveAudioDTO audioPkt) in PATH_TO_PROJECT\PROJECT_NAME\PROJECT_NAME.API\Hubs\LiveDataHub.cs:line 72
at Microsoft.Extensions.Internal.ObjectMethodExecutor.<>c__DisplayClass33_0.b__0(Object target, Object[] parameters)
at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.ExecuteMethod(ObjectMethodExecutor methodExecutor, Hub hub, Object[] arguments) at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.g__ExecuteInvocation|18_0(DefaultHubDispatcher1 dispatcher, ObjectMethodExecutor methodExecutor, THub hub, Object[] arguments, AsyncServiceScope scope, IHubActivator1 hubActivator, HubConnectionContext connection, HubMethodInvocationMessage hubMethodInvocationMessage, Boolean isStreamCall)

StefH · 2023-07-25T16:48:46Z

Can you provide your file here as attachment?

FossaMalaEnt · 2023-07-26T09:55:30Z

I used this code to save the file after recording the audio:

 this.mediaRecorder.ondataavailable = (e) => {
   const a = document.createElement("a");
   document.body.appendChild(a);
   const url = window.URL.createObjectURL(e.data);
   a.href = url;
   a.download = "test.webm";
   a.click();
   window.URL.revokeObjectURL(url);
   document.body.removeChild(a);

   //...
}

Here the test file recorded and saved with the posted code:
test.webm

The problem persist when I try to extract the ogg from this webm, but the default media player of windows 10/11 can play the audio without problems.

StefH · 2023-07-27T05:52:11Z

It can be that this file follows a newer EBML / Matroska specification.

I'll take a look if I can update some code...

StefH · 2023-07-29T12:19:08Z

@FossaMalaEnt
I've made a fix.

Can you try version 0.0.11-preview-01. ?

StefH · 2023-08-05T10:32:44Z

@FossaMalaEnt did you have time to test that preview version?

FossaMalaEnt · 2023-08-10T09:18:57Z

@StefH, sorry for the delay. I downloaded the code from the "issue16" branch to give it a try, but I'm having trouble compiling it in both VS 2019 and 2022. It seems that VS is unable to locate the "Span<>" class. Have you published the DLL somewhere?

StefH · 2023-08-10T09:50:44Z

I did push version 0.0.11-preview-01

Make sure to enable preview in VS:

StefH · 2023-12-29T07:46:41Z

@FossaMalaEnt
Did you have time to validate that preview version?

StefH · 2024-05-09T06:44:11Z

@FossaMalaEnt
Did you have time to validate that preview version?

StefH · 2024-09-17T19:21:46Z

@FossaMalaEnt
Did you have time to validate that preview version?

FossaMalaEnt changed the title ~~Unable to deserialize webm from recorded Chrome~~ Unable to deserialize webm from audio recorded in Chrome Jul 25, 2023

StefH self-assigned this Jul 26, 2023

StefH mentioned this issue Jul 28, 2023

EbmlReader.ReadElement is wrong? OlegZee/NEbml#13

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to deserialize webm from audio recorded in Chrome #16

Unable to deserialize webm from audio recorded in Chrome #16

FossaMalaEnt commented Jul 25, 2023 •

edited

Loading

StefH commented Jul 25, 2023

FossaMalaEnt commented Jul 25, 2023 •

edited

Loading

StefH commented Jul 25, 2023

FossaMalaEnt commented Jul 25, 2023 •

edited

Loading

FossaMalaEnt commented Jul 25, 2023

StefH commented Jul 25, 2023

FossaMalaEnt commented Jul 26, 2023 •

edited

Loading

StefH commented Jul 27, 2023

StefH commented Jul 29, 2023

StefH commented Aug 5, 2023

FossaMalaEnt commented Aug 10, 2023

StefH commented Aug 10, 2023

StefH commented Dec 29, 2023

StefH commented May 9, 2024

StefH commented Sep 17, 2024

Unable to deserialize webm from audio recorded in Chrome #16

Unable to deserialize webm from audio recorded in Chrome #16

Comments

FossaMalaEnt commented Jul 25, 2023 • edited Loading

StefH commented Jul 25, 2023

FossaMalaEnt commented Jul 25, 2023 • edited Loading

StefH commented Jul 25, 2023

FossaMalaEnt commented Jul 25, 2023 • edited Loading

FossaMalaEnt commented Jul 25, 2023

StefH commented Jul 25, 2023

FossaMalaEnt commented Jul 26, 2023 • edited Loading

StefH commented Jul 27, 2023

StefH commented Jul 29, 2023

StefH commented Aug 5, 2023

FossaMalaEnt commented Aug 10, 2023

StefH commented Aug 10, 2023

StefH commented Dec 29, 2023

StefH commented May 9, 2024

StefH commented Sep 17, 2024

FossaMalaEnt commented Jul 25, 2023 •

edited

Loading

FossaMalaEnt commented Jul 25, 2023 •

edited

Loading

FossaMalaEnt commented Jul 25, 2023 •

edited

Loading

FossaMalaEnt commented Jul 26, 2023 •

edited

Loading