Audio & Video Sync Issue

[nirmala-ncompass]

Hi @pedroSG94 My Custom Implementation is:
I am recording audio and video in file, mean while I am extending and modified AndroidMuxerRecordController class to get audio and video buffers. In my custom class I am receiving it in one method (onBuffervailable) Here based on my video frame count i am splitting chunks and writing in different file. Here I am resetting Buffer PTS value for each file. Kindly check below code. Audio and video is good in local video. But i my case when i am stetching those videos in mediaLive. The video is good but audio is playing bit delay. Its not in sync. Could you please help me How to set new PTS value for the buffers in one file with audio and video in sync?

`

   private static final int VIDEO_TIME_BASE = 1000 * 1000; 
   private static final int AUDIO_TIME_BASE = 1000 * 1000;
  private static final int VIDEO_FRAME_RATE = 30;
  private static final long VIDEO_FRAME_DURATION = VIDEO_TIME_BASE / VIDEO_FRAME_RATE;
  private static final int AUDIO_SAMPLE_RATE = 32000; // 32 kHz
  private static final int AUDIO_FRAME_SIZE = 1024; // AAC frame size
 private static final long AUDIO_FRAME_DURATION = (long) AUDIO_FRAME_SIZE * AUDIO_TIME_BASE / 
 AUDIO_SAMPLE_RATE;
 private long videoPtsCounter = 0;
 private long audioPtsCounter = 0;

     public void onBufferAvailable(ByteBuffer byteBuffer, MediaCodec.BufferInfo bufferInfo, boolean isVideo) {
   
      synchronized (this) {
        
      // Copy the ByteBuffer to prevent overwriting
        ByteBuffer bufferCopy = ByteBuffer.allocate(byteBuffer.remaining());
        bufferCopy.put(byteBuffer);
        bufferCopy.flip();

        long calculatedPts;

        if (isVideo) {
            // Calculate video PTS
            calculatedPts = videoPtsCounter * VIDEO_FRAME_DURATION;
            videoPtsCounter++;
        }
        else {
            // Calculate audio PTS
            calculatedPts = audioPtsCounter * AUDIO_FRAME_DURATION;
            audioPtsCounter++;
        }

        // Set the final adjusted PTS
        bufferInfo.presentationTimeUs = calculatedPts;
        
        // Create buffer data object
        BufferData bufferData = new BufferData(bufferCopy, new MediaCodec.BufferInfo(), isVideo);
        bufferData.bufferInfo.set(bufferInfo.offset, bufferInfo.size, bufferInfo.presentationTimeUs, bufferInfo.flags);

        // Track video frame count
        if (isVideo) {
            videoFrameCounter++;
        } else audioFrameCounter++;
        // Add buffer data to the buffer list
        if (useFirstArray) {
            bufferList1.add(bufferData);
        } else {
            bufferList2.add(bufferData);
        }
        // Process buffer if we hit the frame limit
        if (videoFrameCounter >= MAX_VIDEO_FRAMES) {
            processBufferInBackground(useFirstArray ? bufferList1 : bufferList2, false);
            if (useFirstArray) {
                bufferList1.clear();
            } else {
                bufferList2.clear();
            }
            audioPtsCounter = 0;
            videoPtsCounter = 0;
            audioFrameCounter = 0;
            videoFrameCounter = 0; // Reset frame counter
            useFirstArray = !useFirstArray; // Switch buffer list
        }
    }
}

`

[pedroSG94]

Hello,

Did you tried using the clock to create the pts instead of calculate it based in the config?
You can try it in both (video/audio) or only in one to test:

bufferInfo.presentationTimeUs = System.nanoTime() / 1000 - presentTimeUs;

Where presentTimeUs is (when you start encoder or receive the first frame):

System.nanoTime() / 1000

[nirmala-ncompass]

Thanks @pedroSG94 I tried your logic for PTS calculation But in media live, the entire audio is playing like cracked audio. In local video its good. Basically I want the video and audio is in sync and also good. every file buffers need to reset the PTS. Is it any way to find the fps received in any class method? How could I achieve this custom implementation with audio and video good quality? Give me insights.
`

          if(isVideo && videoBaseTimestamp == -1)
                   videoBaseTimestamp = System.nanoTime() / 1000;
          else if(!isVideo && audioBaseTimestamp == -1)  
                  audioBaseTimestamp = System.nanoTime() / 1000;

        if(isVideo)  {
            bufferInfo.presentationTimeUs = System.nanoTime() / 1000 - videoBaseTimestamp;
        }
        else  {
            bufferInfo.presentationTimeUs = System.nanoTime() / 1000 - audioBaseTimestamp;
        }`

[pedroSG94]

Hello,

I have other possible solution.

If you always have the same delay you can try change the start PTS to the audio artificially using your first way to calculate the PTS.
For example, if you have the audio with a delay of 500ms you can start with PTS value of 500_000_000 (500ms in nanoseconds) instead of 0.

[nirmala-ncompass]

Hi @pedroSG94 Audio and video is good and also in sync with 30FPS devices. For some devices FPA is low - 14 to 28, so audio and video not in sync. I am calculating VIDEO_FRAME_DURATION with static 30FPS value. Is it because of that? Is there any way to know frames received in a devices ? How could I calculate PTS value with support all the devices?

[pedroSG94]

Hello,

You can try calculate only video PTS using the clock like this:

Thanks @pedroSG94 I tried your logic for PTS calculation But in media live, the entire audio is playing like cracked audio. In local video its good. Basically I want the video and audio is in sync and also good. every file buffers need to reset the PTS. Is it any way to find the fps received in any class method? How could I achieve this custom implementation with audio and video good quality? Give me insights. `

          if(isVideo && videoBaseTimestamp == -1)
                   videoBaseTimestamp = System.nanoTime() / 1000;
          else if(!isVideo && audioBaseTimestamp == -1)  
                  audioBaseTimestamp = System.nanoTime() / 1000;

        if(isVideo)  {
            bufferInfo.presentationTimeUs = System.nanoTime() / 1000 - videoBaseTimestamp;
        }
        else  {
            bufferInfo.presentationTimeUs = System.nanoTime() / 1000 - audioBaseTimestamp;
        }`

[nirmala-ncompass]

Hi @pedroSG94 I have a quick question. In Samsung F13 (Android 14), When I am using filters with timer running functionality, the FPS is not in consistent range(below 24), In AWS medialive, its showing 29 for input frame rate in health session. while recording, because of that, audio and video is not in sync. Without filters, its working fine. Please suggest me any way to resolve this issue. Other models are working as expected

[pedroSG94]

Hello,

Sorry for late response

Filters could reduce the fps depend the filter and device that you are using.
Can you tell me the filter used? maybe I can try optimize it.

[nirmala-ncompass]

Hi @pedroSG94 I am using AndroidViewFilterRender

[pedroSG94]

Ok, I will check if I can optimize it.
Meanwhile, are you using the clock to calculate the video timestamp? I recommend you calculate video timestamp using clock.
If you use the fps to calculate video timestamp but the fps is reduced, then you will have that sync problem.

[nirmala-ncompass]

Hi @pedroSG94 I am using FPS logic. But its working fine in all the models except this samsung F13.

[pedroSG94]

Yes, the reason is that Samsung F13 can't produce 30fps (as you said it is working with 24fps) and your method needs stable 30 FPS to work correctly.

I recommend you adapt your method to fix it or use clock only for video because using the clock you don't have that problem