AV1 support for HW decoding #1691

legraphista · 2024-12-23T12:42:13Z

legraphista
Dec 23, 2024

Overview

#1685 implemented HW support for CUDA devices. This support works for H264, VP9 but not AV1 (on supported devices)

Expected behavior

Decoding to be done on GPU.

Actual behavior

Decoding is done in software (even though allow_software_fallback is False)

Traceback:
n/a

Investigation

sample media: av1.webm

import av
import time

file = 'av1.webm'

hwaccel = av.codec.hwaccel.HWAccel(device_type='cuda', allow_software_fallback=False)
container = av.open(file, hwaccel=hwaccel)

start_time = time.time()
frame_count = 0
for packet in container.demux(video=0):
    for _ in packet.decode():
        frame_count += 1

hw_time = time.time() - start_time
hw_fps = frame_count / hw_time
container.close()

print(f"Decoded with cuda in {hw_time:.2f}s ({hw_fps:.2f} fps).")

Sanity Check:
FFmpeg:

$ ffmpeg -c:v av1_cuvid -i av1.webm -f null -
...
frame=  300 fps=0.0 q=-0.0 Lsize=N/A time=00:00:10.00 bitrate=N/A speed=17.3x    
# 30 FPS * 17.3x realtime ~ 500FPS decode speed

Py Sample:

$ python test.py
Decoded with cuda in 3.18s (94.25 fps).

Reproduction

see above

Versions

OS: ubuntu 24.04
PyAV runtime:
Built from master, post PR Implemented hardware decoding #1685

PyAV v14.0.1
library configuration: --disable-static --enable-shared --libdir=/tmp/vendor/lib --prefix=/tmp/vendor --disable-alsa --disable-doc --disable-libtheora --disable-libfreetype --disable-libfontconfig --disable-libbluray --disable-libopenjpeg --disable-mediafoundation --enable-gmp --enable-gnutls --enable-libaom --enable-libdav1d --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libspeex --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libxml2 --enable-lzma --enable-zlib --enable-version3 --enable-libx264 --disable-libopenh264 --enable-libx265 --enable-gpl
library license: GPL version 3 or later
libavcodec     61. 19.100
libavdevice    61.  3.100
libavfilter    10.  4.100
libavformat    61.  7.100
libavutil      59. 39.100
libswresample   5.  3.100
libswscale      8.  3.100

I am/tried using the binary wheels
I compiled from source

Research

I have done the following:

Checked the PyAV documentation
Searched on Google
Searched on Stack Overflow
Looked through old GitHub issues

Additional context

Tests done on an RTX4090, reproduced also on an L4 in GCP

matthewlai · 2024-12-23T15:24:06Z

matthewlai
Dec 23, 2024

A few questions:

How do you know it's not doing hardware decoding? Is it just the speed difference? Currently the hw decode pipeline in PyAV hasn't been very optimised, so the speed is not expected to match ffmpeg CLI. You should be able to verify with nvidia-smi while the decode is happening.
Does this work? ffmpeg -hwaccel cuda -i av1.webm -f null -
Are you building PyAV with custom ffmpeg? The ffmpeg shipped with PyAV wheels don't have CUDA support

0 replies

legraphista · 2024-12-23T15:29:32Z

legraphista
Dec 23, 2024
Author

How do you know it's not doing hardware decoding? Is it just the speed difference? Currently the hw decode pipeline in PyAV hasn't been very optimised, so the speed is not expected to match ffmpeg CLI.

I re-compiled ffmpeg without libaom and the above code stopped working, whereas the ffmpeg command that uses HW decoding continued to work

Does this work? ffmpeg -hwaccel cuda -i av1.webm -f null -

$ ffmpeg -hwaccel cuda -i av1.webm -f null -
...
frame=  300 fps=0.0 q=-0.0 Lsize=N/A time=00:00:10.00 bitrate=N/A speed=11.8x

$ echo $?
0

Full Logs

Tested on an GCP L4.
av1.mkv and av1.webm are the exact same file, github wouldn't allow me to upload an mkv.

(base) stefan@av1-decode-test-on-l4-with-pyav:~$ ffmpeg -c:v av1_cuvid -i av1.mkv -f null -
ffmpeg version git-2024-12-23-6c9218d Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --enable-nonfree --enable-cuda-nvcc --enable-cuda --enable-cuvid --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --enable-nonfree --enable-gpl --enable-cuda-nvcc --disable-static --enable-shared
  libavutil      59. 51.100 / 59. 51.100
  libavcodec     61. 28.100 / 61. 28.100
  libavformat    61.  9.101 / 61.  9.101
  libavdevice    61.  4.100 / 61.  4.100
  libavfilter    10.  6.101 / 10.  6.101
  libswscale      8. 12.100 /  8. 12.100
  libswresample   5.  4.100 /  5.  4.100
  libpostproc    58.  4.100 / 58.  4.100
Input #0, matroska,webm, from 'av1.mkv':
  Metadata:
    ENCODER         : Lavf61.9.101
  Duration: 00:00:10.00, start: 0.000000, bitrate: 59 kb/s
  Stream #0:0: Video: av1 (Main), yuv420p(tv, progressive), 3840x2160, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn
    Metadata:
      ENCODER         : Lavc61.27.101 libaom-av1
      DURATION        : 00:00:10.000000000
Stream mapping:
  Stream #0:0 -> #0:0 (av1 (av1_cuvid) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf61.9.101
  Stream #0:0: Video: wrapped_avframe, nv12(tv, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30 fps, 30 tbn
    Metadata:
      encoder         : Lavc61.28.100 wrapped_avframe
      DURATION        : 00:00:10.000000000
[out#0/null @ 0x55beb87c8100] video:129KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame=  300 fps=0.0 q=-0.0 Lsize=N/A time=00:00:10.00 bitrate=N/A speed=  10x




(base) stefan@av1-decode-test-on-l4-with-pyav:~$ ffmpeg -hwaccel cuda -i av1.mkv -f null -
ffmpeg version git-2024-12-23-6c9218d Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --enable-nonfree --enable-cuda-nvcc --enable-cuda --enable-cuvid --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --enable-nonfree --enable-gpl --enable-cuda-nvcc --disable-static --enable-shared
  libavutil      59. 51.100 / 59. 51.100
  libavcodec     61. 28.100 / 61. 28.100
  libavformat    61.  9.101 / 61.  9.101
  libavdevice    61.  4.100 / 61.  4.100
  libavfilter    10.  6.101 / 10.  6.101
  libswscale      8. 12.100 /  8. 12.100
  libswresample   5.  4.100 /  5.  4.100
  libpostproc    58.  4.100 / 58.  4.100
Input #0, matroska,webm, from 'av1.mkv':
  Metadata:
    ENCODER         : Lavf61.9.101
  Duration: 00:00:10.00, start: 0.000000, bitrate: 59 kb/s
  Stream #0:0: Video: av1 (Main), yuv420p(tv, progressive), 3840x2160, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn
    Metadata:
      ENCODER         : Lavc61.27.101 libaom-av1
      DURATION        : 00:00:10.000000000
Stream mapping:
  Stream #0:0 -> #0:0 (av1 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf61.9.101
  Stream #0:0: Video: wrapped_avframe, nv12(tv, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30 fps, 30 tbn
    Metadata:
      encoder         : Lavc61.28.100 wrapped_avframe
      DURATION        : 00:00:10.000000000
[out#0/null @ 0x55f467eb5100] video:129KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame=  300 fps=176 q=-0.0 Lsize=N/A time=00:00:10.00 bitrate=N/A speed=5.88x    




(base) stefan@av1-decode-test-on-l4-with-pyav:~$ ffmpeg -i av1.mkv -f null -
ffmpeg version git-2024-12-23-6c9218d Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --enable-nonfree --enable-cuda-nvcc --enable-cuda --enable-cuvid --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --enable-nonfree --enable-gpl --enable-cuda-nvcc --disable-static --enable-shared
  libavutil      59. 51.100 / 59. 51.100
  libavcodec     61. 28.100 / 61. 28.100
  libavformat    61.  9.101 / 61.  9.101
  libavdevice    61.  4.100 / 61.  4.100
  libavfilter    10.  6.101 / 10.  6.101
  libswscale      8. 12.100 /  8. 12.100
  libswresample   5.  4.100 /  5.  4.100
  libpostproc    58.  4.100 / 58.  4.100
Input #0, matroska,webm, from 'av1.mkv':
  Metadata:
    ENCODER         : Lavf61.9.101
  Duration: 00:00:10.00, start: 0.000000, bitrate: 59 kb/s
  Stream #0:0: Video: av1 (Main), yuv420p(tv, progressive), 3840x2160, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn
    Metadata:
      ENCODER         : Lavc61.27.101 libaom-av1
      DURATION        : 00:00:10.000000000
Stream mapping:
  Stream #0:0 -> #0:0 (av1 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[av1 @ 0x55998e39d300] Your platform doesn't support hardware accelerated AV1 decoding.
[av1 @ 0x55998e39d300] Failed to get pixel format.
[av1 @ 0x55998e39d300] Get current frame error
[vist#0:0/av1 @ 0x55998e38ab80] [dec:av1 @ 0x55998e39c780] Error submitting packet to decoder: Function not implemented
[av1 @ 0x55998e39d300] Your platform doesn't support hardware accelerated AV1 decoding.
[av1 @ 0x55998e39d300] Failed to get pixel format.
[av1 @ 0x55998e39d300] Get current frame error
[vist#0:0/av1 @ 0x55998e38ab80] [dec:av1 @ 0x55998e39c780] Error submitting packet to decoder: Function not implemented
[av1 @ 0x55998e39d300] Your platform doesn't support hardware accelerated AV1 decoding.
[av1 @ 0x55998e39d300] Failed to get pixel format.
[av1 @ 0x55998e39d300] Get current frame error
[vist#0:0/av1 @ 0x55998e38ab80] [dec:av1 @ 0x55998e39c780] Error submitting packet to decoder: Function not implemented
[av1 @ 0x55998e39d300] Your platform doesn't support hardware accelerated AV1 decoding.
[av1 @ 0x55998e39d300] Failed to get pixel format.
[av1 @ 0x55998e39d300] Get current frame error
[vist#0:0/av1 @ 0x55998e38ab80] [dec:av1 @ 0x55998e39c780] Error submitting packet to decoder: Function not implemented
[av1 @ 0x55998e39d300] Your platform doesn't support hardware accelerated AV1 decoding.
[av1 @ 0x55998e39d300] Failed to get pixel format.
[av1 @ 0x55998e39d300] Get current frame error

... truncated for brevity ...

[vist#0:0/av1 @ 0x55998e38ab80] [dec:av1 @ 0x55998e39c780] Error submitting packet to decoder: Function not implemented
[av1 @ 0x55998e39d300] Your platform doesn't support hardware accelerated AV1 decoding.
[av1 @ 0x55998e39d300] Failed to get pixel format.
[av1 @ 0x55998e39d300] Get current frame error
[vist#0:0/av1 @ 0x55998e38ab80] [dec:av1 @ 0x55998e39c780] Error submitting packet to decoder: Function not implemented
[av1 @ 0x55998e39d300] Your platform doesn't support hardware accelerated AV1 decoding.
[av1 @ 0x55998e39d300] Failed to get pixel format.
[av1 @ 0x55998e39d300] Get current frame error
[vist#0:0/av1 @ 0x55998e38ab80] [dec:av1 @ 0x55998e39c780] Error submitting packet to decoder: Function not implemented
[vist#0:0/av1 @ 0x55998e38ab80] [dec:av1 @ 0x55998e39c780] Decode error rate 1 exceeds maximum 0.666667
[vist#0:0/av1 @ 0x55998e38ab80] [dec:av1 @ 0x55998e39c780] Task finished with error code: -1145393733 (Error number -1145393733 occurred)
[vist#0:0/av1 @ 0x55998e38ab80] [dec:av1 @ 0x55998e39c780] Terminating thread with return code -1145393733 (Error number -1145393733 occurred)
[vf#0:0 @ 0x55998e39b980] No filtered frames for output stream, trying to initialize anyway.
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf61.9.101
  Stream #0:0: Video: wrapped_avframe, yuv420p(progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30 fps, 1k tbn
    Metadata:
      encoder         : Lavc61.28.100 wrapped_avframe
      DURATION        : 00:00:10.000000000
[out#0/null @ 0x55998e396740] video:0KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
[out#0/null @ 0x55998e396740] Output file is empty, nothing was encoded(check -ss / -t / -frames parameters if used)
frame=    0 fps=0.0 q=0.0 Lsize=N/A time=N/A bitrate=N/A speed=N/A    
Conversion failed!

3. Are you building PyAV with custom ffmpeg? The ffmpeg shipped with PyAV wheels don't have CUDA support

Yes, compiled ffmpeg with CUDA support & shared libraries, installed on the system (sudo make install && sudo ldconfig) before (re-)installing pyav from master

0 replies

matthewlai · 2024-12-23T15:57:17Z

matthewlai
Dec 23, 2024

If you run python -m av --hwconfigs do you get a hwconfig for av1?

Can you do print(container.streams.video[0].codec_context.is_hwaccel)?

The way allow_software_fallback works right now is a bit misleading. It controls the case where the hardware decoder can be found and opened, but something goes wrong trying to set it up to decode the actual video (eg. resolution too high, subsampling mode not supported, etc). In that case if allow_software_fallback is set, we allow the decoder to do software decoding instead.

In the case that the hardware decoder is never there to begin with, we silently fall back to software decoding, regardless of the allow_software_fallback setting. This is necessary because in many files you have multiple streams with different encodings, and there will be at least one that can't be hardware decoded (eg MJPEG). We don't want to fail in those cases. Maybe we can raise an exception if no stream can be accelerated.

0 replies

legraphista · 2024-12-23T16:11:11Z

legraphista
Dec 23, 2024
Author

If you run python -m av --hwconfigs do you get a hwconfig for av1?

(base) stefan@av1-decode-test-on-l4-with-pyav:~$ python -m av --hwconfigs
Hardware configs:
    av1
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33475c00>
    av1_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33146890>
    h263
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x3347e5a0>
    h263p
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x3347e5a0>
    h264
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x3347e680>
    h264_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33146890>
    hevc
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x3347e800>
    hevc_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33146890>
    mjpeg
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x3347fa20>
    mjpeg_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33146890>
    mpeg1_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33146890>
    mpeg1video
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x334800f0>
    mpeg2_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33146890>
    mpeg2video
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33480080>
    mpeg4
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33480460>
    mpeg4_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33146890>
    vc1
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33483ac0>
    vc1_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33146890>
    vp8
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33483f60>
    vp8_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33146890>
    vp9
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33484100>
    vp9_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x33146890>
    wmv3
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x334839e0>

Can you do print(container.streams.video[0].codec_context.is_hwaccel)?

(base) stefan@av1-decode-test-on-l4-with-pyav:~$ python
Python 3.10.15 | packaged by conda-forge | (main, Oct 16 2024, 01:24:24) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import av
>>> container = av.open('av1.mkv')
>>> print(container.streams.video[0].codec_context.is_hwaccel)
False
>>> hwaccel = av.codec.hwaccel.HWAccel(device_type='cuda', allow_software_fallback=False)
>>> container = av.open('av1.mkv', hwaccel=hwaccel)
>>> print(container.streams.video[0].codec_context.is_hwaccel)
True
>>>

strangely, the paths diverge here.
On the GCP L4, I cannot decode the stream.
But on my PC where I have libaom compiled and configured (and can decode the file using the weird software unintended fallback, verified with nvtop), I get the following:

container = av.open('av1.mkv')
print(container.streams.video[0].codec_context.is_hwaccel)
# False

hwaccel = av.codec.hwaccel.HWAccel(device_type='cuda', allow_software_fallback=False)
container = av.open('av1.mkv', hwaccel=hwaccel)
print(container.streams.video[0].codec_context.is_hwaccel)
# False

both systems have AV1 HW support

Hardware configs:
    av1
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x7a6ba5c0>
    av1_cuvid
        <av.HWConfig device_type=cuda format=cuda is_supported=True at 0x7a3bba80>

0 replies

matthewlai · 2024-12-23T17:38:37Z

matthewlai
Dec 23, 2024

Yeah that is very strange.

In the first case, it seems like everything looks fine on our side, except the decoder is doing software decode behind our back. I'm not sure what would cause that. I suspect this will require a deeper dive into libav than I have time for. If you want to debug this, I would start from the ffmpeg hw_decode example, see if that works. If it does, compare all the libav calls it makes to the ones we make, and see if you spot any relevant difference. The existing implementation should already follow that pretty closely.

For the second case, you can add some prints here to figure out why we can't find a matching hardware config even though it seems to exist.

0 replies

matthewlai · 2024-12-23T18:07:53Z

matthewlai
Dec 23, 2024

My suspicion is that we are looking up a decoder to use from just the stream, and it returns the general av1 decoder. That's supposed to support accessing hardware decoders through some hooks. This is supposed to work, and does work with h264 and hevc, but seemingly not with av1 for some reason.

The comments here suggests it's better to use the external decoder directly instead. But it's not clear how to find a suitable decoder in that case (we are using avcodec_find_decoder right now).

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AV1 support for HW decoding #1691

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

AV1 support for HW decoding #1691

legraphista Dec 23, 2024

Overview

Expected behavior

Actual behavior

Investigation

Reproduction

Versions

Research

Additional context

Replies: 6 comments

matthewlai Dec 23, 2024

legraphista Dec 23, 2024 Author

matthewlai Dec 23, 2024

legraphista Dec 23, 2024 Author

matthewlai Dec 23, 2024

matthewlai Dec 23, 2024

legraphista
Dec 23, 2024

matthewlai
Dec 23, 2024

legraphista
Dec 23, 2024
Author

matthewlai
Dec 23, 2024

legraphista
Dec 23, 2024
Author

matthewlai
Dec 23, 2024

matthewlai
Dec 23, 2024