Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implemented hardware decoding #1685

Merged
merged 6 commits into from
Dec 17, 2024
Merged

Implemented hardware decoding #1685

merged 6 commits into from
Dec 17, 2024

Conversation

matthewlai
Copy link
Contributor

This implements hardware decoding continuing from the original work of @rvillalba-novetta and refactoring by @mikeboers in main...rvillalba-novetta:PyAV:hwaccel (and children commits)

I've completed the refactoring, fixed a few bugs, and brought the changes up to date with both PyAV at HEAD and ffmpeg 7.

I've also added a test and an example script. The test cannot be run in CI for obvious reasons (GitHub runners don't have GPUs), but I've manually run it on Mac, Windows, and Linux (WSL).

I understand that @mikeboers didn't find it worthwhile back in the days, but the situation has changed significantly since then - now almost every machine has a supported decoder, and modern encodings (HEVC, VP9, AV1) are so compute-intensive to decode that they have practically been designed for hardware decoding. As an example, I've been working with 4K HEVC videos (standard format coming out of action cameras and phones these days), and my Apple M3 can only decode them at 15 fps, while VideoToolbox easily does 100+ fps. I believe now and especially going forward, hardware decoding will be a necessity for almost any application that cares about performance or need realtime decoding.

Performance tests with a 4K HEVC video (using the new example script) -
MacBook Air M3:
CPU (M3) - 16.3 fps
GPU (M3, videotoolbox) - 121.4 fps

Windows:
CPU (AMD Ryzen 5 3500X) - 15.6 fps
GPU (NVIDIA RTX 3060 Ti, using d3d11va) - 49.1 fps

Linux (WSL):
CPU (AMD Ryzen 5 3500X) - 15.5 fps
GPU (NVIDIA RTX 3060 Ti, using cuda) - 91.3 fps

Notes:

  • Windows accelerated performance is much lower than using ffmpeg directly (ffmpeg -hwaccel d3d11va -i ... -f null -). I don't know why but it's hard for me to work on Windows because I haven't figured out how to build PyAV on Windows and have to rely on GitHub Action. Using ffmpeg I get around 120 fps with d3d11va, and 150 fps with cuda. Maybe we need threading to interleave PCI-E transfers with decode?
  • On Linux it's a bit slower but probably close enough. On Mac it's almost as fast as ffmpeg CLI.
  • The ffmpeg included in wheels already has videotoolbox and d3d11va/d3d12va, so Windows and Mac are pretty well covered. Linux mostly only has vendor APIs and none are being built right now. Windows would also benefit from vendor APIs, but D3D11 is probably good enough for now (for NVIDIA it's about 15% slower).
  • No new failures in existing tests.

Thanks!

av/codec/codec.pyx Outdated Show resolved Hide resolved
@WyattBlue
Copy link
Member

make lint is failing.

12 tests are also failing on my physical MacOS machine. The errors are the same as Github Actions

@matthewlai
Copy link
Contributor Author

Sorry about that. I thought they were existing failures but I forgot to do a make clean when I syned back, and apparently the .so's didn't get rebuilt? I'll investigate, unless you can see why formats may be getting set to None in streams?

@matthewlai
Copy link
Contributor Author

matthewlai commented Dec 15, 2024

All tests pass locally for me now. Also made make lint happy.

This implements hardware decoding continuing from the work of @rvillalba-novetta and @mikeboers in main...rvillalba-novetta:PyAV:hwaccel (and children commits)
@WyattBlue WyattBlue force-pushed the hwaccel branch 2 times, most recently from 44f8397 to 741834d Compare December 17, 2024 11:26
@WyattBlue
Copy link
Member

WyattBlue commented Dec 17, 2024

I've cleaned up the things that remained a problem.

@WyattBlue WyattBlue merged commit 2c63608 into PyAV-Org:main Dec 17, 2024
8 checks passed
Comment on lines +32 to +37
if HW_DEVICE is None:
av.codec.hwaccel.dump_hwdevices()
print("Please set HW_DEVICE.")
exit()

assert HW_DEVICE in av.codec.hwaccel.hwdevices_available, f"{HW_DEVICE} not available."

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the example is a little outdated here:

  • there is no dump_hwdevices member in av.codec.hwaccel
  • av.codec.hwaccel.hwdevices_available is a cyfunction, not a list, set or dict

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1689 should fix it.

@matthewlai
Copy link
Contributor Author

The HWAccel and HWAccelContext merge is also creating problems with multiple streams sharing one context. I'm preparing a new PR to fix these issues.

@legraphista legraphista mentioned this pull request Dec 23, 2024
6 tasks
@legraphista
Copy link

The HWAccel and HWAccelContext merge is also creating problems with multiple streams sharing one context. I'm preparing a new PR to fix these issues.

If you're doing a second PR, could you also have a look at #1690?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants