Kernel panic when NVD_BACKEND=direct is used

[TheAifam5]

CPU: AMD Ryzen Threadripper 2950X
OS: Gentoo Linux
Kernel: 6.10.0-pf4-x86_64
Desktop: Hyprland (Wayland)
GPUs: NVIDIA GeForce GTX 970 (primary), NVIDIA GeForce RTX 2070
Drivers: 565.57.01 with CUDA 12.7
Nvidia VAAPI Driver Version: 0.0.13

Affected Software: Firefox 132.0.2
Problem: Opening any stream at twitch.tv immediately crashes the driver. Tested twitch.tv only.
Workaround: Use NVD_BACKEND=egl.

Environment Variables:

AQ_DRM_DEVICES=/dev/dri/card0
XDG_BACKEND=wayland
MOZ_DRM_DEVICE=/dev/dri/renderD128
LIBVA_DRIVER_NAME=nvidia
XDG_SESSION_DESKTOP=Hyprland
XDG_SESSION_TYPE=wayland
WLR_DRM_DEVICES=/dev/dri/card0
MOZ_DISABLE_RDD_SANDBOX=1
EGL_PLATFORM=wayland
MOZ_GMP_PATH=/usr/lib64/nsbrowser/plugins/gmp-gmpopenh264/system-installed
NVD_BACKEND=direct
MOZ_ENABLE_WAYLAND=1
__GLX_VENDOR_LIBRARY_NAME=nvidia
GBM_BACKEND=nvidia-drm
MOZ_WAYLAND_USE_VAAPI=1

Project's Compilation Flags (GCC):
CFLAGS: -march=native -ggdb3 -O3 -pipe -ftree-vectorize -flto=7
CXXFLAGS: -march=native -ggdb3 -O3 -pipe -ftree-vectorize -flto=7
LDFLAGS: -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs

Firefox Compilation Flags (Clang + PGO):
CFLAGS: -march=native -pipe -ftree-vectorize
CXXFLAGS: -march=native -pipe -ftree-vectorize
LDFLAGS: -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,relro -Wl,-z,now -Wl,--undefined-version -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags

NVIDIA Compilation Flags (GCC):
same as project's compilation flags

Modprobe Config:

options nvidia-drm modeset=1

options nvidia \
	NVreg_PreserveVideoMemoryAllocations=1 \
	NVreg_TemporaryFilePath=/var/tmp \
	NVreg_DeviceFileGID=27 \
	NVreg_DeviceFileMode=432 \
	NVreg_DeviceFileUID=0 \
	NVreg_ModifyDeviceFiles=1 \
	NVreg_UsePageAttributeTable=1 \
	NVreg_InitializeSystemMemoryAllocations=1 \
	NVreg_EnableStreamMemOPs=1 \
	NVreg_EnablePCIeGen3=1 \
	NVreg_DynamicPowerManagement=0x02

options nvidia-drm fbdev=1

Log:

Nov 17 00:25:19.109792 pussyripper kernel: Oops: general protection fault, probably for non-canonical address 0x22818395b152149d: 0000 [#1] PREEMPT SMP NOPTI
Nov 17 00:25:19.109974 pussyripper kernel: CPU: 1 PID: 6977 Comm: MediaPD~oder #1 Tainted: P           O       6.10.0-pf4-x86_64 #2
Nov 17 00:25:19.110004 pussyripper kernel: Hardware name: System manufacturer System Product Name/ROG STRIX X399-E GAMING, BIOS 1602 01/30/2024
Nov 17 00:25:19.110029 pussyripper kernel: RIP: 0010:_nv017261rm+0x54/0x220 [nvidia]
Nov 17 00:25:19.110053 pussyripper kernel: Code: 1f 00 00 00 48 39 f1 0f 82 17 01 00 00 49 8b 45 00 48 85 c0 75 14 e9 8e 01 00 00 0f 1f 00 48 8b 50 28 48 85 d2 74 1c 48 89 d0 <48> 3b 48 08 72 ee 48 3b 70 10 0f 86 64 01 00 00 48 8b 50 30 48 85
Nov 17 00:25:19.110123 pussyripper kernel: RSP: 0018:ffffb54657be78f8 EFLAGS: 00010206
Nov 17 00:25:19.110149 pussyripper kernel: RAX: 22818395b1521495 RBX: ffff924864197dd0 RCX: 00000000caf007ff
Nov 17 00:25:19.110173 pussyripper kernel: RDX: 22818395b1521495 RSI: 00000000caf007ff RDI: ffff924864197dd0
Nov 17 00:25:19.110196 pussyripper kernel: RBP: ffff9249799fdae0 R08: ffff9248b7efb408 R09: ffff924864197c08
Nov 17 00:25:19.110219 pussyripper kernel: R10: 0000000000000000 R11: ffffe12a84906400 R12: ffff9248e0d26430
Nov 17 00:25:19.110242 pussyripper kernel: R13: ffff9248db84b428 R14: 0000000000000000 R15: ffff9248db84b008
Nov 17 00:25:19.110265 pussyripper kernel: FS:  000078f9fbe006c0(0000) GS:ffff92577e280000(0000) knlGS:0000000000000000
Nov 17 00:25:19.110288 pussyripper kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 17 00:25:19.110315 pussyripper kernel: CR2: 000078f9fbdfc2b0 CR3: 0000000250cc4000 CR4: 00000000003506f0
Nov 17 00:25:19.110338 pussyripper kernel: Call Trace:
Nov 17 00:25:19.110361 pussyripper kernel:  <TASK>
Nov 17 00:25:19.110413 pussyripper kernel:  ? __die_body.cold+0x19/0x27
Nov 17 00:25:19.110444 pussyripper kernel:  ? die_addr+0x38/0x60
Nov 17 00:25:19.110470 pussyripper kernel:  ? exc_general_protection+0x1dc/0x4b0
Nov 17 00:25:19.110491 pussyripper kernel:  ? asm_exc_general_protection+0x22/0x30
Nov 17 00:25:19.110511 pussyripper kernel:  ? _nv017261rm+0x54/0x220 [nvidia]
Nov 17 00:25:19.110531 pussyripper kernel:  ? _nv039465rm+0x29e/0x800 [nvidia]
Nov 17 00:25:19.110551 pussyripper kernel:  ? _nv002766rm+0x4e/0x70 [nvidia]
Nov 17 00:25:19.110571 pussyripper kernel:  ? os_alloc_mem+0xb8/0xd0 [nvidia]
Nov 17 00:25:19.110591 pussyripper kernel:  ? _nv002843rm+0x29/0x70 [nvidia]
Nov 17 00:25:19.110611 pussyripper kernel:  ? _nv006110rm+0x130/0x200 [nvidia]
Nov 17 00:25:19.110631 pussyripper kernel:  ? _nv005374rm+0x4b/0x80 [nvidia]
Nov 17 00:25:19.110651 pussyripper kernel:  ? _nv049255rm+0x98/0x1b0 [nvidia]
Nov 17 00:25:19.110702 pussyripper kernel:  ? _nv011460rm+0x275/0x5d0 [nvidia]
Nov 17 00:25:19.110723 pussyripper kernel:  ? _nv018258rm+0x23a/0x2b0 [nvidia]
Nov 17 00:25:19.110740 pussyripper kernel:  ? _nv051307rm+0x2ac/0x2f0 [nvidia]
Nov 17 00:25:19.110760 pussyripper kernel:  ? _nv049395rm+0xe2/0x1a0 [nvidia]
Nov 17 00:25:19.110780 pussyripper kernel:  ? _nv049394rm+0x2c/0x40 [nvidia]
Nov 17 00:25:19.110798 pussyripper kernel:  ? _nv051387rm+0x20/0x20 [nvidia]
Nov 17 00:25:19.110818 pussyripper kernel:  ? _nv024637rm+0x426/0x6e0 [nvidia]
Nov 17 00:25:19.110838 pussyripper kernel:  ? _nv024637rm+0x5d5/0x6e0 [nvidia]
Nov 17 00:25:19.110861 pussyripper kernel:  ? rm_gpu_ops_dup_memory+0x28/0x70 [nvidia]
Nov 17 00:25:19.110881 pussyripper kernel:  ? nvUvmInterfaceDupMemory+0x85/0xd0 [nvidia]
Nov 17 00:25:19.110902 pussyripper kernel:  ? uvm_map_external_allocation_on_gpu+0x204/0x480 [nvidia_uvm]
Nov 17 00:25:19.110919 pussyripper kernel:  ? uvm_api_map_external_allocation+0x4dd/0x560 [nvidia_uvm]
Nov 17 00:25:19.110951 pussyripper kernel:  ? uvm_ioctl+0x1654/0x19b0 [nvidia_uvm]
Nov 17 00:25:19.110972 pussyripper kernel:  ? srso_return_thunk+0x5/0x5f
Nov 17 00:25:19.110992 pussyripper kernel:  ? uvm_ioctl+0x1654/0x19b0 [nvidia_uvm]
Nov 17 00:25:19.111009 pussyripper kernel:  ? srso_return_thunk+0x5/0x5f
Nov 17 00:25:19.111027 pussyripper kernel:  ? __mod_memcg_lruvec_state+0xa6/0x150
Nov 17 00:25:19.111046 pussyripper kernel:  ? srso_return_thunk+0x5/0x5f
Nov 17 00:25:19.111060 pussyripper kernel:  ? __lruvec_stat_mod_folio+0x64/0xa0
Nov 17 00:25:19.111081 pussyripper kernel:  ? srso_return_thunk+0x5/0x5f
Nov 17 00:25:19.111095 pussyripper kernel:  ? do_anonymous_page+0x6f6/0x920
Nov 17 00:25:19.111115 pussyripper kernel:  ? _nv048010rm+0x10/0x20 [nvidia]
Nov 17 00:25:19.111135 pussyripper kernel:  ? srso_return_thunk+0x5/0x5f
Nov 17 00:25:19.111166 pussyripper kernel:  ? __pte_offset_map+0x17/0x130
Nov 17 00:25:19.111184 pussyripper kernel:  ? srso_return_thunk+0x5/0x5f
Nov 17 00:25:19.111201 pussyripper kernel:  ? __handle_mm_fault+0xc43/0x11b0
Nov 17 00:25:19.111218 pussyripper kernel:  ? uvm_unlocked_ioctl_entry.part.0+0x49/0xa0 [nvidia_uvm]
Nov 17 00:25:19.111238 pussyripper kernel:  ? __count_memcg_events+0x54/0xf0
Nov 17 00:25:19.111259 pussyripper kernel:  ? srso_return_thunk+0x5/0x5f
Nov 17 00:25:19.111276 pussyripper kernel:  ? handle_mm_fault+0x1a9/0x2c0
Nov 17 00:25:19.111296 pussyripper kernel:  ? srso_return_thunk+0x5/0x5f
Nov 17 00:25:19.111312 pussyripper kernel:  ? __x64_sys_ioctl+0x93/0xd0
Nov 17 00:25:19.111332 pussyripper kernel:  ? do_syscall_64+0x5b/0x170
Nov 17 00:25:19.111349 pussyripper kernel:  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 17 00:25:19.111369 pussyripper kernel:  </TASK>
Nov 17 00:25:19.111389 pussyripper kernel: Modules linked in: nvidia_uvm(PO) xt_conntrack xt_MASQUERADE bridge stp llc nf_conntrack_netlink xt_addrtype nft_compat rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device nf_conntrack_netbios_ns nf_conntrack_broadcast des3_ede_x86_64 des_generic libdes md4 nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables uhid bnep binfmt_misc nvidia_drm(PO) nvidia_modeset(PO) amd_atl intel_rapl_msr intel_rapl_common snd_hda_codec_realtek snd_hda_codec_generic snd_hda_scodec_component rtw88_8822be rtw88_8822b snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi rtw88_pci rtw88_core edac_mce_amd nvidia(PO) snd_hda_codec snd_hda_core btusb mac80211 snd_hwdep eeepc_wmi btrtl kvm_amd snd_pcm btintel asus_wmi btbcm libarc4 snd_timer sparse_keymap btmtk kvm asus_wmi_sensors platform_profile wmi_bmof mxm_wmi igb snd rapl video cfg80211 pcspkr acpi_cpufreq soundcore bluetooth drm_kms_helper
Nov 17 00:25:19.112166 pussyripper kernel:  i2c_algo_bit k10temp i2c_piix4 dca rfkill wmi gpio_amdpt joydev gpio_generic fuse drm nfnetlink ip_tables raid1 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 nvme zfs(PO) sp5100_tco ccp nvme_core spl(O) pkcs8_key_parser
Nov 17 00:25:19.112183 pussyripper kernel: ---[ end trace 0000000000000000 ]---
Nov 17 00:25:19.112195 pussyripper kernel: RIP: 0010:_nv017261rm+0x54/0x220 [nvidia]
Nov 17 00:25:19.112205 pussyripper kernel: Code: 1f 00 00 00 48 39 f1 0f 82 17 01 00 00 49 8b 45 00 48 85 c0 75 14 e9 8e 01 00 00 0f 1f 00 48 8b 50 28 48 85 d2 74 1c 48 89 d0 <48> 3b 48 08 72 ee 48 3b 70 10 0f 86 64 01 00 00 48 8b 50 30 48 85
Nov 17 00:25:19.112215 pussyripper kernel: RSP: 0018:ffffb54657be78f8 EFLAGS: 00010206
Nov 17 00:25:19.112224 pussyripper kernel: RAX: 22818395b1521495 RBX: ffff924864197dd0 RCX: 00000000caf007ff
Nov 17 00:25:19.112234 pussyripper kernel: RDX: 22818395b1521495 RSI: 00000000caf007ff RDI: ffff924864197dd0
Nov 17 00:25:19.112242 pussyripper kernel: RBP: ffff9249799fdae0 R08: ffff9248b7efb408 R09: ffff924864197c08
Nov 17 00:25:19.112250 pussyripper kernel: R10: 0000000000000000 R11: ffffe12a84906400 R12: ffff9248e0d26430
Nov 17 00:25:19.112259 pussyripper kernel: R13: ffff9248db84b428 R14: 0000000000000000 R15: ffff9248db84b008
Nov 17 00:25:19.112269 pussyripper kernel: FS:  000078f9fbe006c0(0000) GS:ffff92577e280000(0000) knlGS:0000000000000000
Nov 17 00:25:19.112277 pussyripper kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 17 00:25:19.112286 pussyripper kernel: CR2: 000078f9fbdfc2b0 CR3: 0000000250cc4000 CR4: 00000000003506f0

[elFarto]

I'm not sure there's much I can do about a kernel panic. NVD_BACKEND=egl doesn't work, so it's likely just falling back to software decoding.