Userspace and kernel version mix+matching

[orowith2os]

Would this ever be possible with the Nvidia drivers? It's a major PITA to use them with Flatpak or any form of sandboxing, since the userspace components need to perfectly match the kernel modules.

Why can't these drivers (or even the old ones) allow userspace mismatching and still function? Mesa can do it just fine.

[aaronp24]

This can be explored, but please note that it comes with a significant engineering cost (not just for us; as far as I understand, Mesa expends significant time and energy on maintaining legacy interfaces to make that work).

There is already a set of stable driver ABIs defined by specifications such as GLX, OpenGL, Vulkan, X11, etc. In my personal opinion, mixing and matching internal driver components is slicing the software stack in the wrong place.

[mtijanic]

Agreed. CUDA has limited support for kernel driver version mismatch and at times it has been quite a pain when developing. We do have some vague plans to support stable/versioned API/ABI to usermode, similar to the ones towards GSP as explained in #372 (comment) but this is not something that is actively being worked on currently.

By the way, in addition to standardized specs like OGL/Vulkan/etc we also have libraries like NVML and NVAPI whose main purpose is to provide an API/ABI-stable way to communicate with the kernel driver. Bit more info at: #157 (comment)

I know none of these really help if you want to sandbox the UMD, and we can look into ways to make that flow easier, but I really doubt we could commit to maintaining ABI stability at this level.

[orowith2os]

Agreed. CUDA has limited support for kernel driver version mismatch and at times it has been quite a pain when developing. We do have some vague plans to support stable/versioned API/ABI to usermode, similar to the ones towards GSP as explained in #372 (comment) but this is not something that is actively being worked on currently.
...
I know none of these really help if you want to sandbox the UMD, and we can look into ways to make that flow easier, but I really doubt we could commit to maintaining ABI stability at this level.

Would it make it easier if the mismatching were to only work with Vulkan, OpenGL, and whatever else a sandbox on the desktop would usually need? This way, you wouldn't need to maintain CUDA backwards-compatibility, just leave that to external tools, like Flatpak;

Flatpak already has a utility in place for updating the flatpak version of the userspace components, but it needs constant maintenance and requires a recheck on every single update.

Additionally, separating the components of the driver may help: separate more dev-oriented things into a nvidia-cuda-whatever repo/package, and normal userspace components like Vulkan into a nvidia repo/package. This way it can maintain some mismatching where easiest, while also allowing CUDA to work in a sandbox with other mismatched userspace components

[mtijanic]

I can not recommend this for anything even resembling production use, but if you're just playing around, it's worth knowing about the __RM_NO_VERSION_CHECK env variable (can be set per-process).
The valid values, from nv-ioctl.h:

#define NV_RM_API_VERSION_CMD_STRICT         0 // default, exact match
#define NV_RM_API_VERSION_CMD_RELAXED       '1' // only require major (i.e. 515.xx) version match
#define NV_RM_API_VERSION_CMD_OVERRIDE      '2' // always match even across major versions (i.e. 470.xx / 515.xx)

The code to evaluate this is at https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/src/nvidia/arch/nvalloc/unix/src/osapi.c#L997-L1099

Keeping in mind that there is a good reason why the default is strict, and that things can and do break when you mix-and-match, this knob can still be very useful when testing out smaller applications and tools.
Usually, the ABI breakages are in the form of a struct field being added/removed, and since all structures are sanity checked by their size (check user passed the exact size expected by kernel) typically this means that a call will fail early with a reasonable error code.