[BUG] Kernel packages in toolbox not self-consistent, don't match silverblue system

[bnordgren]

This issue tracker is intended only for Silverblue specific issues. We would like to ask you to try to reproduce the issue on a relevant Fedora Workstation release. If you will be able to reproduce there, then please report it in Red Hat Bugzilla or in upstream (preferred for GNOME projects) and not in this issue tracker.

Describe the bug
Compiling kernel modules in a toolbox requires that the kernel packages inside the toolbox be:

1. all the same version
2. the same version as the running kernel outside the toolbox

To Reproduce
Please describe the steps needed to reproduce the bug:

1. You are at the mercy of the packagers.

Expected behavior
Installing the following packages inside a toolbox should result in them all being related to the same version of the kernel. Exactly the same version. Not "close".

Current state of F34 (5.11.17-300.fc34.x86_64) machine:

• kernel-headers-5.11.16-300.fc34.x86_64
• kernel-core-5.11.17-300.fc34.x86_64
• kernel-tools-libs-5.11.16-300.fc34.x86_64
• kernel-tools-5.11.16-300.fc34.x86_64
• kernel-modules-5.11.17-300.fc34.x86_64
• kernel-devel-5.11.17-300.fc34.x86_64

Current state of F33 (5.11.16-200.fc33.x86_64) machine:

• kernel-headers-5.11.16-200.fc33.x86_64
• kernel-core-5.11.17-200.fc33.x86_64
• kernel-tools-libs-5.11.16-200.fc33.x86_64
• kernel-tools-5.11.16-200.fc33.x86_64
• kernel-modules-5.11.17-200.fc33.x86_64
• kernel-devel-5.11.17-200.fc33.x86_64

Additional context
The kernel-core package is actually the one that has /lib/modules/VERSION/build and friends. This one must match exactly. Please note at this time for the F33 machine, kernel-core is more up to date in the toolbox than the running kernel is, and there is no ability to downgrade to the running version.

⬢[bnordgren@toolbox bnordgren]$ dnf --showduplicates list kernel-core
Installed Packages
kernel-core.x86_64                                                5.11.17-200.fc33                                                @updates
Available Packages
kernel-core.x86_64                                                5.8.15-301.fc33                                                 fedora  
kernel-core.x86_64                                                5.11.17-200.fc33                                                updates 

If the toolbox is the go-to solution for compiling kernel modules, there must be better QA and better coordination between the Silverblue and toolbox packagers.

[tpopela]

If the toolbox is the go-to solution for compiling kernel modules, there must be better QA and better coordination between the Silverblue and toolbox packagers.

I don't think it is. Can you please let me know where it is mentioned (so we can edit it). It might be in the future - see openshift/enhancements#357 , but we are not there yet. You might also find the discussion in https://pagure.io/workstation-ostree-config/pull-request/214 interesting.

It's not expected that you will have the same version of packages between your i.e. Silverblue 34 system and your Fedora Workstation 34 container at the given point in time. In Workstation the packages are pushed to the stable repositories once they pass the bodhi process. For Silverblue the compose needs to happen and as it's happening once a day you would see a difference when the Silverblue users will get the package and when the Workstation users will.

Also this isn't something that Toolbox developers will have to resolve. I think that it's currently expected from the user to install the kernel development files manually for the currently running SiIverblue system (until we have something that's supported and has a better user experience).

Maybe @cgwalters or @dustymabe or @otaylor might add more.

[owtaylor]

The idea of matching the kernel packages to the host is, I think, rather conceptually difficult:

• If the host is Fedora 34, the user might be using toolboxes for Fedora 33 or Fedora 35, or an entirely different operating system
• You don't generally need kernel or kernel-devel in your toolbox, and these are big, frequently changing packages
• The user might want to compile a kernel module for the rpm-ostree version that they they are upgrading to, not the current one

My thought is that if we want to support kernel module compilation in a container, it should be a much automated process, probably not involving toolbox at all. Conceptually, some sort of:

 silverblue-compile-module --release=f34 --kernel-version=5.12.2-300.fc34 <modulel-sources>

I think @dustymabe 's work that @tpopela pointed to is headed in that direction, but I haven't tracked down the details of what is the current plan there.

That all being said, the recently added archive updates repository is probably useful for someone trying to compile a module for the host inside the toolbox, and perhaps it would be useful somewhere to have instructions somewhere for doing that.

[bnordgren]

As far as I know, this open ticket is the current advice for the v4l2loopback kernel module: coreos/rpm-ostree#1091 (comment)

Note that v4l2loopback is a key contributor to videoconferencing capabilities, and so it's taking on elevated importance currently.

[travier]

If you have a custom kernel module, your current best option is to package it as an akmod RPM like it's done for the NVIDIA one.

[travier]

Closing as toolbx packages are independent from the host by design.

[bnordgren]

I suppose lest anyone get confused, "completed" is an optimistic state for this ticket. The crux of the problem is being tracked in #1091, which was opened in 2017 and shows no signs of nearing any kind of resolution in 2022.

Using a toolbox is the only method (shy of installing compilers and development libraries on the host) to get unpackaged kernel modules built and installed. 1.25 years after opening this ticket, the coordination between kernel packages and toolbox packages is the first thing I check on any upgrade. If they don't match, it's time to rollback. Try again some other day.

I'm not fluent in akmods as mentioned above, but as it appears that the kernel modules are automatically compiled on the host (every boot?), you'd have to install compilers, devel libs, and other dependencies directly on the host in order for that solution to be viable. Even if it used a container for the compilation, the same problem of coordinating container packages with host packages remains. The only novel thing about akmod is the automatic detection of the need for a recompile. I know when I need to recompile: right after I tell the system to upgrade. If the official solution is to install all of this stuff on the host, then much that is distinctive about Silverblue (and any other rpm-ostree distribution) is eliminated. Likewise, if using the host directly instead of using a toolbox is the solution, I don't need akmod. "Make clean && make" should do it.

Is it at all possible to have a less frequently updating fedora silverblue stream which archives the package repositories from which the base image was built, then point the toolbox at that? Along with some means to point the toolbox at a "newer" repo of course...Or even just freeze the repos from which the nightly images are made for a day, so that those who do a back to back "rpm-ostree upgrade" followed by "toolbox enter \n sudo dnf upgrade" (or "toolbox create") will get a consistent package set deployed to their machine? You don't have to archive them forever, you just have to freeze them for as long as rpm-ostree upgrade installs the image derived from them. While this doesn't allow you to smoothly roll back to a consistent package set both inside and outside the container, going forward it at least provides a reliable means for you to work with what is actually installed on the host.

Right now upgrading is kind of a crap shoot.

[travier]

I'm closing this issue because I don't think there is anything to be done here.

If you want your kernel package version in toolbx to match the one on the host then you can add the archive repo in your toolbx and install the matching package by specifying the version explicitly.

Outside of toolbx, our current best option is to use akmods which build kernel modules during upgrade (and not on boot) on rpm-ostree systems. This installs the compiler and tools on the host and this how we support the NVIDIA driver right now. We use the kernel-devel-matched meta package to ensure versions are in sync.

[bnordgren]

Is there any trick to forcing akmods to compile? After you said this, I tried installing the nvidia akmod on my clean F36 Silverblue install, but it's just not even trying to build the kernel module at all. Also, I can't seem to find any information about the akmod packaging system (landing page, maintainers, issue tracker, forum, documentation...) I see oblique references to it on mailing lists back to 2009, but no indications as to where to ask questions about it. I've packaged things for rpm and pacman in the (distant) past, and there's oodles of resources. This is somewhat different...

Can you point me at the project page/issue tracker/documentation for this preferred system?

[travier]

Some links:

• https://rpmfusion.org/Packaging/KernelModules/Akmods
• https://rpmfusion.org/Packaging/KernelModules/Kmods2
• How this interacts on Silverblue: https://src.fedoraproject.org/rpms/akmods/pull-request/3#comment-90939

[travier]

Akmods package in Fedora: https://src.fedoraproject.org/rpms/akmods