libkmod: tools: Introduce 'module fallback' directory

[vittyvk]

This is more of an RFC but the code compiles and seems to work; testsuite is also adjusted to test new functionality.

Currently, traditional Linux systems using initramfs are forced to keep two copies of the modules which may be needed both during early boot and for normal system operation. This seems to be inevitable for the case when initramfs is built locally but with the rise of distro shipped UKIs the question whether it would be possible to avoid the redundancy arises.

To make it possible to reuse modules from initramfs without copying them to the root filesystem (which may be read-only), the suggested approach is to preserve them on tmpfs, e.g. in /run/modules/uname -r but the challenge is to make kmod search there. It would certainly be possible to e.g. bind mount /run/modules/uname -r to /lib/modules/uname -r but this will make the destination read-only and traditional tools which are used to install packages with modules (e.g. rpm) will break. Instead, a 'fallback' directory approach is suggested: in case main /lib/modules/uname -r is completely missing, kmod tools will check whether /run/modules/uname -r exist and search there.

In theory, it would've been possible to replace this fallback option with a dedicated config for kmod making modules location dynamic but this seems to be a more intrusive change and it will also require additional external actions to switch from 'fallback' to 'main' if/when modules get installed to the main system.

Note, no merging of /run/modules/uname -r and /lib/modules/uname -r is performed. It is expected, that in case /lib/modules/uname -r exists, it is a superset and modules from initramfs are no longer needed.

To make a real world example of the usage, Fedora/CentOS/RHEL distros now ship 'kernel-uki-virt' package with a UKI which is self sufficient in many cases. The package, however, has to have a dependency on 'kernel-modules-core' so basically the same set of modules is available post boot. This is needed when e.g. a filesystem module which was not used for root filesystem need to be loaded to mount some other volume (and ESP is the most common example). The desire is to be able to break this dependency for 'minimal' cloud/VM images but still support the case when people want to install ''kernel-modules-core'/'kernel-modules'/'kernel-modules-extra' post boot and enjoy additional modules.

[vittyvk]

I've also opened https://github.com/orgs/uapi-group/discussions/124 to get additional opinions (and as /run/modules/uname -r path is to become de-facto standard if this PR gets merged).

[lucasdemarchi]

so in the case kernel-modules-core is installed there's still a duplication and this all only works for a minimal distro-shipped initramfs. Why weren't these modules built-in? If modules were split in a way that you don't ever have the duplication, then I think it would be more acceptable.

If you can ship with pre-computed depmod, maybe you want to look at depmod.d(5), for the external command?

[vittyvk]

The distro-shipped initramfs doesn't, in fact, need to be so minimal. E.g. Fedora UKI already has:

• nvme/xen/storvsc storage modules to serve varios clouds
• xfs, ext4, vfat, btrfs, overlayfs filesystems
• virtio,pci,pci-hyperv,... service modules
• dm_crypt, dm_verity,...
  This should be enough to boot the UKI in various virt/cloud environments and in many cases users don't need anything extra on top of that. For a custom built kernel, it is probably possible to compile them all in but a distro doesn't want to have multiple kernel builds (i.e. with different .config) to support, it is simply too hard to support. UKIs are basically built of the same bits as the 'traditional' kernel, the only difference is that the initramfs is built during kernel build time and not on the target system.
  With external, I don't quite see the workflow here, especially if we want to support both 'no modules on rootfs' and 'has full set of modules on initramfs' scenarios but maybe I'm missing something. Like where 'modules.*' files come from and what happens when we install something on the rootfs?

[vittyvk]

Splitting modules in sub-packages so there's no duplication is definitely possible. In that case we will need to support merging 'modules.*' information for what's in the UKI (/run/modules/uname -r) and what's in the main system (/lib/modules/uname -r). I guess this is possible but is likely a more intrusive change. If such approach seems preferable, I can certainly look into it.

[evelikov]

If I remember correctly there was some discussion with kernel devs to actually (allow) unmount the initrd from the host... Not 100% if I am not butchering it though.

I'm expecting people to weigh-in on the discussion post linked.

If we go with this I think we want more elaborate a bit more on the "how" and have those details included in the man page(s). Namely:

• can/should we mix and match from the two locations, what is our heuristics
• should we alert the user (package builder, libkmod, modinfo, modprobe) that the fallback is available and/or when it's used
• should it really be a fallback, or the first option searched - /run/foo usually takes higher precedence than /lib/foo

[vittyvk]

@evelikov all good questions, thanks! My take

• If we decide to mix, then we need to define what takes precedence if the same module is available in both locations. I don't think it matters for 'normal' scenarios as modules for the same uname -r are supposed to be identical in both but we need this at least for completeness.
• I don't see how to alert users reliably to be honest. As the code is implemented right now, it prevents an imminent failure to locate a module so it is really a 'fallback'.
• In this particular case /run is normally 'thinner' than /lib as the former comes from initramfs and the later can have all sorts of additional modules installed (e.g. third-party modules).

[evelikov]

@vittyvk thanks. I realised that my questions might have been better suited in the discussion thread, so I tweaked and posted them there as well.

[lucasdemarchi]

• should it really be a fallback, or the first option searched - /run/foo usually takes higher precedence than /lib/foo

that's because they are not the same kind of /run. Once you have something in /lib/modules/$(uname -r), whatever you had in initramfs when you booted is outdated. From a kernel developer pov, it's often the case I'm replacing the module for tests and I'd be surprised if kmod decided to pick the old, outdated module from another location.

Even from the distro pov: if you install a package that brings one additional (external) module, that had a previous version in the initramfs.... it would be very error prone to let /run take precedence.

• If we decide to mix, then we need to define what takes precedence if the same module is available in both locations. I don't think it matters for 'normal' scenarios as modules for the same uname -r are supposed to be identical in both but we need this at least for completeness.

I think it does matter, as per above

[vittyvk]

Even from the distro pov: if you install a package that brings one additional (external) module, that had a previous version in the initramfs.... it would be very error prone to let /run take precedence.

Yes, I agree /run should be the fallback, not the default (and that's how the code in the PR works).

• If we decide to mix, then we need to define what takes precedence if the same module is available in both locations. I don't think it matters for 'normal' scenarios as modules for the same uname -r are supposed to be identical in both but we need this at least for completeness.

I think it does matter, as per above

Ok, thanks, I completely forgot about such 'developer' case but this (again) points us toward making /lib higher priority location than /run even if we decide to 'mix'. If the module is missing in /lib then fine, let's look in /run but if it's there than /run is irrelevant.