[New to bootc] bootc vs PXE stateless image

[th-hummel]

Hello,

I've read bootc documentation and still have some basic questions in the following context :

I'm running an HPC cluster which nodes PXE boot a so-called "stateless" os image which is created, provisionned and packed on a management server using a dedicated provisionning software (xCAT or Warewulf). So everyhing is in RAM, nothing on disk.

My thinking revolves around if bootc could help me or not handle image versionning and what makes it different in my case than what I'm already doing

Today, the so called image is just a packed chroot (plus an initialramfs (xCAT), or as a big initialramfs (Warewulf)).
This chroot can be the import from the merged rootfs of an OCI image. So I also played with the concept of a "bootable container". Basically it's a podman/buildah created image derived from Redhat UBI inside which necessary packages (including kernel) has been installed and necessary systemd units has been unmasked/enabled.
Bootloader part is handled by iPXE scripts provided by xCAT or Warewulf

So an update in my case de facto means rebooting on a new image (which still have to be rebuild though)

My understanding of bootc is that :

1. the bootable image is (usually) derived from an already bootable base image. One just add desired packages
2. it is aimed at resolving the upgrade problem and does so by staging image n+1 and rebooting (with potential rollback, the whole process being atomic)
3. /etc + /var are excluded from this image and considered persistent

What I still have to figure out is :

1. what program should we use to actually build the bootc OCI image ? Our usual tool (podman, buildah, docker ,...) ?
2. how do we build image version n+1 ? is that the goal of bootc-edit ? does the image has to change name or tag ?
   I mean if, as podman bootc extension states, no rebuild is needed (" Once a machine is created from the disk image, it can apply transactional updates "in place" from newly pushed container images (without creating a new disk image)." the container in which we make n+1 diffs must be somehow commit-ed into an image
3. how does bootc on host knows there is an update available : it noticed a new layer exist ?
4. do you confim than "in place" still needs a reboot in any case, hence, in my fully stateless case, `/etc/' and '/var' state would be lost anyway as everything is in RAM ?
5. is bootc supported by RHEL ?

Thanks for your help

--
Thomas HUMMEL

[miabbott]

I'll make an attempt at addressing your questions:

what program should we use to actually build the bootc OCI image ? Our usual tool (podman, buildah, docker ,...) ?

The container images that bootc interacts with are just OCI images: any existing container tooling that can build OCI images can build a container image that bootc can interact with.

how do we build image version n+1 ? is that the goal of bootc-edit ? does the image has to change name or tag ?

Building the next version of an image is just another container build process. How you tag/version the container image is up to the user. For example, it may be desirable to always have a :latest tag, but also maintain individual versions like :1.0.0, :1.0.1, :1.2.0, etc.

The note about no rebuild needed is specific to the disk image that is created from the container image. The disk image is something like a .qcow2, .raw, .iso, etc. The disk image is intended to facilitate the initial install of the OS to a system; once the install is completed, the system can be updated by pulling an newer container image.

how does bootc on host knows there is an update available : it noticed a new layer exist ?

In a very basic configuration, the system would have an automatic update timer/service that would reach out to the container registry for the tag the system is following (i.e. :latest) and be able to detect that there is a new version of the image available. This would kick off the bootc upgrade process in the background, which would fetch the new image, and stage it in the background. When the system reboots, the staged update will be used as the new version of the OS that the system uses. See https://containers.github.io/bootc/upgrades.html#the-bootc-upgrade-verb

It is possible to devise more sophisticated upgrade logic/detection, but that is out of scope for bootc.

do you confim than "in place" still needs a reboot in any case, hence, in my fully stateless case, `/etc/' and '/var' state would be lost anyway as everything is in RAM ?

bootc is backed by ostree which requires that the system is rebooted for a new version of the OS to be used. Currently, there is no way to avoid the reboot to use the newer version of the OS/container image.

is bootc supported by RHEL ?

bootc is considered Tech Preview in RHEL 9.4

[th-hummel]

Hello,

thanks for your answers, which confirms that my use case is a different one. Still I'm interested.

So now my understanding is that the "magic" of bootc is not the fact of being able to build a bootable image (I mean build an OCI image with sufficient content) as I initially thought but rather the image-based handling of:

a) the bootloader necessary work to boot on the correct tree
b) the upgrade mechanism
c) the possibility (with side tools ?) to install the container on a some disk media to make it bootable and bootstrap the install/upgrade process

Besides, it seems bootc is meant for upgrading stateful (~on disk) installs.

Am I getting this right ?

I'm not sure though I understand what 'backed by ostree' means and why this implies a reboot. Does it means that once the new image is staged, it more or less like an ostree fetch/update ?
Does it also mean that some day it may be independent or less dependent from ostree and one could upgrade the image without reboot (if components like kernel or glibc are not invovled) ?

Thanks for your help

--
Thomas HUMMEL

[miabbott]

Besides, it seems bootc is meant for upgrading stateful (~on disk) installs.

That is the "expected" or "traditional" use case that I think will cover most users, but I believe it would still be possible to PXE boot a disk image that was created from a bootc container image for stateless use cases. I think the additional step in the stateless case is the need to generate new disk artifacts for each new container image, so that the PXE client is booting the latest version of the OS.

I'm not sure though I understand what 'backed by ostree' means and why this implies a reboot. Does it means that once the new image is staged, it more or less like an ostree fetch/update ?

This is an implementation detail of bootc and ostree. See this docs update #541 and the associated docs that it changed for more details.

Does it also mean that some day it may be independent or less dependent from ostree and one could upgrade the image without reboot (if components like kernel or glibc are not invovled) ?

It may be possible to do a "live-apply" of the changes from a newer container image in the future. rpm-ostree currently has this capability - https://coreos.github.io/rpm-ostree/apply-live/

As you noted, reboots are required for new versions of the kernel.

[th-hummel]

Ok, thanks for those answers.

--
Thomas HUMMEL

[cgwalters]

Longer term we as part of the Fedora/CentOS bootc project I think we will enable a flow with bootc-image-builder related to this, I just filed osbuild/bootc-image-builder#433

Basically once we start supporting generating an iso of that type, it's just one step further to unpack the ISO into a PXE setup with split kernel/initrd + rootfs.

Which I think would be what you want, and the advantage of this over what you're doing is that you'd at least get the bootc immutability at runtime and having bootc status e.g. show you the booted container image state.

[th-hummel]

Hello, thanks for your comment.

Well that's what I try to wrap my head around (and I'm still confused at this stage as bootc is new to me) : it seems the intersection between what bootc does and what I'm doing is not quite empty.

Actually, using containers, which is new to us also in this context (as we used only chroots) opens new horizons as the "image" may be provisionned (let's say with ansible) like if it was a VM by running the container, which then can be commited to a (bootable) image. This is more elegant and less tricky than chrooting by hand. Old behavior is still possible by mouning image rootfs.

I'm not sure about what you are refering to with bootc immutability : I mean if you use some /usr overlay to let the possibility of live patching your image (which is needed for us), you don't really care if the image is rw has after a reboot by construction everyhing is reset in a stateless node case ?

Or to put it in another way : in a fully stateless (pxe -> os in ram) case, nothing is staged and the "upgrade" mechanism is inexistent : whenever you boot, you pull the latest image.

Do you mean that bootc image is ro because it inherits ostree behavior where (because of checkout as hardlinks as I understand) /usr and /boot are ro and /etc /var are statefull (survive reboots) and some files may have to be symlinked inside those ?

Thanks for your help

--
Thomas HUMMEL