Optional systemd-nspawn support

[antonlacon]

This adds systemd-nspawn and machined to the image, locked behind the NSPAWN_SUPPORT distribution option. It's not enabled by default.

Systemd-nspawn is systemd's chroot, container, and virtual machine manager. This has enough for the first two; VM support is unknown and not my interest. It can run whole OSes, or single applications. I've only tested running an OS. My understanding is that it can run images made by docker (via --oci-bundle=), but I haven't tried it.

/var/lib/machines is the recommended location containers are expected. This has a helper service to get that populated based on the contents of /storage/containers.

systemd-nspawn, machinectl, and systemd-machined combined are somewhere around 600KB uncompressed.

If you're going to try it, and want containers to start on boot, machines.target needs to be enabled first (the machine too).

If using a full OS, machinectl's login defaults the terminal to vt220, so no color on the console. Add an environment override in the container to set TERM to xterm to get color back. Alternatively, use machinectl shell or systemd-nspawn directly to avoid this.

[vpeter4]

Can you give a link to some very simple quick tutorial with using this functionality?

[antonlacon]

I'm not aware of a single good howto for using systemd-nspawn. It's mostly blogs about how the author solved a specific issue. My usage is from reading the manpage (https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html) and Arch's wiki for troubleshooting using as a service (https://wiki.archlinux.org/title/systemd-nspawn).

The overview for getting an OS container to start is:

Have a directory with the OS's filesystem in it
chroot (systemd-nspawn -D /path/to/container) into it to set root passwd, needed compatibility steps, or other OS set up
Boot the container to start its init and bring up its services (systemd-nspawn -b -D /path/to/container)

(systemd-nspawn can also work with raw disk images and check if they have a correct cryptographic signature but I haven't tried either.)

Here's an example of how to get Alpine Linux up and running:

Place to hold the container:

cd /storage/containers
mkdir alpine
cd alpine

Download the mini root filesystem from alpinelinux.org (this is aarch64):

wget https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/aarch64/alpine-minirootfs-3.19.1-aarch64.tar.gz
tar xvf alpine-minirootfs-3.19.1-aarch64.tar.gz
rm alpine-minirootfs-3.19.1-aarch64.tar.gz

Prepare the container for booting:

cd ..
systemd-nspawn -D /storage/containers/alpine

Set root's passwd
passwd

Update the package repo:
apk update

Install alpine's meta package for minimal install:
apk add alpine-base

Add nano (unless you like vi):
apk add nano

Configure timezone (optional)

apk add tzdata
ln -s /usr/share/zoneinfo/WANTED/TIMEZONE /etc/localtime

Configure login consoles to disable tty and add console (the commented tty and console line below):
nano -Ecw /etc/inittab

#/etc/inittab

::sysinit:/sbin/openrc sysinit
::sysinit:/sbin/openrc boot
::wait:/sbin/openrc default

# Set up a couple of getty's
#tty1::respawn:/sbin/getty 38400 tty1
#tty2::respawn:/sbin/getty 38400 tty2
#tty3::respawn:/sbin/getty 38400 tty3
#tty4::respawn:/sbin/getty 38400 tty4
#tty5::respawn:/sbin/getty 38400 tty5
#tty6::respawn:/sbin/getty 38400 tty6

console::respawn:/sbin/getty 38400 console

# Put a getty on the serial port
#ttyS0::respawn:/sbin/getty -L ttyS0 115200 vt100

# Stuff to do for the 3-finger salute
::ctrlaltdel:/sbin/reboot

# Stuff to do before rebooting
::shutdown:/sbin/openrc shutdown

Permit root logins (the pts/# lines below):
nano -Ecw /etc/securetty

console
tty0
tty1
tty2
tty3
tty4
tty5
tty6
tty7
tty8
tty9
tty10
tty11
hvc0
ttyS0
ttyS1
ttyS2
ttyGS0
ttyAMA0
ttyAMA1
ttyTCU0
ttyTHS0
ttyTHS1
ttymxc0
ttymxc2
pts/0
pts/1
pts/2

Enable boot services:

rc-update add hostname boot
rc-update add bootmisc boot
rc-update add syslog boot

Enable shutdown services

rc-update add killprocs shutdown
rc-update add savecache shutdown

Exit and boot the container

exit
systemd-nspawn -bD /storage/containers/alpine

If everything went to plan, you should have a booted Alpine Linux install to add more packages / services / etc. It'll stay booted until you shutdown (halt in Alpine), or close your shell/connection. If you need to force kill it, hit ctrl + ] three times in quick succession. By default when launched with systemd-nspawn, it'll share the network with the host OS, so be mindful of sharing ports.

To have it keep running, you'll want it to be a service, which is where the symlinks to /var/lib/machines come in. (Symlinks need to be full paths if you're not using the helper service.) Something like machinectl start alpine will start the container as a service, but it'll be a private network (I don't know the details for configuring it - I put in a service override for mine to share the network with the host). To have machinectl login $container or machinectl shell $container work, I believe it needs the container to be running systemd as init too, but I may have just not configured something correctly for Alpine. To login with it as a service, I'd set it up with ssh on a different port. Alternatively, shutdown the service machinectl stop $container and launch it with systemd-nspawn to get shell access.

I'm using Gentoo with systemd on LE which does have machinectl working as expected to interact with it. Installing it was less work than the above (or I'm just more used to it) - unpack the stage3 tarball and do the system configuration steps form the Gentoo handbook (no kernel config, no hard drive partitioning, etc).

The Arch wiki linked above has instructions on creating the filesystem for Arch, Debian/Ubuntu and Fedora/Alma (these assume the use of Arch / full distro).

There are scripts here https://gist.github.com/sfan5/52aa53f5dca06ac3af30455b203d3404 that are meant to be run on another system to prepare an alpine, arch or ubuntu filesystem directory. I've referred to it for image setup steps before, but haven't actually run the scripts before.

I think debian could be set up from their cloud images: https://cloud.debian.org/images/cloud/. The genericcloud .tar.xz files. I might try that one later. They're disk images while I'm looking for a rootfs tarball.

[antonlacon]

Gentoo's wiki on systemd-nspawn is pointing to https://nspawn.org/images/ as premade images suitable for using with systemd-nspawn.

[vpeter4]

Sounds useful to me. Not sure yet how to place it along docker.

[antonlacon]

Additional notes/observations:

Alpine wasn't a great base as it's not using systemd, which means machinectl couldn't control it from LE. It was useful for building a debian container, however. (Fun fact: the RPi5 needs about 4 hrs to build an RPi5 image on an sd card; measured before recent speedups.)

The containers can have certain configuration settings that would be used on the CLI be instead stored in an *.nspawn file for use by machinectl, where * is the name of the container. *.nspawn files are stored alongside the container ("untrusted") or in /etc/systemd/nspawn ("trusted"). Certain settings are only available from trusted *.nspawn files. This affects using bind mounts, getting shared networking, and running privileged containers. (pretty much every option in https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html that mentions "privilege".)

By default, when launching with machinectl, containers have private networking. In our use, this means no networking. With private networking disabled, containers share the network interface with the host. If a privileged port is desired (rsync in my case), private users must be disabled too (this results in a privileged container).

TODO setup symlink from /storage/.config/nspawn to /etc/systemd/nspawn for privileged configuration options.

[antonlacon]

Another note is being able to run programs within containers with systemd-run. It can be used like systemd-run --machine=$CONTAINER --pipe $MY_COMMAND. This executes the desired command inside the container without needing to login to it. Commands are executed as the user to run systemd-run unless instructed otherwise (in our case root). It needs to use paths as the container understands them, so if one wanted to use a container's ffmpeg, for example, the media would need to be bind mounted into the container (or otherwise copied into it).