prepare-var: Do not remount stateroot var in-place, and unmount the temporary var mount after real /var is mounted

[ruihe774]

This is split out of some merge-ready commits in #3358.

Currently, in ostree-prepare-root, we create a read-writable bind-mount of /sysroot/ostree/deploy/$stateroot/var in-place, and in a unit generated by ostree-system-generator, it is then bind-mounted to /var. This approach causes some problem:

• We have to make the mount slave+shared to prevent sub-mounts from being propagated into /sysroot/ostree/deploy/$stateroot/var.
• We need to write some bootloader-specific code in ostree-system-generator to get the path to stateroot.
• /sysroot/ostree/deploy/$stateroot/var is not unmounted afterwards, polluting the global mount namespace.

This PR tries to solving these problem by instead of bind-mounting var in-place, bind-monting it to /run/ostree/.private/var in ostree-prepare-root. In this way, ostree-system-generator can bind-mount a fixed path to /var, and the mount is not necessary to be made slave+shared. Also, ostree-system-generator creates a service ostree-unmount-temp-var.service that unmounts the temporary mount of var after real /var is mounted.

[openshift-ci]

Hi @ruihe774. Thanks for your PR.

I'm waiting for a ostreedev member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[cgwalters]

/ok-to-test

[cgwalters]

Thanks for splitting out preparatory patches like this, it's really helpful for review.

At a very high level what you're saying sounds plausible, but there are quite an array of details here.

In the immediate term it looks like this breaks a whole lot of the CoreOS tests (the logs are currently annoying to get at - go to the jenkins instance and download kola.tar.xz from artifacts, then get console.txt).

Anyways the main failure is:

[ 10.003359] ignition-ostree-populate-var[1195]: mkdir: cannot create directory '/sysroot/var/lib': Read-only file system

The ostree initramfs handling is already quite complex on its own, but Ignition is also a huge amount of complexity in the initramfs too, and it's not surprising that a change like this would break the combination of the two.
The relevant code is around https://github.com/coreos/fedora-coreos-config/blob/testing-devel/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-mount-var.sh - presumably it needs some adaption?

[cgwalters]

Marking as requested-changes because of the CI failure at least and to clear the immediate needs-review flag.

[cgwalters]

/ok-to-test
Label race condition