Implement NixOS container networking with networkd

[arianvp]

We are in the process to moving to networkd based networking for 20.03.

nixos-container module neither uses scripted networking nor systemd-based networking,
but its own set of scripts. We should move nixos-container to use networkd for setting up the network.

Currently if you enable networking.useNetworkd = true; , nixos-container will stop working because the default rules that networkd ships sets up DHCP on the ve-* and vz-* interfaces as it assumes these are created by systemd-nspawn and that's the documented behaviour for the tool (See https://github.com/NixOS/systemd/tree/nixos-v243/network)

the matches look like this:

# 80-container-ve.network
[Match]
Name=ve-*
Driver=veth
[Network]
# Default to using a /24 prefix, giving up to 253 addresses per virtual network.
Address=0.0.0.0/24
LinkLocalAddressing=yes
DHCPServer=yes
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge

# 80-container-vz.network
[Match]
Name=vz-*
Driver=bridge
[Network]
# Default to using a /28 prefix, giving up to 13 addresses per container.
Address=0.0.0.0/28
LinkLocalAddressing=yes
DHCPServer=yes
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge

We should come up with our own systemd.network files that implement all the features that nixos-container supports.

However, I would also like to have vanilla systemd-nspawn work as documented in the systemd manpages.

This means that the .network rules that nixos-container should generate should be:

1. more specific: e.g. Match=ve-mycontainer vs Match=ve-*
2. have a lower number than 80- such that they're executed before the ones shipped with systemd

Implementing this is a blocker for switching to useNetworkd=true by default.

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

cc @fpletz @flokli @andir @Mic92

[arianvp]

I'm not sure why this was labelled kind: bug. could someone remove that for me? :)

[ck3d]

If host and container uses networkd the network is configured correctly by networkd.
The the test nixos/tests/systemd-machinectl.nix tests that configuration.
I think there is no need for a special set of configuration files.

[arianvp]

I'm pretty sure that we need to take a more careful look especially since networkd takes ownership of the veth-pairs nixos-container makes by default. Weird things might happen that we currently don't test for. E.g. DHCP will be on by default and networkd is handing out IPs on the container interface, eventhough nixos-container sets useDHCP = false;

[arianvp]

At best, all the code that calls ip link up and ip link down can just go once we've switched to networkd

[Ma27]

@elseym discussed some ideas in the past and started an initial attempt at NixCon. We'll probably be able to create a proof of concept on the networkd-sprint next weekend :)

[bqv]

Is there a workaround for this?

[Ma27]

In #84608 is an example how to use the nspawn module itself for "containers". You could just configure your networkd inside the container/machine-config.

[Ma27]

FYI I'm actively working on building a useful proposal for networkd-based nspawn-units that can be configured using NixOS. I'll do my best to publish a first draft to discuss during the next weeks :)

[Ma27]

I'm very sorry for the long time this issue was stale. Since I had a brief discussion with
@fpletz last weekend I'd like to share the results with you to gather even further feedback.
At the end of this comment I'll post the next steps I've planned on this.

Note: as this comment became way longer and more detailed than I anticipated, I'm wondering
if the whole process of replacing the current container implementation should go through an RFC.

Summary

• We'd like to get rid of the containers.* module and the nixos-container package in the
  long run, however we know that it's very hard to get rid of it, so we'd wait at least 3
  NixOS releases after those proposed changes landed in a stable release.

  The deprecation is from our PoV necessary for the following reasons:

  • nixos-container is mostly a wrapper around
    machinectl(1), when
    using .nspawn-units for containers, we can use machinectl for most of the tasks

  • nspawn-machines usually expect to get configured using networkd. We work around this
    by configuring the host-side network interfaces after the container has booted in an
    ExecStartPost= hook. Because of that, containers don't have any uplink at boot-time
    which can lead to unbootable containers (because of timed-out services) in the worst case.

• We want to introduce a layer on top of systemd.nspawn (which is a simple module to generate
  nspawn files) that can create a machine from a NixOS config and optionally configures some basic
  networking.

Networking

• Automatically assigned addresses:

  • By default, a container should get a local address from 192.168/16 (as it is currently
    the case in upstream systemd).

  • Additionally an IPv6 ULA-prefix for containers should be generated
    to assign an IPv6 address (or prefix) to a container. This isn't implemented yet in upstream
    systemd which is why I started writing a patch a while ago (which needs a lot of work though
    before getting usable).

  • When using this approach, a container can optionally get internet-access using NAT.

• Static network configuration:

  • Optionally it should be possible to statically declare the container's network configuration
    to e.g. assign public addresses to a container.

  • We want to keep this pretty thin however and make it rather simple for administrators
    to customize the network configuration for their containers.

Additional Features

• nixos-container can be replaced by machinectl and a new tool which generates a rootfs
  and places it into /var/lib/machines using a NixOS configuration for a container.

• Only mount store-paths into the container that are actually needed (I've written a draft for this,
  I mostly have to figure out the performance implications of creating lots of bind-mounts).

• switch-to-configuration should be able to decide whether to restart a container or whether to
  only activate the new configuration when deploying a host.

Migration

As mentioned above, we'd like to replace the current container feature with a very long transition
period.

I'm not sure yet how a migraiton-path should look in detail, but since we're not switchting
technologies, it should be possible to write down a detailed migration recipe (in theory at least).

Roadmap

Please note though that I'll have a few exams at the end of July and in August, so this won't
make it into 20.09 (I'd target 21.03 for now) to get extensive community feedback on this.

I'll try to publish a usable draft until the end of August/September including IPv6 support.
Then we'd have a basic proof-of-concept to work with.

I planned further steps on this in this year's fall.

[arianvp]

This sounds like an excellent plan. I wonder if we should turn it into an RFC to collect feedback and assess fallout?

…

On Mon, Jul 13, 2020, 20:10 Maximilian Bosch ***@***.***> wrote: I'm very sorry for the long time this issue was stale. Since I had a brief discussion with @fpletz <https://github.com/fpletz> last weekend I'd like to share the results with you to gather even further feedback. At the end of this comment I'll post the next steps I've planned on this. *Note:* as this comment became way longer and more detailed than I anticipated, I'm wondering if the whole process of replacing the current container implementation should go through an RFC. Summary - We'd like to get rid of the containers.* module and the nixos-container package in the long run, however we know that it's very hard to get rid of it, so we'd wait *at least* 3 NixOS releases after those proposed changes landed in a stable release. The deprecation is from our PoV necessary for the following reasons: - nixos-container is mostly a wrapper around machinectl(1) <https://www.freedesktop.org/software/systemd/man/machinectl.html>, when using .nspawn-units for containers, we can use machinectl for most of the tasks - nspawn-machines usually expect to get configured using networkd. We work around this by configuring the host-side network interfaces *after* the container has booted in an ExecStartPost= hook. Because of that, containers don't have any uplink at boot-time which can lead to unbootable containers (because of timed-out services) in the worst case. - We want to introduce a layer on top of systemd.nspawn (which is a simple module to generate nspawn files) that can create a machine from a NixOS config and optionally configures some basic networking. Networking - Automatically assigned addresses: - By default, a container should get a local address from 192.168/16 (as it is currently the case in upstream systemd). - Additionally an IPv6 ULA-prefix for containers should be generated to assign an IPv6 address (or prefix) to a container. This isn't implemented yet in upstream systemd which is why I started writing a patch a while ago (which needs a lot of work though before getting usable). - When using this approach, a container can optionally get internet-access using NAT. - Static network configuration: - Optionally it should be possible to statically declare the container's network configuration to e.g. assign public addresses to a container. - We want to keep this pretty thin however and make it rather simple for administrators to customize the network configuration for their containers. Additional Features - nixos-container can be replaced by machinectl and a new tool which generates a rootfs and places it into /var/lib/machines using a NixOS configuration for a container. - Only mount store-paths into the container that are actually needed (I've written a draft for this, I mostly have to figure out the performance implications of creating lots of bind-mounts). - switch-to-configuration should be able to decide whether to restart a container or whether to only activate the new configuration when deploying a host. Migration As mentioned above, we'd like to replace the current container feature with a very long transition period. I'm not sure yet how a migraiton-path should look in detail, but since we're not switchting technologies, it should be possible to write down a detailed migration recipe (in theory at least). Roadmap Please note though that I'll have a few exams at the end of July and in August, so this won't make it into 20.09 (I'd target 21.03 for now) to get extensive community feedback on this. I'll try to publish a usable draft until the end of August/September including IPv6 support. Then we'd have a basic proof-of-concept to work with. I planned further steps on this in this year's fall. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#69414 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZNI2WSTMYXKRDZHVOXZLR3NE3FANCNFSM4I2K5BSQ> .

[bqv]

Thanks, @Ma27 !

[Ma27]

RFC to collect feedback and assess fallout

Yeah, I had a similar feeling when I realized that my comment slightly escalated 😅

[arianvp]

When using this approach, a container can optionally get internet-access using NAT.

Only for the ipv4 part yet? the ipv6 just needs routes.

Note that we can use networkd to configure both

[Ma27]

Only for the ipv4 part yet? the ipv6 just needs routes.

Are you talking about public prefixes or ULA prefixes? Not sure if I'm misunderstanding you, but when using an ULA address for a container, we'd need something like v6 NAT or NPTv6, right?

Btw, I'll try to write a draft for an RFC at one of the next weekends :)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/issues-using-nixos-container-to-set-up-an-etcd-cluster/8438/2

[netvl]

Not sure if this is the correct place to ask, but is this a blocker for using networkd inside the containers? Currently it seems that using networking.useNetworkd = true inside the container configuration:

containers.example = {
  config = { ... }: {
    networking.useNetworkd = true;
  }
}

results in an error:

Failed assertions:
- Using host resolv.conf is not supported with systemd-resolved

which I don't know how to fix.

I need this for correct setup of --network-zone-based containers, where networkd on the host uses 80-container-vz.network, while in the container it uses 80-container-host0.network. I can add these units to the system configuration by linking etc/system/network/80-*.network files to the corresponding files in the systemd package (and it seems to work correctly for the host machine), but when I enable networkd in the container, nixos-rebuild fails with the above error.

Alternatively, I'm okay with setting up containers directly with nspawn systemd units, but I don't have enough knowledge to correctly set up the NixOS configuration -> machine image (or whatever it is called) conversion; is there any example of how to do it somewhere?

---

Edit: ok, I'm not quite sure why, but it seems that contrary to what the options docs say (https://search.nixos.org/options?channel=unstable&show=networking.useHostResolvConf&from=0&size=30&sort=relevance&query=resolv), it seems that this option was true for me by default. Setting it to false explicitly removed this particular error, although it seems there are still other issues I have to resolve...

[Ma27]

A quick&dirty writeup on how to use nspawn units with for nixos-containers can be found in https://gist.github.com/Ma27/ecb59ade6e3241cd393e717ff6a6feb5. Please note that this won't work with nixos-container then.

In fact, I procrastinated this far too long because of other duties, but - interesting coincidence btw :D - I resumed my work last week. The problem is that nixos-containers in the current state don't really work well with networkd, so I'm afraid that it may take a while until I can provide some usable things.

[arianvp]

nspawn's unprivileged mode means that it uses user namespacing to map uid so that root in container != root outside container. However it still requires root to start up containers at this point. It requires root privileges to set up the networking namespace.

[tgunnoe]

alright, thanks. It's not a related issue then.

[Ma27]

OK I figured I could give a small update here:

• the actual container implementation from the branch I linked above works quite fine, I'm using it for my own stuff for quite a while now.
• I'm currently working on a replacement for imperative containers (this time written in Python since I consider it to be more accessible than Perl). It has less features (since most stuff can be done now via machinectl) and also aims to support more container features (such as extra-containers does).
• There's a draft for an RFC lying around locally, but it's still missing a few things and perhaps requires some polishing.

All in all, I'm hoping to actually "release" something in a foreseeable future :)

(and thanks for your patience :p)

[Ma27]

Discussion can be resumed at #140669 (and the corresponding RFC NixOS/rfcs#108).

EDIT: leaving it open on purpose until there's an actual change on master.

[stale]

I marked this as stale due to inactivity. → More info

[Ma27]

Still relevant and on my mind, but fairly busy these days.

[cfouche3005]

Any news on this ? I am trying to use NixOS as lxc container and when I want to build an lxc image using nixos-generator, I get this error :
`
error:
… while calling the 'derivationStrict' builtin

     at /builtin/derivation.nix:9:12: (source not available)

   … while evaluating derivation 'tarball'
     whose name attribute is located at /nix/store/lbqj1cndic4121whnx8xm0jgb1c8x4xx-source/pkgs/stdenv/generic/make-derivation.nix:333:7

   … while evaluating attribute 'closureInfo' of derivation 'tarball'

     at /nix/store/lbqj1cndic4121whnx8xm0jgb1c8x4xx-source/nixos/lib/make-system-tarball.nix:51:3:

       50|
       51|   closureInfo = closureInfo {
         |   ^
       52|     rootPaths = objects;

   (stack trace truncated; use '--show-trace' to show the full trace)

   error:
   Failed assertions:
   - Using host resolv.conf is not supported with systemd-resolved

`

with this network config :

networking = {
    interfaces.eth0.useDHCP = true;
    useNetworkd = true;
    networkmanager.enable = false;
    firewall.enable = false;
  };

I supose my issue is related to this issue

[fpletz]

@cfouche3005 Unfortunately, if you set boot.isContainer, networking.useHostResolvConf is also being set to true. You need to set it to false if you aren't using nixos-container which bind-mounts /etc/resolv.conf from the host. This is not supported by systemd-resolved because resolved manages /etc/resolv.conf. Since you're using DHCP anyway, you don't need to use the nameservers configured on the host anyway.

[giacomocariello]

@Ma27 any news on this?

[Ma27]

I have a few words to say about this and I'll try to do so in the next days.
If I forget, feel free to ping me again.

[Ma27]

So, personally, I won't push the nixos container thing forward. I came to the conclusion that I prefer the approach where you basically nixos-install into a container at /var/lib/machines/foo and write an nspawn unit for that (bootstrapping is a little tedious and something that I'd like to improve eventually).

If people are specifically interested in RFC108-like containers, there are two things in progress:

• https://github.com/fpletz/nixos-nspawn (for declarative containers)
• https://github.com/m1cr0man/python-nixos-nspawn (for imperative containers)

After a few discussions with others I realized that there are a few fixes in #140669 (and my fork that I use for containers as described above) that should be upstreamed into nixpkgs to make sure all implementers of such containers can profit from that. One example is #345899. I need to check if there are more things and after a chat with @ck3d I also plan to consolidate which things from nixos/tests/systemd-machinectl.nix may be good to have in our modules.

The future of RFC108-like implementations is something that @fpletz and @m1cr0man may have an answer for.

I'll remove my assignment here: this is explicitly for nixos-container.

[m1cr0man]

Damn, I was not aware of this ticket until now, nor was I aware of @fpletz work. I have been working hard on python-nixos-nspawn for quite some time, and I have almost completed the declarative container support.

In terms of upstreaming, I have had #216025 open for almost 2 years now, with no progress. I did discover the work on switch-to-configuration-ng recently, and will eventually have to adapt it to support the imperative containers, but that PR I have is still in a good state and I have been running it on my own systems since I implemented it.

@fpletz could we sync up at some point? I will need to read your repo, but it's likely I have duped a ton of work. I ported @Ma27 's old test suite into my repo, and just today got the entire lot passing (commit pending).

python-nixos-nspawn is generally in a good state. I was planning to write docs over the next few weekends then publish something on discourse.

[fpletz]

I haven't really made my work public to a wider audience yet since it is a toy project right now and very much work in progress. The idea was to completely rewrite the container module in order to simplify it with mandatory networkd support and, at least initially, only declarative and ephemeral containers.

@m1cr0man Yeah, happy to sync. I have looked at your work before I started my rewrite and I don't think there is too much duplication actually. I was a bit overwhelmed to review @Ma27's and your changes and sought for a simpler solution that would fit my personal use case. So nothing against both of your work, I just wanted to play a bit more with nspawn without all the abstractions we already have. It is not my intention to force a rewrite but gain experience how to integrate with nspawn/networkd in the most elegant way. If we decide that refactoring the existing NixOS containers is the best way forward, I'm all in of course. 👍

[m1cr0man]

Thanks for the info fpletz. That all sounds great. To date, my direction has been to continue this hard fork of the existing container infrastructure rather than migrate the existing stuff, only because I assumed from the start that there was some exploration made before my time. If it is is a feasible option, I would be keen to explore it also.

Right now though, the fork in my repo is almost feature complete. One thing I could definitely do with help is the network configuration side of things, as my knowledge of all things networking is limited. Between when Max wrote the first POC and now, there have been a number of improvements in systemd itself. Namely, the ipv6 stack got router advertisement and NAT support, and I don't think radvd is necessary any more. Testing this theory was my motive for getting the test suite working again. If you have anything to share from your own research I'd be keen to hear it.

The other benefit to this being a hard fork is the fact that it can (mostly) live in a separate flake/repo, with only the small changes in #216025 required in nixpkgs. This makes it really easy for people to try it out.

One last thing I have been thinking about is the use of imperative containers on non-NixOS systems. I haven't tested it yet, but throughout the implementation there is nothing specific to NixOS there. I think it would be really interesting to support this and offer NixOS Nspawn containers as an option on any systemd platform. There's some really interesting use cases that opens up, but perhaps I'm being too bullish 😉

[Ma27]

Namely, the ipv6 stack got router advertisement and NAT support, and I don't think radvd is necessary any more

Fwiw in my current infra my nspawn stuff (both ends networkd) do use networkd only for router advertisements, so I'm pretty sure this is obsolete.