systemd-resolved resets /etc/resolv.conf and vpn-based name resolution breaks

[uishon]

Running openfortivpn. It starts smoothly - /etc/resolv.conf seems to be updated correctly and vpn-based name resolution works, But, after a couple of minutes, changes are reset and name resolution stops.

Tried to play with --no-dns and --pppd-use-peerdns options but nothing helped.

Happened to me with openfortivpn 1.6 on Ubuntu 18.04.,

Was hoping this will be resolved with openfortivpn 1.12 on Ubuntu 20.04 but it's not. Same behavior.

Seems to be systemd-resolved related as I have another Ubuntu 18.04 machine which is not running systemd-resolved and there openfortivpn based name resolution works continuously.

Any idea?

[DimitriPapadopoulos]

Can you confirm it happens both when openfortivpn is responsible for updating DNS parameters (--set-dns=1 --pppd-use-peerdns=0) and when this is handed over to pppd (--set-dns=0 --pppd-use-peerdns=1)?

I'm asking because while I know openfortivpn may require some fixes, I would expect pppd to do the right thing.

Indeed I agree with you that either systemd-resolvd or NetworkManager are overwriting the changes applied to /etc/resolv.conf by openfortivpn (and perhaps pppd too).

One possible workaround would be to let NetworkManager manage everything by using the NetworkManager_fortisslvpn GUI, which calls openfortivpn under the hood with options --set-dns=0 --pppd-use-peerdns=0 and takes over routing and DNS servers.

The legacy method for changing DNS servers has indeed been to modify /etc/resolv.conf directly. This used to work - not that well but there weren't alternatives.

Then some subsystems started taking over /etc/resolv.conf, such as NetworkManager or systemd. Unfortunately because of the variety of these subsystems it hasn't got easier to handle DNS parameters in an portable way - on the contrary. A recent attempt to fix that was to add resolvconf support, which precisely promises to offer a standard interface to set DNS parameters. Unfortunately resolvconf is not always available (it actually tends to disappear for example on Ubuntu 20.04) and when it is available functionality varies wildly depending on the resolvconf executable:

• the resolvconf from resolvconf does seem to be working,
• the resolvconf from openresolv does seem to be working,
• the resolvconf from systemd is hopelessly broken and breaks the promise of a consistent way to change DNS parameters whatever the backend.

I need to know if resolvconf is available on your system and if so which version of it. Can you please run these commands?

which resolvconf
dpkg -l | fgrep resolv
ls -l /etc/resolv.conf

In the long term we should probably support systemd-resolvd directly when available. That's on the TODO list (#600).

[uishon]

Just double checked. It happens both with --set-dns=1 --pppd-use-peerdns=0 and --set-dns=0 --pppd-use-peerdns=1.

As for resolvconf, it's not there - neither on my Ubuntu 18.04 nor on Ubuntu 20.04.

Below is the command output you've asked for.

Ubuntu 18.04

uri@neptune:~$ which resolvconf
uri@neptune:~$ dpkg -l | fgrep resolv
ii  libgeoip1:amd64                         1.6.12-1                                        amd64        non-DNS IP-to-country resolver library
ii  python3-json-pointer                    1.10-1                                          all          resolve JSON pointers - Python 3.x
uri@neptune:~$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Feb 14  2019 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
uri@neptune:~$

Ubuntu 20.04

ubuntu@uranus:~$ which resolvconf
ubuntu@uranus:~$ dpkg -l | fgrep resolv
ii  libc-ares2:amd64                        1.15.0-1build1                        amd64        asynchronous name resolver
ii  python3-json-pointer                    2.0-0ubuntu1                          all          resolve JSON pointers - Python 3.x
ubuntu@uranus:~$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Apr 23 09:40 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
ubuntu@uranus:~$

Ubuntu 18.04 (where openfortivpn name resolution does work for me)

uri@mars:~$ which resolvconf
uri@mars:~$ dpkg -l | fgrep resolv
ii  libgeoip1:amd64                            1.6.12-1                                         amd64        non-DNS IP-to-country resolver library
uri@mars:~$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Jan 20 20:49 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
uri@mars:~$

[DimitriPapadopoulos]

OK, it looks like pppd does not do the right thing either. I was hoping it would, like on macOS where relying on ppp is the proper workaround. It shows Canonical themselves (who package pppd) are unable to do the right thing when the name resolution subsystem keeps changing with each release.

Right now you have a couple workarounds:

• Install and use the NetworkManager-fortisslvpn DEB package, which takes over routing and name resolution changes.
• Write your own if-up/if-down scripts for pppd.
• Install DEB package resolvconf which (I think) does the right thing. I have been using the latest version of openfortivpn on Ubuntu 20.04 for a couple weeks and it has been working for me, except I just noticed I have been using resolvconfall this time - left over from my tests.

In the long term we really need to improve how we interact with the system to modify name resolution, taking into account all possibilities:

1. NetworkManager is installed and in use.
2. systemd-resolved is installed and in use.
3. resolvconf (from packages resolvconf or openresolv) is installed and in use.
4. As a last chance attempt modify /etc/resolv.conf directly.

See #600.

[uishon]

Installed resolvconf DEB package and it works 😄.
A huge thanks for the quick support.

[DimitriPapadopoulos]

The strange thing is that I have removed resolvconf from my workstation and yet /etc/resolv.conf is not reset even after hours of operation. I think I have a pretty standard Ubuntu 20.04 installation.

Perhaps there are other factors here, such as

• renewed DHCP lease,
• other subsystems enabled, or not, such as systemd-resolved.

[uishon]

Well, I was under the impression my installations are pretty standard too... Anyway, your assumption regarding other factors playing part here makes sense. As a matter of fact, system-resolved is running on all systems I have with this openfortivpn DNS issue. I never explicitly enabled/disabled system-resolved. I think it might be a factor of the installed Ubuntu flavor. The systems where system-resolved is running are all Ubuntu server installation. The system where systemd-resolved is not running (and where I had no DNS issue to being with) is an Ubuntu desktop install.

[DimitriPapadopoulos]

Ah, I do have a desktop install indeed. But how do you define "systemd-resolved enabled"? On my workstation:

$ systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2020-05-01 11:00:49 CEST; 48min ago
       Docs: man:systemd-resolved.service(8)
             https://www.freedesktop.org/wiki/Software/systemd/resolved
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 1037 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 19024)
     Memory: 10.8M
     CGroup: /system.slice/systemd-resolved.service
             └─1037 /lib/systemd/systemd-resolved
[...]

That said I cannot be 100% certain I haven't changed the defaults while testing openfortivpn.

[uishon]

Well, I just double checked my systems' systemd-resolved status by running systemctl status systemd-resolved and I see it is running on both the Ubuntu desktop where openfortivpn was working properly in the first place and the Ubuntu server where it wasn't (until I've installed resolvconf). Not sure what difference I saw at the time :-( Trying to figure out...

[DimitriPapadopoulos]

Perhaps NetworkManager?

$ systemctl status NetworkManager
● NetworkManager.service - Network Manager
     Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2020-05-01 11:00:49 CEST; 11h ago
       Docs: man:NetworkManager(8)
   Main PID: 1111 (NetworkManager)
      Tasks: 3 (limit: 19024)
     Memory: 16.2M
     CGroup: /system.slice/NetworkManager.service
             └─1111 /usr/sbin/NetworkManager --no-daemon
[...]

[DimitriPapadopoulos]

Must be NetworkManager, see Ubuntu Network Manager: Enabling and disabling NetworkManager on Ubuntu:

An alternative to NetworkManager is systemd-networkd. On Ubuntu desktop, network manager is the default service that manages network interfaces through the graphical user interface. Therefore, If you want to configure IP addresses via GUI, then the network-manager should be enabled.

An Alternative to Ubuntu network manager is systemd-networkd, which is the default backend service in Ubuntu server 18.04.

Again on my workstation:

$  systemctl status systemd-networkd
● systemd-networkd.service - Network Service
     Loaded: loaded (/lib/systemd/system/systemd-networkd.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:systemd-networkd.service(8)