Include support for Palo Alto Globalprotect split-tunneling includes

[vstoyanov]

Hello,

I think globalprotect offers split-tunneling include configuration, which is ignored in the current release.

Unknown GlobalProtect config tag <include-split-tunneling-domain>:
            *.database.windows.net:1433
            *.azure.com:443
            *.windows.net:1433
       
Unknown GlobalProtect config tag <exclude-video-redirect>: yes

It seems to me that this bit of configuration is pretty straight-forward/obvious to reverse engineer. Given some guidance I could also try and help with the implementation.

Best regards,
Vasil

[dlenski]

Interesting. As you can see, I haven't encountered these config tags before, and I'm not 100% sure how to interpret them.

• Does the <include-split-tunneling-domain> mean that all the wildcard domains listed therein should be routed through the VPN tunnel? Or shouldn't be routed via the VPN tunnel?
• IP routing is done based on IP addresses, not based on DNS names…
  • GP config already has mechanisms for supporting split tunneling based on IP addresses, which OpenConnect fully supports.
  • Doing routing based on DNS wildcards would mean running a masquerade/intermediary DNS server on the localhost, intercepting DNS lookups, deciding if they match the VPN wildcards, and if so dynamically modifying the routing table appropriately. I don't know of any tool that does this automatically; the standard vpnc-script certainly doesn't, my vpn-slice doesn't, and it seems like it'd be a pain to write such a tool, a potential security nightmare, and difficult to manage with multiple simultaneous VPN connections.
  • It's not clear how DNS- and IP-based routing can be combined in a consistent and unsurprising way. A hostname listed in the wildcard could have the same IP address as another hostname not listed, yet the traffic for the second hostname would get (surprisingly) redirected after a DNS lookup of the first one.

So I'm not sure how to handle this DNS-based split tunneling, and I'm kind of suspicious of the security characteristics or value to the end-user. Anything you think I'm missing here?

As for exclude-video-redirect, I have no idea what this one means. 🤷‍♂️

[vstoyanov]

Hello Dan,
Thank you for paying attention to the issue and apologies for the late response.

According to the semantics of the PaloAlto GP configurations I have seen, I am pretty certain that the domains listed in include-split-tunneling-domain should be routed through the VPN tunnel. The workaround we use for now is sudo ip route add 40.0.0.0/8 dev tun0 scope global which routes all traffic to microsoft through the tunnel, so that we can access firewalled services.

I agree that split tunneling on FQDN is a complex task, but since this feature is mainly used for statically configured firewalled services I reckon their addresses won't change that often (with the exception of DNS routing), so my proposed implementation would be to do a DNS resolve on establisihng connection and then modify the routing table based on that.

About exclude-video-redirect I have the suspicion it actually relates to all UDP traffic, but don't have a Windows machine on hand to verify that.

I will try to gather more information and get back in touch.

Thanks,
Vasil

[dlenski]

… my proposed implementation would be to do a DNS resolve on establisihng connection and then modify the routing table based on that.

How would we do a DNS resolve on something like *.database.windows.net:1443? There's no guarantee that all the hosts matching that pattern have IP addresses in a single subnet which is determinable at VPN connection time… is there?

About exclude-video-redirect I have the suspicion it actually relates to all UDP traffic, but don't have a Windows machine on hand to verify that.

Plausible theory, yeah. Would be good to confirm.

If I'm understanding all this correctly, it seems that the office GlobalProtect client for Windows actually runs a sort of transparent proxy that intercepts and rewrites TCP/UDP traffic dynamically. (Much like sshuttle, actually, but without the general purpose usefulness 🤷‍♂️.)

[dlenski]

See upstream MR: https://gitlab.com/openconnect/openconnect/-/merge_requests/132

[Vanuan]

Please also add the dynamic-split-include-domains as a way to configure those, as described in dlenski/vpn-slice#68

It is a Cisco ASA custom attribute which appears to be similar to Globalprotect's include-split-tunneling-domain

[Vanuan]

… my proposed implementation would be to do a DNS resolve on establisihng connection and then modify the routing table based on that.

How would we do a DNS resolve on something like *.database.windows.net:1443? There's no guarantee that all the hosts matching that pattern have IP addresses in a single subnet which is determinable at VPN connection time… is there?

Hm. Well, that might be the reason why this feature is not supported by the official Anyconnect linux client.

On Windows and OS X it appears to be solved by installing a custom driver.

On Linux it seems to be complicated:
https://superuser.com/a/1210156

[Vanuan]

I'm thinking maybe it's possible to subscribe to the D-Bus system-resolved messages? So that when some domain name gets resolved, openconnect will receive a signal "example.com was resolved to 10.0.0.1" and add the route "10.0.0.1 via tunnel0" according to the rules specified by the domain-based split tunneling.

[Vanuan]

Or, alternatively, start a tiny DNS server and use DBus to add it for the tunnel interface. In this DNS server you would spy on the DNS query and add/update a route whenever there's a match. Then, proxy the request to a real DNS server.

[dlenski]

I'm thinking maybe it's possible to subscribe to the D-Bus system-resolved messages? So that when some domain name gets resolved, openconnect will receive a signal "example.com was resolved to 10.0.0.1" and add the route "10.0.0.1 via tunnel0" according to the rules specified by the domain-based split tunneling.

Yes, this is the (least-in)sane way to handle a name-based split tunnel for *.database.windows.net:1443…

1. Run a local DNS server (systemd-resolved or dnsmasq).
2. Get notified whenever there's a DNS lookup for a matching host.
3. Add route(s) to the resolved IP(s) of that host, either including them in the VPN tunnel or excluding them as appropriate.
4. To handle the split at the transport layer (TCP port) you'd have to run some sort of kernel-level TCP proxy ("sshuttle but on drugs") which inspects TCP packets' and forwards them towards either the VPN route or the WAN route depending on destination port.

Even if you did all this…

• It still won't work for software which has cached DNS entries (e.g. web browsers) or doesn't use the system DNS resolver for whatever other reason.
• It will completely bork other domain names which point to the same IP address(es), because DNS and IP addresses are not 1:1 or even 1:many.

This is kind of 🤬 nuts. It's completely mismatched to the layered nature of DNS/IP/TCP on the real Internet.

My thought is that it can't possibly be reliable enough for any VPN administrators who know what they're doing to use it for anything that's performance- or security-critical. It might be useful for excluding bandwidth-heavy traffic from the VPN (e.g. *.sipe.skype.com or *.media.youtube.com).

[Vanuan]

It still won't work for software which has cached DNS entries (e.g. web browsers) or doesn't use the system DNS resolver for whatever other reason.

Well, if it works for windows, I don't see why it won't work for linux. There's a TTL specifically for this issue. As for not using system resolver, well, that's another can of worms...

It will completely bork other domain names which point to the same IP address(es), because DNS and IP addresses are not 1:1 or even 1:many.

Yes, there is a presumption that people do sane things and don't expect domain-based split routing to be magic. Otherwise deep packet inspection / policy based routing must be used to inspect whether we can get the domain from the TCP protocol.

This is a brave new world, where government censorship and the pandemic-imposed work from home policy dictate new rules for the internet :(

[jocado]

@dlenski Is the WIP MR https://gitlab.com/openconnect/openconnect/-/merge_requests/132 usable , or waiting on anything else before it could be tested ?

We are looking to get some early testing on how well this could work for Linux [ openconnect ] clients, as we have recently started leveraging this functionality if other platform clients.

[dlenski]

@dlenski Is the WIP MR https://gitlab.com/openconnect/openconnect/-/merge_requests/132 usable , or waiting on anything else before it could be tested ?

@jocado In terms of how it extracts the contents of the <include-split-tunneling-domain> tag, it's “usable.”

The issue is that we don't have a clear consensus on how domain-based split-tunneling should be handled not only for GlobalProtect but for other protocols as well. See the discussions at vpnc-scripts#5.

We are looking to get some early testing on how well this could work for Linux [ openconnect ] clients, as we have recently started leveraging this functionality if other platform clients.

Unless you have modified the existing vpnc-script or vpn-slice to handle domain-based split tunneling in an intelligent way, then there is no way to test this functionality on Linux.

And if you have done that, I'd really like to see the code you've got 😬.

Or if you have some thoughts on how your organization is handling domain-based split tunneling on other platforms, and want to propose them as the standard behavior for OpenConnect too, please chime in on vpnc-scripts#5. It'll certainly give me more confidence to push out a change if someone can give me a fairly detailed and coherent explanation of how their organization uses domain-based split tunneling.

[jocado]

Unless you have modified the existing vpnc-script or vpn-slice to handle domain-based split tunneling in an intelligent way, then there is no way to test this functionality on Linux.

That is what I meant when I was asking if it was waiting for anything else before it could be tested :)

Actually, I'm glad you mentioned vpn-slice, as I had not heard of it before. It's possible that could be used to achieve some of what we want.

Or if you have some thoughts on how your organization is handling domain-based split tunneling on other platforms, and want to propose them as the standard behavior for OpenConnect too, please chime in on vpnc-scripts#5. It'll certainly give me more confidence to push out a change if someone can give me a fairly detailed and coherent explanation of how their organization uses domain-based split tunneling.

I will go back to the team that are looking after the solution and get the exact requirements of the problem it's solving, and also try and dig out implementation details for how this works on the other platforms. It will be GP centric, but ff I can get some useful info I will report back here.

[616b2f]

There is also a exclude-split-tunneling-domain tag possible in GlobalProtect config e.g.:

 <exclude-split-tunneling-domain>
         <member>windowsupdate.microsoft.com:80,443</member>
 </exclude-split-tunneling-domain>

are there any plans yet to include it too?

[dlenski]

There is also a exclude-split-tunneling-domain tag possible in GlobalProtect config e.g.:

 <exclude-split-tunneling-domain>
         <member>windowsupdate.microsoft.com:80,443</member>
 </exclude-split-tunneling-domain>

are there any plans yet to include it too?

The standard vpnc-script also does not handle split-tunneling/split-DNS in a coherent way. https://gitlab.com/openconnect/vpnc-scripts/-/issues/5

Merely reading and storing the <{include,exclude}-split-tunneling-domain> tags from the config is trivial (I have an MR up to do it in OpenConnect: https://gitlab.com/openconnect/openconnect/-/merge_requests/132)… but figuring out how to handle them in a sane cross-platform way is difficult.

Personally, I don't think name-based includes and excludes are things that a VPN should do. They require deep packet inspection and rewriting to handle (semi)sanely, and there are tons of corner cases that will make it a huge unreliable mess, as I wrote in a comment above.

[616b2f]

The standard vpnc-script also does not handle split-tunneling/split-DNS in a coherent way. https://gitlab.com/openconnect/vpnc-scripts/-/issues/5

I see, my problem is also that I use NetworkMonitor and it does not provide a way how I can use vpnc-script at all.

Merely reading and storing the <{include,exclude}-split-tunneling-domain> tags from the config is trivial (I have an MR up to do it in OpenConnect: https://gitlab.com/openconnect/openconnect/-/merge_requests/132)… but figuring out how to handle them in a sane cross-platform way is difficult.

IMHO the responsibility of OpenConnect client is only to recognize those and pass them along, regardless of the possibility of vpnc-script to handle it properly. I have not that much experience in networking, especially in VPN but it does not look to me that other VPN clients (GP, Cisco Any Connect etc.) will remove this config. It is really hard to use OpenConnect without this feature when you need it to connect to those Gateways (this is also only my opinion).

Personally, I don't think name-based includes and excludes are things that a VPN should do. They require deep packet inspection and rewriting to handle (semi)sanely, and there are tons of corner cases that will make it a huge unreliable mess, as I wrote in a comment above.

I understand your worries about how this could end up in a big mess, but I also think we need to look in the future, what I mean by that is the Cloud Services are evolving and relying only on IPs is kind of outdated IMHO. IPs change now more often than ever and I believe that they will change more often in the future, the world is more dynamic then ever.

Do we know how the other clients handle the config? e.g. official GP clients?

In this post: https://superuser.com/a/1210156 Possum described some Methods/Strategies how such Policy Based Routing can be done on a Linux system (when I understood correctly some of them match what you described in the comment above as a possible solution), what I thought, maybe we could implement a simple and a more advanced one and select which one to use. What do you think?

[markcellus]

Thanks for opening this ticket. This issue is the only thing holding me back from using this client with my VPN provider.

Is there any update? Or should we request an update in https://gitlab.com/openconnect/openconnect/-/merge_requests/132?

[jocado]

CISCO_SPLIT_DNS is actually supported now by vpnc-scripts for systemd with systemd-resolved. This makes sense, because stemd-resolved seems to handle routed dns domains very well.

Please see comment 2 here: https://gitlab.com/openconnect/vpnc-scripts/-/issues/5

So, in theory, if openconnect parsed include-split-tunneling-domain into a usable form in CISCO_SPLIT_DNS, it should work for systems using systemd-resolved.

I'm not sure you could sensibly support the above example which seems to include ports, so you would have to strip them off I think.

Our infra returns the data in this form, which is currently picked up by the other Win or Macos clients:

< 		<include-split-tunneling-domain>
< 			<member>*.foo.bar</member>
< 			<member>*.moo.bar</member>
< 			<member>*.1.10.in-addr.arpa</member>
< 			<member>*.2.10.in-addr.arpa</member>
< 		</include-split-tunneling-domain> 

To make that list usable for systemd-resolvd you would have to remove the leading *. from each entry, so the variable would be:

CISCO_SPLIT_DNS=foo.bar,moo.bar,1.10.in-addr.arpa,2.10.in-addr.arpa

They could also be supplied to systemd-resolved in this form, which would make them route only dns routing domain only, and not a dns search domain.

CISCO_SPLIT_DNS=~foo.bar,~moo.bar,~1.10.in-addr.arpa,~2.10.in-addr.arpa

This second form is preferable IMO, but potentially not as widely expected.

There's a nice explanation systemd-resolved behaviour in this regard here: https://systemd.io/RESOLVED-VPNS/

Would it be reasonable to populate CISCO_SPLIT_DNS in this case as above ?

Cheers,
Just