This article was motivated by a problem I had: VPN access to my work environment from my new Linux install didn’t work - and some basic searches on the internet provided no obvious answers. It was therefore necessary to look into how VPNs work in some more detail…
The VPN solution I am using is Cisco Anyconnect (aka Openconnect), but much of the info below should apply to other VPN products.
The environment I am setting up the VPN on is Ubuntu 19.10 with Gnome; this implies network-manager for network configuration and systemd-resolved for DNS.
Note that I am a developer, not a networking specialist - any corrections welcome.
Installing Openconnect is pretty easy:
sudo apt install openconnect network-manager-openconnect network-manager-openconnect-gnome
Then go to Gnome’s Settings/Network and add a VPN definition. The VPN can then be enabled from either the settings window or the status-tray.
What a VPN does
So what does “enabling” the VPN actually do?
State With VPN Not Active
Here’s the relevant state of my network without the VPN active:
$ ifconfig enp0s31f6: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether 98:fa:9b:00:7b:d3 txqueuelen 1000 (Ethernet) ... lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) ... wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.147 netmask 255.255.255.0 broadcast 192.168.1.255 inet6 fd96:ffc0:5eb9:0:d46b:1e4:5624:5f20 prefixlen 64 scopeid 0x0<global> inet6 fd96:ffc0:5eb9::43a prefixlen 128 scopeid 0x0<global> ... $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 600 0 0 wlp3s0 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 wlp3s0 192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp3s0 $ cat /etc/resolv.conf nameserver 127.0.0.53 options edns0 search lan $ systemd-resolve --status ... Link 3 (wlps30) Current Scopes: DNS DefaultRoute setting: Yes Current DNS Server: 192.168.1.1 DNS Domain: ~.
The ifconfig entries show that this system has one external network interface named “wlp3s0”, which has been allocated a local IP address of 192.168.1.147 (by the DHCP server it talked to on initialisation). This network interface name effectively identifies a driver context within the Linux kernel; in this case the context is associated with a Wifi driver. The interface named “lo” is associated with a “loopback” driver context, and is not really relevant here.
The second routing entry specifies that all packets sent to addresses of form
169.254.0.0/16) are to be processed by the driver context “wlp3s0”. This range is the link-local address range - a special set of addresses that can only be used to access stations on the same local network segment (ie even more restricted than the private address range).
Similarly, the third routing entry specifies that any packets sent to
192.168.1.* are to be processed by that same context. I presume this is set up to match the gateway address (see below).
The first routing entry specifies that any packet whose destination address is not matched by a more-precise rule should be handled by “wlp3s0”. That first routing entry is special in that its flags include a “G” (Gateway) and it has a gateway address. This tells the code associated with “wlp3s0” that before it sends the packet out over the wireless device, it should wrap the packet in an envelope whose destination-address is the gateway address - ie the packet is redirected. That address happens to be the incoming-packet interface within the router that is attached to the wifi device; the driver within the router will then extract the original packet from the envelope and send it on to one of the ports attached to it, depending on the original address. For addresses that are “not local”, the packet is sent through the port that leads to an ISP which will then forward it into the wide internet.
DNS resolution has had a long and somewhat complicated history in Linux. The main Linux distributions now use systemd-resolved for resolving names; calls from applications to the standard C library functions that map name-to-address get forwarded to this daemon which then emits DNS queries to obtain the info (or returns values from a local cache).
The systemd-resolve output indicates that wlp3s0 is a valid interface to send DNS queries over, and gives the address to send them to. It also specifies for which domain-names this interface should be used to find ip addresses - “~.” means all domain-names.
State With VPN Active
After enabling the Cisco Openconnect VPN via the Gnome UI, things now look somewhat different. Clearly the VPN is at least partially working:
enp0s31f6: (unchanged) lo: (unchanged) wlp3s0: (unchanged) vpn0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1300 inet 172.17.35.95 netmask 255.255.254.0 destination 172.17.35.95 inet6 fe80::440c:186a:c381:d61 prefixlen 64 scopeid 0x20<link> ... $ route (same 3 entries above) 184.108.40.206 192.168.1.1 255.255.255.255 UGH 600 0 0 wlp3s0 220.127.116.11 0.0.0.0 255.255.255.255 UH 50 0 0 vpn0 18.104.22.168 0.0.0.0 255.255.255.0 U 50 0 0 vpn0 .. plus many entries similar to the above two
A new interface “vpn0” has been defined; this corresponds to a context within the kernel that is associated with the Cisco VPN driver code. This code encrypts outgoing packets and wraps them in an envelope that points at the remote VPN endpoint, then resubmits them to the networking layer within the kernel. This context also receives incoming packets, unwraps and decrypts them, then resubmits them to the networking layer for delivery to the relevant application.
The new UGH entry is telling the kernel that all packets with destination
22.214.171.124 (the address on the envelope that the VPN driver adds to all outgoing packets) should be sent via the standard wifi driver and then out via my router’s standard route to my ISP. This address is the remote endpoint of the VPN - obviously all packets need to go over my wifi driver and then to my ISP in order to reach the VPN.
The other entries are ensuring that packets destined for specific address-ranges should be handled by the kernel driver responsible for interface “vpn0”.
Note that this VPN does not simply redirect all outgoing traffic; instead it redirects only specific IP-ranges which the server has been configured to intercept. This is certainly more efficient than catching all traffic; it means my employer does not need to act as a relay when I am interacting with sites that are not of interest. On the other hand, it does mean that my employer’s VPN does not protected from observation or data-manipulation when accessing unrelated insecure sites.
The DNS Problem
The problem that originally motivated me to look into all this is that although the above traffic-routing was working fine, DNS was not.
$ cat /etc/resolv.conf nameserver 127.0.0.53 options edns0 search lan $ systemd-resolve --status ... Link 5 (vpn0) Current Scopes: none DefaultRoute setting: no
Note here that vpn0 is not marked as being suitable for DNS requests, no DNS server is defined, and no domain-names are associated with the link.
Therefore, any request to map a name to an ip-address still goes out interface wlp3s0 to my ISP’s DNS server, and will return NXDOMAIN (not found) for any hostname that is defined only in my company’s internal DNS servers.
Solving the DNS Problem
One good step to resolving the problem is to install a few extra packages:
sudo apt install resolvconf resolvconf-admin
When activating the VPN,
systemd-resolve --status now reports that link vpn0 is suitable for DNS and has a DNS server set.
Sadly, there are still no domain-names associated with that link. Any lookup for a domain-name will therefore still go to the default location, not over the VPN.
Interestingly, assuming my company’s internal DNS names are of form
systemd-resolve somehost.example.com ==> lookup fails systemd-resolve --interface vpn0 somehost.example.com ==> lookup succeeds
The following manual step therefore resolves the issue by binding one or more domains to the interface:
sudo systemd-resolve --interface vpn0 --set-domain example.com --set-domain otherexample.com
There is in fact a standard package that does this for OpenVPN:
sudo apt install openvpn-systemd-resolved
which applies the (manually configured) domains associated with the virtual network whenever the VPN is enabled.
There is no such package for Cisco “openconnect” (yet) as far as I know. I believe that an openconnect server provides a list of the domains that it hosts, ie it is possible for suitable client-side software to download and apply the domain redirects automatically (which is what happens on other platforms such as windows and mac). Of course that means that when activating the VPN, you are not in control of which domains will be routed via the VPN - a potential privacy issue.
For me, running a single command to bind the necessary domains to the interface after activating the VPN is sufficient. I have a script which does this, and run it manually after activating the VPN - a minor nuisance but bearable.
All this should of course be automatic, and hopefully will be in the not-to-distant future. On the other hand, some of the bug-reports listed in the references below indicate that for a “split VPN” of this type (where not everything is redirected) some developers think that the current behaviour is “correct”. This appears to be a core philisophical difference between convenience (automatically map domains) vs privacy (only remap domains the user wants to remap).
Running set-domain automatically
If you use the suggested systemd-resolve command above to configure routing via the VPN, then the following will ensure it is run every time the VPN is started:
TARGET=/etc/network/if-up.d/02-vpn-init sudo tee $TARGET > /dev/null << EOF #!/bin/sh if [ "$IFACE" = "vpn0" ]; then systemd-resolve --interface vpn0 --set-domain .... fi EOF sudo chmod a+x $TARGET
Current versions of Ubuntu use NetworkManager for VPN management, and file
/etc/NetworkManager/dispatcher.d/01-ifupdown ensures the traditional scripts in
/etc/network/if-up.d are executed.
- Linuxconfig.org: VPNs - a nice article on the topic, but focused on OpenVPN
- Stackexchange: Understand the Routing Table - a question on stackexchange with some very good answers
- Ubuntu Bugtracker: DNS domain search paths not updated - from October 2017!
- Ask Ubuntu: No DNS Resolution