Results for 29 VPN Services Tested in Windows 7

As Figure 1 shows, all but six of the 29 Windows VPN clients tested either leaked IPv6 packets whenever the Internet uplink was active, or leaked IPv4 (and generally, IPv6) and/or DNS queries while reconnecting after uplink interruption, or all three. Those six are AirVPN, FrootVPN, IVPN, Mullvad, Perfect Privacy and SlickVPN.

The rest leaked DNS lookups, and IPv4 and IPv6 traffic, under various circumstances. Almost half leaked DNS lookups while reconnecting after an uplink interruption. Over half leaked IPv6 traffic whenever the Internet was reachable, even when the VPN was stably connected. Over half leaked IPv4 traffic while the VPN was reconnecting after uplink interruption. And over a third leaked in all three ways.

Methods

Testing Environment: I used VirtualBox 5.0.18 x64, running in Debian 8.4 x64, on a quad-core i5 box with 8GB RAM and an SSD RAID10 array. I used a separate Windows 7 SP2 x64 VM for testing each VPN service. The VMs were attached to a VirtualBox internal network with full IPv6 connectivity, provided by an IPv4/IPv6 OpenVPN TAP tunnel to a VPS hosted by GigaTux, and configured as recommended by OpenVPN.

VPN Clients: I drew on That One Privacy Site in selecting VPN services. I knew that IPv6 leakage would be a serious issue, and so I included all VPNs with claims about IPv6. Otherwise, I picked those with the highest ratings for privacy and technical issues.

I only tested OpenVPN clients. If a custom client was available, I used it. If the provider recommended a particular third-party client, I used that. Otherwise, I used stock OpenVPN v2.3.10 (which was the latest version at the time). I generally used VPN servers in Amsterdam, and always in UDP mode.

The stock OpenVPN client has no leak-protection options. For custom clients, I started by accepting the defaults. If there was an option to disable or block IPv6, I used it. If there was an option to secure DNS by using VPN-specified DNS servers, or specifying custom third-party DNS servers, I used it. But I did not enable DNSCrypt. I enabled firewalls, and Internet kill switches that work at the network level, by reconfiguring or disabling network adapters and so on. I did not use kill switches that close particular applications when the VPN connection fails. See Table 1 for specifics.

Table 1. Clients

Testing Protocol: I used a battery of nine tests, and repeated it five times: 1) before connecting the VPN; 2) after connecting the VPN; 3) after interrupting the uplink, and letting the VPN reconnect, or reconnecting manually if necessary; 4) after disconnecting the VPN; and 5) after quitting the client. Briefly, the tests were: 1) ipconfig /all to get IPv4 and IPv6 addresses; 2) WinDump -D to get interface numbers; 3) https://dnsleaktest.com/ to see what DNS server(s) were being used; 4) http://test-ipv6.com/ to get public IPv4 and IPv6 addresses, and information about IPv6 connectivity; 5) http://whatismyipaddress.com/ to determine whether IPv4 or IPv6 was preferred; 6) http://ipv6.google.com/ to test for IPv6 DNS resolution; 7) ping 2a00:1450:4001:816::200e to test for IPv6 Internet connectivity; 8) ping 192.168.100.1 to test for IPv4 LAN connectivity; and 9) ping [/64 used for LAN]:1 to test for IPv6 LAN connectivity. I used wtee to save all command output, printed all test sites to XPS files, and retained all VMs.

After connecting the VPN, I started capturing packets on the physical LAN interface using WinDump and WinPcap, and excluded traffic with the VPN server. To ensure that there would be traffic as the VPN was reconnecting after uplink interruption, I started scripts to ping 8.8.8.8 (1 sec interval) and to wget google.com (10 sec interval). The wget script ran at a CygWin (Unix) prompt.

Analysis of Packet Captures: The default WinDump output format is very simple, and easy to parse:

[timestamp] [packet type] [source address].[port] > [destination address].[port]: [data]

I did all of the work in Gnumeric. As the first step, I excluded arp and other intra-LAN packets. I considered intra-LAN IPv4 packets to be those with both source and destination address being among the following:

0.0.0.0 | 224.0.0.* | 239.255.255.* | 192.168.100.*

I considered intra-LAN IPv6 packets to be those with both source and destination address being among the following:

:: | FF02::* | FE80::* | [/64 used for LAN]:*

I segregated IPv4 and IPv6 packets. For the non-LAN IPv6 data, I simply identified all of the packet blocks, considering gaps longer than 20 seconds to delimit blocks. I classified the non-LAN IPv4 data in six categories: 1) secondary VPN servers hit during reconnection; 2) GigaTux nameserver (ns1.velia.net); 3) Google nameservers (Google prefix and port 53); 4) other nameservers (port 53 but not Google prefix); 5) other Google addresses (Google prefix but not port 53); and 6) other remote addresses (neither VPN server nor Google prefix nor port 53). I obtained Google prefixes from the Hurricane Electric BGP Toolkit, and supplemented the list with Google prefixes identified in the data. I excluded secondary VPN servers, and identified packet blocks for the other five categories, as for the IPv6 data.

Finally, for non-LAN IPv6 and each of the five non-LAN IPv4 categories, I considered the timing of packet blocks versus seven events: 1) initial VPN connection; 2) disconnecting the virtual LAN from the Internet; 3) reconnecting the virtual LAN; 4) reconnection of the VPN; 5) manual disconnection of the VPN; 6) exiting from the VPN client; and 7) ending the packet capture.

Results

As Figure 1 shows, all but nine of the 29 VPN clients tested either leaked IPv6 packets whenever the Internet uplink was active, or leaked IPv4 (and for the most part, IPv6) while the client was reconnecting after uplink interruption, or both. Those nine are AirVPN, CyberGhost, FrootVPN, IVPN, Mullvad, oVPN.to, Perfect Privacy, SecureVPN.to and SlickVPN. One of them, CyberGhost, did hit a site via CloudFlare, whenever the uplink was active, but it didn't leak in any other way.

Figure 1. Leaks

All of the VPN clients that didn't reconnect automatically after uplink interruption leaked IPv4 packets. See Table 2 and Figure 1. However, four of the VPNs that did reconnect automatically also leaked IPv4 packets during reconnection (MyIP, TorGuard, VPNArea and VyprVPN). It's arguable that my testing protocol – constantly pinging 8.8.8.8 and hitting http://google.com/ – was too extreme. But the key point, I believe, is that nine of the VPN clients that I tested did not leak under those circumstances. Also, it's not uncommon for VPN users to be running BitTorrent clients, Bitcoin wallets, and other online apps unattended. And it's likely that at least some of them will restart failed VPN connections without closing those online apps.

Table 2. Uplink Flap

Most of the VPN clients that reconnected automatically after uplink interruption were custom. See Figure 2. But some of the custom clients did not reconnect automatically, and some of the stock OpenVPN clients did. Also, almost half of the custom clients that reconnected automatically leaked in some way.

Figure 2. Client Reconnection

For most of the leak-free VPN clients, no public IPv6 address was visible, and there was no IPv6 connectivity. See Table 3. Four of the VPN services did provide a new IPv6 address, rather like an IPv6-ready ISP. However, two of those leaked IPv4 and IPv6 packets during reconnection, so only FrootVPN and Perfect Privacy provided leak-free IPv6 connectivity. For almost half of the VPN clients, there was at least some IPv6 connectivity using the GigaTux IPv6 address. For three of them (Predator, CryptoStorm and NordVPN), there was full IPv6 connectivity with IPv6 DNS resolution, allowing them to browse http://ipv6.google.com/.

Table 3. IPv4 and IPv6 Connectivity

Seven VPN clients (AirVPN, FrootVPN, IVPN, Mullvad, MyIP, Perfect Privacy and SlickVPN) replace the default GigaTux DNS server (nameserver) and restrict DNS queries to the VPN tunnel. See Table 4. However, MyIP apparently uses Google nameservers.

Table 4. DNS Servers

Five VPN clients (CyberGhost, oVPN.to, Private Internet Access, SecureVPN.to and TorGuard) specify their own resolving nameservers, but leak by hitting them directly while reconnecting after uplink interruption. Three VPN clients (CryptoStorm, NordVPN and Trust.Zone) leak direct DNS queries to VPN-specified nameservers even while connected, but do not hit the default GigaTux nameserver. The other 14 VPN clients hit the default GigaTux nameserver directly when reconnecting after uplink interruption. And three of them (AzireVPN, Ironsocket and PureVPN) even hit it while connected.

As discussed above, nine VPN clients did not leak IPv4 or IPv6 packets, and did not hit the default GigaTux nameserver. See Figure 1. Six of them (AirVPN, FrootVPN, IVPN, Mullvad, Perfect Privacy and SlickVPN) hit only nameservers through the VPN tunnel. See Table 4. But three of them (CyberGhost, oVPN.to and SecureVPN.to) hit VPN-specified nameservers directly while reconnecting. That leaks information about online activity to ISPs and other local observers.

Most VPN clients do not interfere with LAN access by the machine. See Table 5. That's typically not an issue. For those running servers or file-sharing applications on ports forwarded through the VPN, misconfiguration could lead to unintended LAN access. Some VPN clients entirely disable IPv6, and that also blocks IPv6 LAN access. Two VPN clients (IVPN and Perfect Privacy) have firewalls that block LAN access.

Table 5. LAN Access

IVPN provided funding and technical support for this work. But mirimir is responsible for all content.