Results for 27 VPN Services Tested in Mac OS X

As Figure 1 shows, only four of the 27 OS X VPN clients tested (IVPN, OVPN.se, Private Internet Access and VyprVPN) didn't leak. The rest either leaked IPv6 packets whenever the Internet uplink was active, or leaked IPv4 (and generally, IPv6) and/or DNS queries while reconnecting after uplink interruption, or all three. Of the 27 VPN services tested, only IVPN provided leak-free clients for both Windows 7 and OS X Yosemite.

The rest leaked DNS lookups, and IPv4 and IPv6 traffic, under various circumstances. Almost two thirds leaked DNS lookups while reconnecting after an uplink interruption. Over one third leaked IPv6 traffic whenever the Internet was reachable, even when the VPN was stably connected. And almost half leaked IPv4 traffic while the VPN was reconnecting after uplink interruption.

Methods

Testing Environment: I used VirtualBox 5.0.18 x64, running in Debian 8.4 x64, on a quad-core i5 box with 8GB RAM and an SSD RAID10 array. I used a separate Mac OS X Yosemite x64 VM for testing each VPN service. The VMs were attached to a VirtualBox internal network with full IPv6 connectivity, provided by an IPv4/IPv6 OpenVPN TAP tunnel to a VPS hosted by GigaTux, and configured as recommended by OpenVPN.

VPN Clients: I drew on That One Privacy Site in selecting VPN services. I knew that IPv6 leakage would be a serious issue, and so I included all VPNs with claims about IPv6. Otherwise, I picked those with the highest ratings for privacy and technical issues.

I generally tested OpenVPN clients. If a custom client was available, I used it. Otherwise, I generally used stock Viscosity v1.6.2 (which was the latest version at the time). Ironsocket and MyIP provided OpenVPN Connect v2.0.18.203. I generally used VPN servers in Amsterdam, and always in UDP mode. I found no OpenVPN client or configuration for Ivacy and Privatoria, and so I used L2TP clients.

Stock OpenVPN clients have no leak-protection options. For custom clients, I started by accepting the defaults. If there was an option to disable or block IPv6, I used it. If there was an option to secure DNS by using VPN-specified DNS servers, or specifying custom third-party DNS servers, I used it. But I did not enable DNSCrypt. I enabled firewalls, and Internet kill switches that work at the network level, by reconfiguring or disabling network adapters and so on. I did not use kill switches that close particular applications when the VPN connection fails. See Table 1 for specifics.

Table 1. Clients

Testing Protocol: I used a battery of nine tests, and repeated it five times: 1) before connecting the VPN; 2) after connecting the VPN; 3) after interrupting the uplink, and letting the VPN reconnect, or reconnecting manually if necessary; 4) after disconnecting the VPN; and 5) after quitting the client. Briefly, the tests were: 1) ifconfig to get IPv4 and IPv6 addresses; 2) tcpdump -D to get interface numbers; 3) https://dnsleaktest.com/ to see what DNS server(s) were being used; 4) http://test-ipv6.com/ to get public IPv4 and IPv6 addresses, and information about IPv6 connectivity; 5) http://whatismyipaddress.com/ to determine whether IPv4 or IPv6 was preferred; 6) http://ipv6.google.com/ to test for IPv6 DNS resolution; 7) ping 2a00:1450:4001:816::200e to test for IPv6 Internet connectivity; 8) ping 192.168.100.1 to test for IPv4 LAN connectivity; and 9) ping6 [/64 used for LAN]:1 to test for IPv6 LAN connectivity. I used wtee to save all command output, printed all test sites to PDF files, and retained all VMs.

After connecting the VPN, I started capturing packets on the physical LAN interface using tcpdump, and excluded traffic with the VPN server. To ensure that there would be traffic as the VPN was reconnecting after uplink interruption, I started scripts to ping 8.8.8.8 (1 sec interval) and to wget google.com (10 sec interval).

Analysis of Packet Captures: The default tcpdump output format is very simple, and easy to parse:

[timestamp] [packet type] [source address].[port] > [destination address].[port]: [data]

I did all of the work in Gnumeric. As the first step, I excluded arp and other intra-LAN packets. I considered intra-LAN IPv4 packets to be those with both source and destination address being among the following:

0.0.0.0 | 224.0.0.* | 239.255.255.* | 192.168.100.*

I considered intra-LAN IPv6 packets to be those with both source and destination address being among the following:

:: | FF02::* | FE80::* | [/64 used for LAN]:*

I segregated IPv4 and IPv6 packets. For the non-LAN IPv6 data, I simply identified all of the packet blocks, considering gaps longer than 20 seconds to delimit blocks. I classified the non-LAN IPv4 data in six categories: 1) secondary VPN servers hit during reconnection; 2) GigaTux nameserver (ns1.velia.net); 3) Google nameservers (Google prefix and port 53); 4) other nameservers (port 53 but not Google prefix); 5) other Google addresses (Google prefix but not port 53); and 6) other remote addresses (neither VPN server nor Google prefix nor port 53). I obtained Google prefixes from the Hurricane Electric BGP Toolkit, and supplemented the list with Google prefixes identified in the data. I excluded secondary VPN servers, and identified packet blocks for the other five categories, as for the IPv6 data.

Finally, for non-LAN IPv6 and each of the five non-LAN IPv4 categories, I considered the timing of packet blocks versus seven events: 1) initial VPN connection; 2) disconnecting the virtual LAN from the Internet; 3) reconnecting the virtual LAN; 4) reconnection of the VPN; 5) manual disconnection of the VPN; 6) exiting from the VPN client; and 7) ending the packet capture.

Results

As Figure 1 shows, only four of the 27 OS X VPN clients tested (IVPN, OVPN.se, Private Internet Access and VyprVPN) didn't leak. The rest either leaked IPv6 packets whenever the Internet uplink was active, or leaked IPv4 (and generally, IPv6) and/or DNS queries while reconnecting after uplink interruption, or all three. Well over one third even leaked IPv4 traffic while the VPN was stably connected. Of the 27 VPN services tested, only IVPN provided leak-free clients for both Windows 7 and OS X Yosemite.

Figure 1. Leaks

All leak-free VPN clients (IVPN, OVPN.se, Private Internet Access and VyprVPN) were custom, and all reconnected automatically after uplink interruption. See Figure 2 and Figure 1. However, although most of the other ten custom OpenVPN clients also reconnected automatically after uplink interruption, they still leaked IPv4 and/or IPv6 packets. Predictably, VPN clients that didn't reconnect automatically after uplink interruption leaked IPv4 and/or IPv6 packets.

Figure 2. Client Reconnection

Although Viscosity did not reconnect automatically, restoring the uplink did trigger a prompt to do so. See Table 2. However, there's no configuration option for automatic reconnection, and naive users would arguably be blindsided. Given that my standard here is protecting naive users, I chose to test naively.

Table 2. Uplink Flap

None of the Windows clients tested leaked IPv4 traffic while the VPN was stably connected. However, for well over one third of OSX clients, including some with leak protection enabled, some incoming packets bypassed the VPN tunnel. Compare Windows Figure 1 with Figure 1, and see Table 1. Arguably, these packets must be part of conversations established before connecting the VPN, and so I've not considered them to be leaks. CyberGhost's OS X client hit the same CloudFlare IP that its Windows client did, whenever the uplink was active, and I've also not considered that to be a leak.

It's arguable that my testing protocol – constantly pinging 8.8.8.8 and hitting http://google.com/ – was too extreme. But as with the Viscosity issue, my standard here is protecting naive users. And the key point, I believe, is that four of the VPN clients that I tested did not leak under those circumstances. Also, it's not uncommon for VPN users to be running BitTorrent clients, Bitcoin wallets, and other online apps unattended. And it's likely that at least some of them will restart failed VPN connections without closing those online apps.

Three VPN services (AzireVPN, FrootVPN and MyIP) did provide a new IPv6 address, rather like an IPv6-ready ISP. However, all three leaked IPv4 and/or IPv6 packets during reconnection. See Table 3. For all three leak-free VPN clients, no public IPv6 address was visible, and there was no IPv6 connectivity. For almost one third of VPN clients, there was full IPv6 connectivity with IPv6 DNS resolution, even when the VPN was stably connected. That is, they could directly browse http://ipv6.google.com/, entirely bypassing the VPN tunnel.

Table 3. IPv4 and IPv6 Connectivity

Seven VPN clients (AirVPN, IVPN, NordVPN, OVPN.se, Private Internet Access, SlickVPN and VyprVPN) replace the default GigaTux DNS server (nameserver) and restrict DNS queries to the VPN tunnel. See Table 4. Three VPN clients (Mullvad, Proxy.sh and VPNArea) specify their own resolving nameservers, but leak by hitting them directly while reconnecting after uplink interruption. For Mullvad, that was the only leak observed. The other 17 VPN clients hit the default GigaTux nameserver directly when reconnecting after uplink interruption.

Table 4. DNS Servers

Most VPN clients do not interfere with LAN access by the machine. See Table 5. That's typically not an issue. For those running servers or file-sharing applications on ports forwarded through the VPN, misconfiguration could lead to unintended LAN access. Some VPN clients entirely disable IPv6, and that also blocks IPv6 LAN access. Others have options to block LAN access. See Table 1.

Table 5. LAN Access

IVPN provided funding and technical support for this work. But mirimir is responsible for all content.