VPNs and IPSec

• VPNs and IPSec
  • What a VPN actually does — and what it doesn’t
  • Topologies
  • IPSec: the standards-based suite
    • AH vs ESP — the two protections
    • Transport vs tunnel mode — what gets wrapped
    • IKE — key exchange
  • TLS-based VPNs: OpenVPN
  • WireGuard: the modern approach
  • Worked example: a minimal WireGuard tunnel
  • Limitations and attacks
  • Key takeaways
  • References

What a VPN actually does — and what it doesn’t

A Virtual Private Network (VPN) builds an encrypted tunnel across an untrusted network, so that traffic flowing through it gets confidentiality and integrity even though the underlying path is hostile. The classic threat model is the untrusted local network: coffee-shop Wi-Fi, a hotel LAN, an ISP that logs everything, or the public Internet between two office sites. Inside the tunnel, an eavesdropper sees only ciphertext between you and the VPN endpoint.

That is genuinely useful, but VPNs are surrounded by marketing myths, so it is worth being precise about the boundaries up front:

• A VPN moves your trust, it does not eliminate it. Your ISP can no longer see your traffic — but the VPN provider now can. You have swapped one observer for another, so the provider’s logging policy and jurisdiction matter enormously. (This ties directly into the privacy material.)
• A VPN is not anonymity. The exit point still talks to destinations from a known address; logging, browser fingerprinting, and logged-in accounts still identify you. Tor, not a commercial VPN, is the anonymity tool.
• A VPN protects traffic in transit to the exit, not the destination beyond it, and not your endpoint. Past the VPN exit, traffic relies on its own TLS just like normal.

With the scope honest, the rest of this page is about how the tunnel is built.

Topologies

  Remote-access VPN                         Site-to-site VPN
  ┌────────┐                                ┌──────────┐            ┌──────────┐
  │ laptop │═══tunnel═══▶┌─────────┐        │ Office A  │═══tunnel══▶│ Office B  │
  └────────┘   over      │ VPN     │─▶ LAN  │ gateway   │  over the  │ gateway   │
               Internet  │ gateway │        │ (whole    │  Internet  │ (whole    │
                         └─────────┘        │  subnet)  │            │  subnet)  │

• Remote-access VPN — an individual device tunnels into a network (employee → corporate LAN). This is the “VPN client” most people picture.
• Site-to-site VPN — two gateways join two whole networks across the Internet, transparently to the hosts behind them (branch office ↔ headquarters).

A second axis, for remote access:

• Full tunnel — all of the device’s traffic goes through the VPN (maximum protection and visibility; higher latency).
• Split tunnel — only traffic destined for the protected network uses the tunnel; everything else goes out directly (faster, but the untrusted-network exposure returns for that traffic). The split-vs-full choice is a least-privilege-flavored trade-off between exposure and performance.

IPSec: the standards-based suite

IPSec is the IETF’s protocol suite for securing traffic at the network layer (Layer 3) — it protects IP packets themselves, so it works for any application transparently. It is the backbone of most site-to-site VPNs and is built into virtually every OS and router. IPSec is a suite, not one protocol; the pieces you need to know:

AH vs ESP — the two protections

• AH (Authentication Header, RFC 4302) — provides integrity and authentication of the packet, but no encryption. Rarely used alone today (and it breaks through NAT).
• ESP (Encapsulating Security Payload, RFC 4303) — provides encryption and integrity/authentication. This is what essentially all modern IPSec VPNs use.

Transport vs tunnel mode — what gets wrapped

• Transport mode — encrypts only the IP payload; the original IP header stays visible. Used host-to-host.
• Tunnel mode — encrypts the entire original packet and wraps it in a new IP header. This is what VPN gateways use: the inner addresses (the real source/destination) are hidden, and only the gateway-to-gateway addresses are exposed.

IKE — key exchange

IPSec needs both ends to agree on keys. IKEv2 (Internet Key Exchange, RFC 7296) does the authenticated key negotiation, using a Diffie–Hellman exchange to derive fresh session keys — providing Perfect Forward Secrecy so that compromising a long-term key does not expose past sessions. IKE is also where IPSec’s notorious complexity lives: many configurable cipher suites, DH groups, and modes, which is both its flexibility and a frequent source of misconfiguration.

TLS-based VPNs: OpenVPN

Not every VPN works at Layer 3. OpenVPN builds the tunnel on top of TLS (the same protocol as HTTPS), running in userspace over UDP or TCP — and, by riding TCP/443, it can traverse restrictive firewalls that block IPSec. It is mature, audited, and cross-platform, at some cost in performance versus kernel implementations. SSL/TLS VPNs are common for remote access where firewall traversal matters.

WireGuard: the modern approach

WireGuard is a newer VPN that has rapidly become a default choice, and its design philosophy is the lesson worth absorbing — it is a case study in economy of mechanism:

• Fixed, modern cryptography, no negotiation. Where IPSec lets you pick among dozens of ciphers and DH groups (and misconfigure them), WireGuard hard-codes one audited stack: the Noise_IK handshake, Curve25519 key exchange, ChaCha20-Poly1305 encryption, BLAKE2s hashing. There is no cipher negotiation to downgrade or get wrong.
• Tiny codebase. ~4,000 lines versus IPSec’s ~100,000+ — small enough to actually audit, and a much smaller attack surface.
• In the Linux kernel (since 5.6) for high performance, and trivial to configure.
• Cryptokey routing. Each peer is identified by its public key, which is bound to the set of IP addresses it is allowed to use. Authentication and routing are the same mechanism — a clean idea.

Worked example: a minimal WireGuard tunnel

WireGuard’s simplicity shows in its configuration. Each side generates a keypair, and each lists the other’s public key plus which IPs it may use.

# 1. Generate a private + public key on each peer
$ wg genkey | tee privatekey | wg pubkey > publickey

# 2. /etc/wireguard/wg0.conf  — on the CLIENT
[Interface]
PrivateKey = <client-private-key>
Address    = 10.0.0.2/24            # this peer's address inside the tunnel
DNS        = 10.0.0.1

[Peer]
PublicKey  = <server-public-key>    # who we trust (cryptokey routing)
Endpoint   = vpn.example.com:51820  # where to reach them (UDP/51820 default)
AllowedIPs = 0.0.0.0/0              # 0.0.0.0/0 = full tunnel (all traffic)
PersistentKeepalive = 25            # hole-punch through NAT

# 3. Bring the tunnel up and inspect it
$ sudo wg-quick up wg0
$ sudo wg show
interface: wg0
  public key: <client-public-key>
peer: <server-public-key>
  endpoint: 203.0.113.9:51820
  allowed ips: 0.0.0.0/0
  latest handshake: 8 seconds ago      ← tunnel is established
  transfer: 1.21 MiB received, 480 KiB sent

AllowedIPs = 0.0.0.0/0 makes this a full tunnel; restricting it to, say, 10.0.0.0/24 would make it a split tunnel that only protects traffic to the corporate subnet.

Limitations and attacks

A VPN is one layer, and like every layer it has failure modes — assuming it makes you “safe” is the real danger:

• TunnelVision (CVE-2024-3661) — a malicious DHCP server uses DHCP option 121 to push a more-specific route that sends traffic around the VPN tunnel while it appears up, exposing supposedly-protected traffic. A worked example is linked from the course readme. It is a reminder that the VPN sits on top of a network stack the attacker may also control.
• DNS leaks — if DNS queries bypass the tunnel, your browsing is exposed even with traffic encrypted; pair a VPN with encrypted DNS and verify there is no leak.
• The trust-the-provider problem — for commercial VPNs, the provider sees everything your ISP used to. “No-logs” is a policy claim, not a cryptographic guarantee.
• Endpoint and split-tunnel exposure — a compromised device, or non-tunneled traffic in a split configuration, is unprotected regardless of how strong the tunnel is.

Key takeaways

• A VPN gives confidentiality and integrity across an untrusted network by tunneling to a trusted endpoint — but it relocates trust to the VPN provider and is not anonymity.
• IPSec secures IP packets at Layer 3: ESP (encryption + integrity) in tunnel mode for gateways, with IKEv2 doing authenticated Diffie–Hellman key exchange and Perfect Forward Secrecy. Powerful but complex.
• OpenVPN builds the tunnel on TLS, easing firewall traversal; WireGuard uses a small, fixed, modern crypto stack (Noise/Curve25519/ChaCha20-Poly1305), ~4,000 lines, in-kernel — a model of economy of mechanism.
• Remote-access vs site-to-site and full vs split tunnel are the deployment choices; split tunneling trades protection for performance.
• VPNs fail too — TunnelVision, DNS leaks, provider logging, and endpoint compromise — so treat a VPN as one layer of defense in depth, not a guarantee.

References

• RFC 4301 — Security Architecture for the Internet Protocol (IPSec). https://datatracker.ietf.org/doc/html/rfc4301
• RFC 4303 — IP Encapsulating Security Payload (ESP). https://datatracker.ietf.org/doc/html/rfc4303
• RFC 4302 — IP Authentication Header (AH). https://datatracker.ietf.org/doc/html/rfc4302
• RFC 7296 — Internet Key Exchange Protocol Version 2 (IKEv2). https://datatracker.ietf.org/doc/html/rfc7296
• J. A. Donenfeld, WireGuard: Next Generation Kernel Network Tunnel (NDSS 2017). https://www.wireguard.com/papers/wireguard.pdf
• WireGuard — Protocol & Cryptography. https://www.wireguard.com/protocol/
• OpenVPN — Security Overview. https://openvpn.net/community-resources/security-overview/
• Leviathan Security, TunnelVision (CVE-2024-3661) — decloaking VPN traffic via DHCP. https://www.leviathansecurity.com/research/tunnelvision
• wg(8) and wg-quick(8) man pages. https://man7.org/linux/man-pages/man8/wg.8.html

Related course pages: Cryptography · Network Security · DNS Security and Privacy · Introduction to Networking · Privacy

🛠️ Maintenance note: WireGuard’s in-kernel status and crypto stack are stable, but verify the wg/wg-quick workflow on the course VM each term. TunnelVision (CVE-2024-3661, disclosed 2024) is current; check whether OS-level mitigations have shipped before presenting it as live.