VPNs and IPsec Tunnels

A VPN is a controlled path through an untrusted or shared network. The hard operational parts are rarely just encryption. Operators also have to reason about identity, routes, traffic selectors, NAT traversal, MTU, firewall policy, failover, and whether both peers agree about what traffic belongs in the tunnel.

Command Examples

ip route
ip rule
ip xfrm state
ip xfrm policy
ipsec statusall
tcpdump -nn -i any udp port 500 or udp port 4500 or esp
ping -M do -s 1372 <remote-ip>

Example output and meaning:

Use these checks to separate routing, IPsec state, policy matching, IKE negotiation, ESP forwarding, NAT traversal, and tunnel MTU issues.

VPN Types

Common VPN patterns:

Every VPN creates at least two views of the network: the outer path that carries encrypted packets and the inner path that applications think they are using.

IPsec Building Blocks

IPsec protects IP traffic with security policy and security associations.

IKE normally uses UDP 500. With NAT traversal, peers use UDP 4500 and ESP is encapsulated so NAT devices can track the flow.

IKEv2 and Child SA Timeline

sequenceDiagram
  participant A as Peer A
  participant B as Peer B

  A->>B: IKE_SA_INIT proposals, DH, nonce
  B-->>A: Chosen proposal, DH, nonce, NAT detection
  A->>B: IKE_AUTH identity, auth, traffic selectors
  B-->>A: IKE_AUTH identity, auth, Child SA accepted/narrowed
  A->>B: ESP or UDP 4500 encrypted data
  B-->>A: ESP or UDP 4500 encrypted data
  A->>B: CREATE_CHILD_SA rekey before lifetime expiry

Negotiation failure map:

Tunnel Mode and Transport Mode

In tunnel mode, IPsec encapsulates the original IP packet and adds a new outer IP header. This is the normal mode for site-to-site and remote-access VPNs because the inner addresses can be private networks.

In transport mode, the original IP header remains and IPsec protects the payload. This is more common for host-to-host or specialized designs.

Operational implications:

• firewalls on the outer path must allow IKE and ESP or UDP 4500,
• firewalls on the inner path still need to allow the actual application ports,
• packet captures on the outer path see peers and ESP, not inner TCP/UDP details,
• packet captures after decapsulation show the inner flow.

Traffic Selectors and Policy-Based VPNs

Traffic selectors define which source and destination prefixes, protocols, and ports are eligible for a Child SA. In policy-based IPsec, selectors are the tunnel contract. If one side says 10.10.0.0/16 -> 172.16.0.0/16 and the other side says 10.10.1.0/24 -> 172.16.0.0/16, negotiation may narrow, fail, or silently protect less traffic than expected depending on the peers.

Troubleshooting policy-based tunnels:

ip xfrm policy
ip xfrm state
ip route get <remote-ip>
tcpdump -nn -i any host <remote-ip>

If routes point correctly but counters on the XFRM state do not move, the packet may not match the IPsec policy or may be NATed before policy lookup in a way that changes the selectors.

Route-Based VPNs

A route-based VPN exposes a virtual tunnel interface or XFRM interface and then routes traffic into it. This usually makes routing, failover, and multiple subnets easier because routes decide what enters the tunnel.

Route-based designs still use IPsec SAs underneath. The difference is operational: routing policy decides traffic placement, and the tunnel interface becomes an observable boundary for counters and captures.

Watch for:

• routes missing or pointing to the wrong table,
• overlapping private ranges,
• asymmetric return paths,
• source address selection problems,
• inner firewall policy blocking tunneled traffic,
• broad 0.0.0.0/0 selectors paired with route filtering.

Split Tunneling and Default Routes

Split tunneling sends only selected destinations through the VPN. Full tunneling sends the default route through it. Both are valid, but they answer different security and usability questions.

Split tunnel risks:

• local LAN access can bypass central controls,
• DNS may leak or resolve differently inside and outside the tunnel,
• overlapping routes can choose the wrong path,
• users can reach some private prefixes but not dependencies outside the selected route set.

Full tunnel risks:

• all traffic depends on VPN capacity and policy,
• internet-bound traffic may hairpin through a gateway,
• local printers or private home networks may stop working,
• MTU problems affect more applications.

MTU and MSS

VPNs add headers. IPsec, UDP encapsulation, PPP, GRE, VXLAN, VLANs, and cloud overlays all reduce the effective payload size. When Path MTU Discovery is broken by blocked ICMP, small tests pass and larger TLS, SSH, or database packets hang.

Checks:

tracepath <remote-ip>
ping -M do -s 1372 <remote-ip>
tcpdump -nn -i any 'icmp or esp or udp port 4500'

Fix the path when possible by allowing ICMP Packet Too Big or fragmentation-needed messages. MSS clamping can be a pragmatic workaround for TCP, but it does not fix UDP applications.

Troubleshooting Flow

1. Confirm the outer path: peer reachability, UDP 500, UDP 4500, ESP, and NAT devices.
2. Confirm identity and authentication: certificate, PSK, EAP, IDs, and trust chain.
3. Confirm proposals: IKE version, encryption, integrity, DH group, and lifetimes.
4. Confirm selectors or routes: what traffic is supposed to enter the tunnel.
5. Confirm XFRM state and policy counters move when real traffic is generated.
6. Check inner firewall, return routing, NAT, and source address expectations.
7. Test MTU with large packets and capture both outer and inner boundaries when possible.

Study Cards

References

• RFC 4301: Security Architecture for IP
• RFC 7296: IKEv2
• strongSwan IPsec protocol introduction
• strongSwan forwarding and split tunneling
• ip-xfrm(8)