Kubernetes Pod Networking and CNI

Kubernetes defines the network model; the CNI plugin implements it. Operators need to know where the Kubernetes API ends and the datapath begins, because Pod-to-Pod failures often live in CNI routes, encapsulation, host firewall rules, cloud routing, MTU, or node agents.

Command Examples

kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl describe node <node>
kubectl -n kube-system get pods -o wide
kubectl exec -it <pod> -- ip addr
kubectl exec -it <pod> -- ip route

Example output and meaning:

Kubernetes Network Model

The Kubernetes model expects:

• every Pod has its own IP,
• Pods can communicate with Pods on other nodes without NAT,
• nodes can communicate with Pods,
• containers in the same Pod share a network namespace and can use localhost,
• Service networking is layered on top of Pod networking.

That model does not mandate how packets move. Common implementations use routed Pod CIDRs, overlays such as VXLAN or Geneve, cloud-native VPC addressing, BGP, iptables, IPVS, nftables, eBPF, or combinations.

CNI Responsibilities

CNI plugins create Pod network interfaces and connect them to the node and cluster network. A plugin may also implement NetworkPolicy, Service load balancing, encryption, observability, egress gateways, or IPAM.

Operational questions:

• Which CNI plugin is installed?
• What Pod CIDRs are assigned to nodes?
• Is traffic routed, encapsulated, or cloud-native?
• Does the plugin enforce NetworkPolicy?
• Does the plugin replace kube-proxy?
• What MTU does it set for Pods and tunnels?

Common CNI shapes:

Pod Packet Paths

Same-node Pod traffic, cross-node Pod traffic, Service traffic, and external egress can use different paths.

kubectl exec <pod> -- ip addr
kubectl exec <pod> -- ip route
kubectl exec <pod> -- ping -c 2 <same-node-pod-ip>
kubectl exec <pod> -- ping -c 2 <cross-node-pod-ip>
kubectl exec <pod> -- curl -v http://<service>.<namespace>.svc.cluster.local:<port>

Capture comparison for same-node versus cross-node:

kubectl get pod -o wide -l app=client
kubectl get pod -o wide -l app=server
kubectl exec <client-pod> -- curl -v http://<same-node-pod-ip>:8080
kubectl exec <client-pod> -- curl -v http://<cross-node-pod-ip>:8080
tcpdump -nn -i any host <cross-node-pod-ip>

If same-node works and cross-node fails, stop debugging the application listener. Focus on node routes, overlay encapsulation, cloud security groups, tunnel MTU, BGP routes, or CNI node agents.

MTU, Encapsulation, and Node Boundaries

Overlays add headers. If Pod MTU does not account for tunnel overhead, large requests can fail while small requests pass. Kubernetes incidents involving TLS, DNSSEC, gRPC, image pulls, or large HTTP responses can be MTU incidents.

Node boundaries matter. Pod-to-Pod on the same node may work while cross-node traffic fails. Cross-zone, cross-subnet, or cross-VPC traffic may add cloud routing and security group behavior.

hostNetwork and Node Locality

Pods using hostNetwork: true share the node network namespace instead of getting normal Pod networking. That can be useful for node agents, but it changes port conflicts, DNS policy defaults, source addresses, and NetworkPolicy expectations.

DNS and NAT Boundaries

Pod DNS and Pod egress are coupled. A Pod may resolve a Service name to a ClusterIP and stay inside Kubernetes service routing, or resolve an external name to a public address and leave through node SNAT, cloud NAT, an egress gateway, or a proxy. If private endpoint DNS is missing, workloads can silently use public NAT egress even when a private route exists.

Operational checks:

• compare nslookup from the Pod with dig from the node and from outside the cluster,
• check whether CoreDNS forwards external queries through a NATed node path,
• verify NetworkPolicy allows UDP and TCP 53 to CoreDNS,
• confirm whether app egress and DNS egress use the same gateway or policy,
• watch NAT gateway metrics when ndots or retry-heavy clients multiply DNS queries.

Hairpin and SNAT Checks

Hairpin traffic happens when a workload reaches a service through an address that loops back through a load balancer or NAT device. It often appears after public DNS is reused inside the cluster.

kubectl exec <pod> -- getent hosts app.example.com
kubectl exec <pod> -- ip route get <resolved-ip>
kubectl exec <pod> -- curl -v https://app.example.com/

If the name resolves to a public load balancer for in-cluster clients, prefer split-horizon DNS, a ClusterIP Service, a private endpoint, or explicit egress-gateway design. Otherwise the return path, source IP, mTLS identity, and firewall policy may differ from the intended internal path.

Troubleshooting Flow

1. Compare same-node and cross-node Pod-to-Pod traffic.
2. Check Pod IPs, node IPs, and Pod CIDRs.
3. Inspect CNI Pods and node agent health.
4. Check Pod routes and interface MTU.
5. Check host routes, tunnel interfaces, and encapsulation.
6. Check cloud routes, security groups, or firewall policy outside Kubernetes.
7. Check whether kube-proxy replacement or eBPF mode changes Service debugging.
8. Check whether DNS answers select ClusterIP, private endpoint, public load balancer, or NAT egress paths.

Study Cards

References

• Kubernetes cluster networking
• Kubernetes networking model
• Container Network Interface specification
• Debug Services