Kubernetes Troubleshooting

Kubernetes troubleshooting is a graph walk. Start from the user’s symptom, then move through the API object, controller, Pod, node, network, storage, and external dependency that must all agree before the workload works.

Command Examples

kubectl get nodes
kubectl get pods --all-namespaces -o wide
kubectl get events --sort-by=.lastTimestamp
kubectl get deployment,statefulset,daemonset,job,cronjob --all-namespaces
kubectl get svc,endpointslice,ingress,networkpolicy --all-namespaces
kubectl get pv,pvc,storageclass --all-namespaces

Example output and meaning:

Read these from most specific to most general:

1. The object that users touched or traffic targets.
2. Its controller status and conditions.
3. The Pod status, events, logs, probes, and mounted config.
4. The node where the Pod landed.
5. Service, EndpointSlice, DNS, NetworkPolicy, ingress, and external dependency paths.

Common Paths

Pod Failure Workflow

Use the Pod as the smallest debuggable unit. A controller may create the Pod, but the Pod status tells you what Kubernetes actually tried to run.

kubectl -n <namespace> get pod <pod> -o wide
kubectl -n <namespace> describe pod <pod>
kubectl -n <namespace> logs <pod> --all-containers
kubectl -n <namespace> logs <pod> --all-containers --previous
kubectl -n <namespace> get pod <pod> -o jsonpath='{.status.containerStatuses[*]}'

Interpret the results:

• Waiting with image pull reasons means the container never started.
• Terminated with a nonzero exit code means the process started and exited.
• Running but not Ready usually points at readiness probes, app startup, or dependency checks.
• Repeated liveness probe failures can hide the real startup error by restarting the container before it can finish initialization.

Service and DNS Workflow

Services do not send traffic to arbitrary Pods. They select ready endpoints. Debug the object chain in order:

kubectl -n <namespace> get svc <service> -o yaml
kubectl -n <namespace> get endpointslice -l kubernetes.io/service-name=<service> -o wide
kubectl -n <namespace> get pods -l '<selector>' -o wide --show-labels
kubectl -n <namespace> exec -it <debug-pod> -- nslookup <service>.<namespace>.svc.cluster.local
kubectl -n <namespace> exec -it <debug-pod> -- nc -vz <service> <port>

If DNS resolves but connections fail, move to Service ports, EndpointSlices, kube-proxy or CNI dataplane, NetworkPolicy, and application listen ports. If DNS does not resolve, inspect CoreDNS, the Pod’s /etc/resolv.conf, namespace, search path, and service name.

Rollout and Controller Workflow

Controllers reconcile desired state into Pods. When a rollout is stuck, compare desired, current, available, and observed generation.

kubectl -n <namespace> rollout status deployment/<deployment>
kubectl -n <namespace> describe deployment <deployment>
kubectl -n <namespace> get rs -l app=<app> -o wide
kubectl -n <namespace> get events --sort-by=.lastTimestamp
kubectl -n <namespace> rollout history deployment/<deployment>

A Deployment that cannot progress usually has a Pod-level reason. Do not tune rollout settings until you know why the new ReplicaSet cannot produce ready Pods.

Node and Cluster Workflow

When many unrelated Pods fail on one node, inspect node health before debugging each workload.

kubectl describe node <node>
kubectl get node <node> -o jsonpath='{.status.conditions}'
kubectl top node <node>
kubectl get pods --all-namespaces --field-selector spec.nodeName=<node> -o wide

Node conditions such as MemoryPressure, DiskPressure, PIDPressure, and Ready=False explain scheduling and eviction behavior. On the node, kubelet, container runtime, CNI plugin, disk, and certificate state are the usual next checks.

Node pressure decision tree:

flowchart TD
  Symptom[Pods evicted or node NotReady] --> Conditions[kubectl describe node conditions]
  Conditions --> Memory[MemoryPressure]
  Conditions --> Disk[DiskPressure]
  Conditions --> PID[PIDPressure]
  Conditions --> Ready[Ready=False]
  Memory --> MemChecks[PSI, OOM events, top RSS, cgroup memory]
  Disk --> DiskChecks[imagefs/nodefs usage, logs, emptyDir, inode pressure]
  PID --> PIDChecks[pid limits, fork storms, process counts]
  Ready --> NodeChecks[kubelet, runtime, CNI, certificates, network]

Treat node pressure as a scheduling and eviction problem first. Deleting Pods without fixing the pressure source usually recreates the same failure on the same or another node.

Image Pull Failure Workflow

Image pulls cross registry naming, credentials, DNS, TLS, network policy, runtime, and node disk state.

kubectl -n <namespace> describe pod <pod>
kubectl -n <namespace> get secret <pull-secret> -o yaml
kubectl get node <node> -o wide
kubectl debug node/<node> -it --image=nicolaka/netshoot
nslookup <registry-host>
curl -vk https://<registry-host>/v2/

Evidence Capture

For incidents, capture state before deleting Pods or restarting components:

ns=<namespace>
app=<app-label>
mkdir -p k8s-capture
kubectl -n "$ns" get deploy,rs,pod,svc,endpointslice,networkpolicy,pvc -l app="$app" -o yaml > k8s-capture/objects.yaml
kubectl -n "$ns" get events --sort-by=.lastTimestamp > k8s-capture/events.txt
kubectl -n "$ns" describe pod -l app="$app" > k8s-capture/pods.describe.txt
kubectl -n "$ns" logs -l app="$app" --all-containers --prefix --tail=300 > k8s-capture/logs.txt

Study Cards

References

• Kubernetes troubleshooting applications
• Kubernetes debugging services
• Kubernetes debugging running Pods