Troubleshooting and Error Handling

Troubleshooting is evidence management under time pressure. Error handling is designing systems so failures are bounded, observable, retryable when safe, and recoverable when not. The same discipline applies across Linux hosts, networks, DNS, Kubernetes, Istio, Ceph, PostgreSQL, and application services.

For a repeatable incident evidence capture script, see Troubleshooting Examples.

Command Examples

date -Is
hostnamectl
systemctl --failed
journalctl -p warning..alert -b
kubectl get events --all-namespaces --sort-by=.lastTimestamp
curl -v https://example.com/
dig example.com

Example output and meaning:

This first pass answers four questions: what time is it on this system, what host or cluster am I on, what is already reporting failure, and does the symptom reproduce at DNS, TCP, TLS, HTTP, or service level?

Incident Frame

Start by writing a short failure statement before changing anything:

Good troubleshooting reduces possibilities. Bad troubleshooting creates new state faster than evidence can explain it.

Universal Method

Use the same loop across Linux, networking, DNS, Kubernetes, databases, storage, and identity:

1. State the exact symptom and scope.
2. Preserve evidence before restarting, deleting, failing over, or repairing.
3. Identify the data plane and control plane involved.
4. Find the owner of the next state transition.
5. Test one layer at a time with the same path the application uses.
6. Compare known-good and known-bad paths.
7. Make the smallest reversible change that tests a hypothesis.
8. Verify recovery from user-visible behavior and lower-level evidence.
9. Record what changed, why it worked, and what guardrail prevents recurrence.

Avoid shotgun debugging: multiple simultaneous changes make it hard to know which change helped and can create a second incident.

Evidence Preservation

Before restarting services or deleting pods, capture the evidence that will disappear. Treat evidence preservation as an operational requirement, not a post-incident luxury:

• logs for the failing time window,
• events and status objects,
• process exit code and signal,
• unit, deployment, and config definitions,
• resource pressure and OOM evidence,
• connection counters, queue sizes, and error counters,
• current leader, primary, quorum, or lock holder,
• recent rollout, migration, or operator reconciliation action.

Useful convention: record the command, timestamp, host or context, and output path in the incident notes. A correlation ID should be carried through logs and traces when the application supports it.

Error Taxonomy

Classify the failure before choosing the fix:

Data Plane vs Control Plane Failures

Separating planes prevents misleading conclusions:

A control plane can be down while current data-plane traffic keeps flowing. A data plane can be broken while APIs show the desired configuration. Check both before deciding the blast radius.

Linux Host Runbook

Linux failures usually surface as resource pressure, service failure, kernel messages, permission problems, or filesystem errors.

Checks:

systemctl status <unit>
journalctl -u <unit> -b --no-pager
systemctl show <unit> -p ExecMainStatus -p Result -p OOMPolicy
dmesg -T | tail -100
cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
df -hT

Interpretation:

Safe handling: restart only after preserving logs and checking whether the service is crash-looping because of bad config, missing dependencies, or resource limits.

Networking and DNS Runbook

Separate name resolution, route selection, connection establishment, TLS, and application response.

dig example.com
getent hosts example.com
ip route get 198.51.100.10
ss -tan state syn-sent
curl -vk --resolve example.com:443:198.51.100.10 https://example.com/
tcpdump -nn -i any host 198.51.100.10

Common traps:

• DNS search paths and ndots create unexpected queries.
• Firewalls may allow ICMP while dropping TCP or UDP.
• MTU black holes often appear only on large TLS or file-transfer traffic.
• A proxy can change DNS view, source IP, Host, SNI, headers, and timeout behavior.
• A VPN can route one prefix correctly while dependency prefixes bypass the tunnel.

Error handling: clients should use explicit connect, TLS handshake, request, and idle timeouts. Retries need backoff with jitter and should be limited to operations that are idempotent or protected by an idempotency key.

Kubernetes Runbook

Kubernetes errors are usually split between desired state, scheduler decisions, kubelet execution, network/storage attachment, and application behavior.

kubectl get pods -A -o wide
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl logs <pod> -n <namespace> --previous
kubectl get deploy,rs,sts,ds -n <namespace>
kubectl auth can-i --as <service-account> <verb> <resource>

High-value states:

Error handling: readiness should mean “can serve this traffic now.” Liveness should detect unrecoverable local deadlock, not slow dependencies. Startup probes protect slow boot from premature liveness kills.

Istio and Service Mesh Runbook

Service mesh failures often look like application failures, but the failing layer may be proxy config, mTLS, authorization, route matching, endpoint discovery, or xDS sync.

istioctl proxy-status
istioctl proxy-config routes <pod> -n <namespace>
istioctl proxy-config clusters <pod> -n <namespace>
kubectl get peerauthentication,authorizationpolicy -A
kubectl logs <pod> -c istio-proxy -n <namespace>

Check:

• sidecar or ambient enrollment,
• proxy readiness and xDS sync,
• PeerAuthentication mode and DestinationRule TLS mode,
• AuthorizationPolicy deny and allow rules,
• VirtualService host and route matches,
• waypoint proxy or ztunnel logs in ambient mode,
• whether direct pod traffic differs from mesh traffic.

Error handling: timeouts, retries, and circuit breaker settings in the mesh must match application semantics. Retrying non-idempotent writes at a proxy can duplicate effects.

Ceph and Rook Runbook

Ceph troubleshooting starts with health detail and PG states, not with restarting daemons.

ceph -s
ceph health detail
ceph osd tree
ceph pg stat
ceph pg dump_stuck
kubectl -n rook-ceph get cephcluster,pods,pvc

Important states:

Error handling: avoid repair commands until backups, affected PGs, and device health are understood. In Rook, separate Kubernetes scheduling/PVC failures from Ceph health failures.

PostgreSQL and CloudNativePG Runbook

Database incidents need caution because a “fix” can destroy evidence or data.

psql -d <database> -c "select now(), state, wait_event_type, wait_event from pg_stat_activity;"
psql -d <database> -c "select * from pg_stat_replication;"
psql -d <database> -c "select * from pg_stat_database_conflicts;"
kubectl cnpg status <cluster>
kubectl get pods,pvc,svc -l cnpg.io/cluster=<cluster>

PostgreSQL clues:

Error handling: wrap external side effects with idempotency keys or an outbox pattern. Use transaction boundaries deliberately. Retry serialization failures and deadlocks where safe; do not blindly retry constraint violations or unknown commit outcomes.

Application Error Handling Patterns

Reliable systems assume partial failure.

Do not retry everything. Retry only when the operation is safe, bounded, observable, and the caller can tolerate the extra latency.

Recovery Checklist

1. Preserve evidence before destructive action.
2. State the blast radius and current user impact.
3. Stop the bleeding with the smallest reversible change.
4. Verify the fix at the failing layer and one layer above it.
5. Watch for delayed errors: retries, queues, backfills, WAL replay, cache expiry.
6. Document root cause, contributing factors, detection gap, and prevention.
7. Add a regression check, alert, runbook step, or safer default.

Study Cards

References

• Linux logs and observability
• Linux systemd troubleshooting
• Networking packet path
• DNS records, responses, and transport
• Kubernetes troubleshooting
• Istio service mesh
• Ceph health checks
• PostgreSQL error codes