Linux eBPF and Tracing

eBPF lets Linux run verified programs inside kernel-controlled hooks. For operators, it is most useful as a targeted observability tool: trace syscalls, block I/O, TCP events, scheduler latency, file opens, DNS lookups, or application functions without rebuilding the kernel or the application.

Use eBPF when normal counters show where to look but not why the behavior is happening. Use logs, metrics, strace, perf, and packet capture first when they already answer the question with less risk.

Command Examples

uname -r
bpftool feature probe
bpftool prog show
bpftool map show
bpftrace -l 'tracepoint:syscalls:sys_enter_*' | head

Example output and meaning:

Confirm kernel support, available helpers, loaded programs, and traceable events before writing a probe. Production systems may restrict BPF through capabilities, lockdown mode, LSM policy, containers, or managed-node policy.

Mental Model

An eBPF program is loaded by user space, verified by the kernel, attached to a hook, and executed when that hook fires.

This model matters because most failures are attach failures, verifier failures, missing privileges, wrong hook names, or probes that add too much overhead.

Hook Types

Prefer tracepoints for durable study examples. Reach for kprobes only when no tracepoint exposes the state you need.

bpftrace Examples

Symptom-driven one-liners:

Treat one-liners as triage, not final proof. Validate field names against your kernel with bpftrace -lv <probe> because tracepoint arguments can vary by kernel version.

Show process exec events:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s pid=%d exec=%s\n", comm, pid, str(args->filename)); }'

Count file opens by process name:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'

Trace TCP retransmits:

sudo bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb { @[comm] = count(); }'

Watch block I/O latency distribution:

sudo bpftrace -e '
tracepoint:block:block_rq_issue { @start[args->sector] = nsecs; }
tracepoint:block:block_rq_complete /@start[args->sector]/ {
  @usec = hist((nsecs - @start[args->sector]) / 1000);
  delete(@start[args->sector]);
}'

BCC Tools

BCC packages common scripts for fast incident work:

sudo execsnoop
sudo opensnoop
sudo tcpretrans
sudo biolatency
sudo runqlat
sudo offcputime -p <pid>

Use these before writing custom probes. They encode common attach points, map usage, and output formatting.

Choosing the Right Tool

eBPF is not a replacement for metrics. It is a way to answer precise questions once metrics point to a layer.

Safety Boundaries

• Avoid tracing every event on hot paths without filters.
• Prefer counts and histograms over printing every event.
• Test probes on staging or a low-traffic node first.
• Record kernel version, tool version, and exact command.
• Keep probe duration short during incidents.
• Remove loaded programs and pinned maps when finished.

Printing per packet, per syscall, or per scheduler event can create the incident you are trying to debug.

Runbook

1. State the question in one sentence: process, syscall, packet event, I/O path, or scheduler behavior.
2. Check whether ordinary metrics, logs, strace, perf, or packet capture already answer it.
3. Pick the most stable hook, usually a tracepoint.
4. Add filters by PID, cgroup, process name, port, or device.
5. Aggregate with counts, histograms, or top-N keys before printing raw events.
6. Run for a bounded window and save the command plus output.
7. Remove probes and verify no unexpected BPF programs or maps remain.

Study Cards

References

• Linux kernel BPF documentation
• bpftrace reference guide
• BCC reference guide
• bpftool documentation