Linux System Call Debugging

System calls are where application intent crosses into the kernel. When logs say “permission denied,” “connection refused,” “timeout,” or “file not found,” the syscall layer can prove what path, socket, credentials, flags, and error code were involved.

Use syscall debugging when application logs are vague, code-level tracing is unavailable, or you need to distinguish application behavior from kernel or environment behavior.

Command Examples

strace -f -p <pid>
strace -ff -o /tmp/trace.log <command>
strace -f -e trace=file,network -p <pid>
strace -ttT -f -p <pid>
strace -yy -s 256 -f -e trace=network,desc -p <pid>
strace -c -f <command>
strace -f -e status=failed <command>
cat /proc/<pid>/syscall

Example output and meaning:

strace uses ptrace, so it can perturb timing and may be blocked by permissions, seccomp, containers, or kernel.yama.ptrace_scope.

What strace Proves

strace proves what a traced process asked the kernel to do and what the kernel returned. It does not prove what the application intended, what remote systems saw, or why userspace chose a call. Use it to answer precise boundary questions:

Do not treat a missing syscall as proof that nothing happened. The event may be in another process, thread, namespace, library cache, kernel path, or already completed before the attach.

Reading strace Output

A line has timestamp, process ID when requested, syscall name, arguments, return value, errno, and optional duration:

12:00:03.451234 connect(7<TCP:[10.0.0.5:45122->198.51.100.10:443]>, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("198.51.100.10")}, 16) = -1 EINPROGRESS (Operation now in progress) <0.000041>
12:00:03.452119 getsockopt(7<TCP:[10.0.0.5:45122->198.51.100.10:443]>, SOL_SOCKET, SO_ERROR, [0], [4]) = 0 <0.000012>

Key flags for readable evidence:

Start broad only long enough to learn the shape, then narrow the trace.

Annotated strace Output Gallery

Errno Is Evidence

Linux syscalls usually return -1 and set errno when they fail. The exact error matters.

Do not collapse these into “the app failed.” Each points at a different layer.

File and Process Syscalls

Common file path:

openat -> read/write -> fsync/fdatasync -> close

Common process path:

clone/fork -> execve -> wait4

Useful filters:

strace -f -e trace=openat,stat,access,read,write,fsync,close -p <pid>
strace -f -e trace=process -p <pid>
strace -f -e fault=openat:error=ENOENT:when=1 <command>
strace -f -P /etc/myapp/config.yaml -e trace=file <command>

Fault injection is useful in labs, not production.

File failures are often directory failures. openat("/srv/app/config.yaml", ...) = -1 EACCES may mean the file mode is wrong, but it can also mean one parent directory lacks execute permission, the mount is read-only, an LSM denied access, or the process is in a different mount namespace. Confirm the namespace and path:

readlink /proc/<pid>/root
readlink /proc/<pid>/cwd
namei -l /srv/app/config.yaml
grep ' /srv ' /proc/<pid>/mountinfo

Socket Syscalls

TCP client setup often looks like:

socket -> connect -> getsockopt -> sendto/write -> recvfrom/read -> close

Server setup often looks like:

socket -> setsockopt -> bind -> listen -> accept4 -> read/write

Useful filters:

strace -f -e trace=network -p <pid>
strace -f -e trace=connect,accept4,sendto,recvfrom,getsockopt -p <pid>
strace -ttT -yy -s 256 -f -e trace=network,poll,epoll_wait -p <pid>
ss -tulpen

connect() returning EINPROGRESS can be normal for nonblocking sockets. The final result may appear later through poll, epoll_wait, or getsockopt(SO_ERROR).

Network interpretation:

DNS Through strace

strace is useful when a program says “DNS failed” but you need to know whether it asked libc, read /etc/hosts, queried a resolver, or used a runtime cache. Trace both file and network calls:

strace -ttT -f -s 256 -e trace=file,network getent hosts api.example.com
strace -ttT -f -s 256 -e trace=openat,connect,sendto,recvfrom,poll getent ahosts api.example.com
strace -ttT -f -s 256 -e trace=file,network curl -v https://api.example.com/health

Look for:

Pair this with DNS tools: getent proves the libc view, nslookup and dig prove direct DNS query behavior, and packet captures prove whether queries left the host or Pod.

Process Trees and Services

Services often fork, exec helpers, reload config in a child, or run under a supervisor. Use -f and -ff when startup behavior matters:

strace -ff -ttT -s 256 -o /tmp/myapp.trace systemctl start myapp.service
strace -ff -ttT -s 256 -e trace=process,file -o /tmp/myapp.exec /usr/bin/myapp --check-config
ls -1 /tmp/myapp.trace.*

For a systemd service, tracing systemctl usually traces the client, not the service workload after systemd starts it. Prefer a foreground check command when available, or attach to the service process after startup:

systemctl show myapp.service -p MainPID
strace -ttT -f -p "$(systemctl show myapp.service -p MainPID --value)"

Blocking Calls

When a process looks hung, find what it is blocked on:

cat /proc/<pid>/syscall
cat /proc/<pid>/stack
strace -ttT -p <pid>

Common blockers:

High time in futex is often application lock contention, not a kernel bug.

Common Troubleshooting Recipes

Use timeout around exploratory traces when attaching during incidents:

timeout 20s strace -ttT -yy -s 256 -f -e trace=network,poll,epoll_wait -p <pid>

Containers and Permissions

Attaching to a process may fail even as root inside a container. Common causes are missing CAP_SYS_PTRACE, seccomp blocking ptrace, Yama ptrace_scope, PID namespace mismatch, or tracing from the wrong host. Check the boundary before assuming the process is special:

cat /proc/sys/kernel/yama/ptrace_scope
grep -E 'CapEff|Seccomp|NoNewPrivs' /proc/<pid>/status
readlink /proc/<pid>/ns/pid
readlink /proc/self/ns/pid

In Kubernetes, use an ephemeral debug container or node-level debug workflow only when policy permits it. Tracing production workloads can expose secrets in syscall arguments, environment reads, headers, SQL, and file paths.

Production Boundaries

• Use filters before tracing high-throughput processes.
• Prefer -c summaries for broad syscall counts.
• Use -ttT when latency per syscall matters.
• Use -yy and -s when file descriptors or strings are the evidence.
• Avoid attaching to latency-critical processes for long periods.
• Capture exact timestamps and command lines.
• Treat trace files as sensitive incident artifacts.
• Consider eBPF tracing when ptrace overhead or process attachment is too risky.

Runbook

1. Identify the process, namespace, user, and container boundary.
2. Reproduce with a narrow command if possible.
3. Trace only relevant syscall classes: file, network, process, signal, or descriptor.
4. Map errno to permissions, path, network, resource limit, or policy.
5. If blocked, inspect /proc/<pid>/syscall, strace -ttT, and thread state.
6. Confirm the finding with a non-strace command such as ss, ls -l, getfacl, or journalctl.
7. If the issue involves DNS, compare strace evidence with getent, nslookup, dig, and packet capture.

Study Cards

References

• strace documentation
• syscalls(2)
• errno(3)
• epoll(7)