Observability starts with what the kernel exposes. The traditional ring buffer holds boot and runtime messages that dmesg can dump (dmesg -T for human timestamps, dmesg -w to follow new entries, and dmesg -l err to filter by severity emerg→debug). On systemd hosts the same data appears under journalctl -k; if persistence is desired, /var/log/dmesg plus the systemd journal in /var/log/journal cover most needs. The /proc filesystem surfaces CPU details (/proc/cpuinfo), memory (/proc/meminfo), load averages (/proc/loadavg), and a per-process directory at /proc/[PID]. The /sys virtual filesystem (sysfs) exposes kernel objects — devices, drivers, buses — and modern tools like lsblk and ip read it directly. Writing to certain paths under /sys/class/ alters kernel state, though the kernel provides no undo, so changes should be deliberate. Persistent logs are managed with logrotate: per-service drop-ins in /etc/logrotate.d/ specify directives such as daily, rotate 14, compress, missingok, and a postrotate block to signal services (e.g., kill -USR1 \$(cat /var/run/nginx.pid)). The logrotate -d flag runs in debug mode without applying changes; logrotate -f forces execution; the daily cron job lives at /etc/cron.daily/logrotate.
CPU and memory pressure reveal themselves through uptime, top, and /proc/loadavg, which report load averages over 1, 5, and 15 minutes. On a system with \(N\) CPU cores, a load of \(N\) means full utilization and a load above \(N\) means a queue is building; uninterruptible (state D) tasks — usually blocked on I/O — also count toward load, so a rising load with stable CPU usage typically signals disk or memory pressure rather than CPU saturation. Three tools from the sysstat package quantify this: vmstat 1 5 samples virtual memory, CPU, and I/O (showing runnable r, blocked b, swap in/out si/so, and CPU time split between user, system, idle, wait, and steal); iostat -xz 1 tallies per-device disk activity; mpstat -P ALL 1 reports per-CPU statistics. The systemd-cgtop command gives a top-like view of control-group resource usage.
When a process misbehaves, deeper introspection arrives through tracers and profilers. strace -p <pid> attaches to a running process and prints every system call it makes — invaluable for "why is this hanging?" — while strace -f -o trace.log cmd traces a new command and follows forks, -e openat,read,write filters calls, -T measures each call's duration, and -c summarizes counts. ltrace does the same for library calls; perf trace provides higher-throughput equivalents. perf itself samples performance counters: perf stat cmd reports aggregate counters, perf record -F 99 -a -g cmd samples at 99 Hz with a call graph for later perf report analysis, perf top provides a live top-like view, and perf record -b base; perf record -b patched; perf diff compares two builds. pidstat -u 1 and pidstat -d 1 produce per-process CPU and I/O statistics, iotop ranks processes by disk activity, nethogs attributes bandwidth to processes, and atop records historical system state for retrospective analysis.
Runtime kernel parameters are managed through sysctl, which reads and writes /proc/sys/. sysctl -a lists every parameter, sysctl net.ipv4.ip_forward reads one, and sysctl -w net.ipv4.ip_forward=1 writes it on the fly. Persistent changes belong in files under /etc/sysctl.d/, applied by sysctl --system or sysctl -p /etc/sysctl.conf; common tunings include net.ipv4.tcp_tw_reuse=1, vm.swappiness=10, and fs.file-max=100000. Accurate time is equally foundational: hwclock -r reads the hardware clock, timedatectl shows the current state, timedatectl set-ntp true enables systemd-timesyncd, and chronyc tracking / chronyc sources reveal the chrony client's view; chrony is generally preferred on virtual machines and laptops with intermittent connectivity.
Cgroups and namespaces together form the substrate of modern containers. Cgroups (control groups) organize processes into hierarchies and impose resource limits — systemd creates one per service at /sys/fs/cgroup/systemd.slice/<name>.service/, and unit files can declare CPUQuota=50%, MemoryMax=512M, IOWeight=500, and TasksMax=100; cgroups v2 (the unified hierarchy) is the default on modern kernels. Namespaces provide isolation across several dimensions: pid for process IDs, net for the network stack, mount for mount points, uts for hostname, ipc for inter-process communication, user for UID mapping, and cgroup for cgroup visibility. Each container receives a private PID, network, mount, and user namespace plus cgroup-enforced CPU and memory limits. Operators can experiment by hand: unshare --pid --fork bash opens a new PID namespace in a shell, while nsenter -t 1 -m -u -i -n -p bash enters existing namespaces — useful for troubleshooting a process that has detached itself. Container runtimes such as runc, crun, and containerd orchestrate these primitives; inspect any running container with cat /proc/1/cgroup and ls -l /proc/1/ns/.