Linux Administration

This deck covers the core command-line skills and system concepts that form the foundation of Linux administration. You'll find questions on everyday commands like ls, grep, find, awk, and sed, as well as important background topics such as the filesystem hierarchy, the /proc virtual filesystem, and how file permissions and ownership work. There are also cards on process management, covering tools like ps, top, kill, nice, and job control, which are essential for keeping a Linux system running smoothly.

It's well suited to anyone preparing for a sysadmin role, a Linux certification, or simply wanting to become more confident on the terminal. Beginners who have just installed their first Linux distribution will find a clear path through the basics, while more experienced users can use the deck to fill in gaps and reinforce less familiar areas like special permission bits or the finer points of process signals.

Because Linux administration is a hands-on subject, the cards will stick much faster if you try out each command on a real machine or virtual environment as you study. A quick tip: aim to review a small batch of cards each day rather than cramming everything at once. Spaced, repeated practice mirrors the way these tools become second nature through daily use, and you'll soon find yourself reaching for the terminal without even thinking about it.

The Linux shell becomes powerful when its small utilities are combined with pipes and redirection. The command ls lists directory contents and accepts flags such as -l for long format, -a for hidden dotfiles, -h for human-readable sizes, -R for recursion, and -t for sorting by modification time. Two other commands make searching trivial: grep filters lines by pattern with options like -i (case-insensitive), -r (recursive), -n (line numbers), -v (invert match), and -E (extended regex); find walks the directory tree, supporting filters by name glob, type, modification time, size, and permission, and can chain a custom action through -exec. Together, ls, grep, and find form the foundation of nearly every sysadmin investigation.

Two classic stream processors handle structured text. sed is a line-oriented editor usually invoked for substitution with s/old/new/, with g for global replacement on each line, -i for in-place editing, -n '/5,10p' for printing selected ranges, and /pattern/d for deleting matching lines. awk treats input as records made of fields: awk '{print \$1, \$3}' file prints columns 1 and 3, -F: customizes the delimiter (useful for /etc/passwd), and the built-in variables NR, NF, and FS expose record number, field count, and field separator. Combining sed and awk with other small tools — head, tail (with -f for live following), cat, tee, sort, uniq -c, wc -l, cut -d: -f1, tr for character translation, and xargs for building commands from streams — turns short pipelines like cat access.log | awk '{print \$1}' | sort | uniq -c | sort -rn into full-featured reporting tools.

The plumbing between these commands is redirection and pipes. The pipe | sends stdout of one process into stdin of the next, while > and >> redirect stdout (overwrite or append), < feeds a file into stdin, 2> and 2>&1 capture stderr, and &> covers both streams. The special file /dev/null, sometimes called the bit bucket, discards anything written to it and returns end-of-file when read; redirecting both streams there (command > /dev/null 2>&1) is the idiomatic way to silence a noisy command or to test success silently inside an if statement. Archiving files into a single transportable bundle is the job of tar: tar -czf archive.tar.gz dir/ creates a gzip-compressed archive, -xzf extracts one (optionally into a target via -C), and -tzf lists contents, with c, x, t, z, j, f, and v as the most common flags for create, extract, list, gzip, bzip2, file, and verbose, respectively.

Files can also be referenced from multiple locations through links. A hard link created with ln file hard points to the same inode as the original, so both names refer to identical on-disk data and the file survives deletion of either name; hard links, however, cannot span filesystems or link to directories. A symbolic link created with ln -s file sym is a small file holding a path string — a "shortcut" — which can cross filesystems and point to directories but breaks if its target is removed. ls -li shows inode numbers, confirming whether two names share storage; stat file exposes inode, link count, and metadata directly.

Linux organizes the entire system under a single hierarchical tree rooted at /. A few top-level directories deserve attention: /etc holds system-wide configuration such as /etc/passwd, /etc/shadow, /etc/group, /etc/fstab, /etc/hosts, /etc/hostname, /etc/resolv.conf, and service-specific files like /etc/ssh/sshd_config; /var stores variable data — logs, spools, caches, and package-manager histories; /home contains per-user home directories; /usr contains the bulk of user-land programs and read-only data; /tmp holds short-lived temporary files and is often wiped at reboot; /proc is a virtual filesystem exposing kernel data structures such as /proc/cpuinfo, /proc/meminfo, /proc/loadavg, and the per-process directory /proc/[PID]; and /dev exposes device files representing block and character devices. Understanding these conventions is essential because nearly every administration task — finding logs, tweaking behavior, or diagnosing a misbehaving service — starts by navigating this layout.

Every file and directory carries three permission sets — owner, group, and others — each expressed as read (r = 4), write (w = 2), and execute (x = 1). The chmod command adjusts them either numerically (e.g., chmod 755 file yielding rwxr-xr-x) or symbolically with letters u/g/o/a for who, +/-/= for the action, and r/w/x for the right, so chmod u+x script.sh makes a script runnable by its owner. The default permissions for newly created files and directories come from umask, which subtracts bits from the maximum (666 for files, 777 for directories); an umask of 022 produces 644 on new files and 755 on new directories, while 077 tightens that to 600 and 700. Persistence is achieved by exporting the umask in ~/.bashrc or the global /etc/profile, and viewing it with umask or umask -S for symbolic output.

Three special permission bits extend the basic model. The SUID bit, set numerically as 4xxx (e.g., chmod 4755 file), causes a program to execute with the file owner's privileges, visible in ls as an s in the owner's execute slot — this is what allows utilities like passwd to write to protected files while running as a normal user. The SGID bit (2xxx) does the same for the group and, when applied to a directory, makes new files inherit the directory's group, which simplifies collaborative folders. The sticky bit (1xxx), often seen as a t on world-writable directories such as /tmp (configured as 1777), ensures only the file's owner (or root) can rename or delete their own files inside it. Ownership itself is managed by chown, which can set user, group, or both simultaneously (chown user:group file, with a leading colon to change only the group), recursively via -R, while chgrp changes group only and is the only ownership command regular users may run on files they do not own. Only root can change the user owner of a file.

When owner/group/other granularity is not enough, POSIX ACLs add per-user and per-group entries. Enabled at mount time via mount -o acl (often the default for ext4), ACLs are inspected with getfacl and modified with setfacl: setfacl -m u:alice:rw file grants alice read and write, -m g:dev:rwx dir/ grants the group full directory access, -d -m u:alice:r dir/ sets a default ACL inherited by newly created files inside, while -x removes a specific entry and -b clears the entire ACL. As soon as any ACL is in place, ls -l appends a + after the mode string to flag it. The chmod symbolic notation is also worth restating: combinations like chmod go-w file (remove write from group and others) or chmod a=r file (read-only for everyone) cover most recurring cases without arithmetic.

Every running program on Linux is a process identified by a PID, and operators routinely need to inspect, signal, and reorder them. The ps command produces a static snapshot: ps aux shows every process with detailed fields, ps -ef uses full-format output, and ps -u username filters by user. The interactive top (and its colorful cousin htop) refreshes continuously and offers single-keystroke actions — k to kill a process, M to sort by memory, P by CPU, and q to quit. To influence a running process, kill sends signals by PID: the default and most graceful is SIGTERM (15); SIGKILL (9) forces immediate termination but cannot be caught or cleaned up; SIGHUP (1) prompts many daemons to reload configuration; SIGSTOP (19) pauses and SIGCONT (18) resumes; kill -l lists every signal number and name. Process priority is adjusted through nice, whose scale runs from -20 (highest priority) to 19 (lowest) — a regular user can only increase niceness — and renice changes it for an existing PID, while nohup shields a command from SIGHUP so that it survives the closing of its controlling terminal, redirecting its output to nohup.out by default.

Job control adds a session-local layer on top of system processes. Appending & launches a command in the background, and Ctrl+Z suspends the current foreground job. The jobs command enumerates shells jobs with their numbers and states (running, stopped, or done), bg %1 resumes job 1 in the background, fg %1 pulls it back into the foreground, and disown %1 detaches it from the shell so it survives logout. PID 1 holds a special place: it is the init process started by the kernel and becomes the ancestor of every other userspace process, automatically adopting any orphaned children (and reaping their zombies). On modern distributions PID 1 is systemd, which is unkillable even by kill -9 — to replace it you must re-execute the binary or reboot. Zombies themselves, marked Z in ps state, only consume a slot in the process table; cleanup requires the parent to call wait() or to be sent SIGCHLD, or, if the parent is gone, for PID 1 to reap them automatically.

systemd has effectively replaced the older SysV init and supersedes ad-hoc scripts. Its primary management tool is systemctl: start, stop, restart, enable (start at boot), and status act on units, while list-units --type=service enumerates them. A unit file lives in /etc/systemd/system/ (or /lib/systemd/system/) and is divided into three sections: [Unit] carries a human-readable Description and ordering hints like After=network.target; [Service] specifies how the daemon runs, including ExecStart, an optional restart policy (Restart=always), and the user to drop privileges to (User=www-data); [Install] integrates with the boot sequence via WantedBy=multi-user.target. After any edit, systemctl daemon-reload picks up the changes. Targets are systemd's replacement for SysV runlevels — poweroff.target, rescue.target (single user), multi-user.target (no GUI), graphical.target, and reboot.target — and systemctl get-default plus set-default change the boot state.

Logs and scheduling round out the systemd toolkit. journalctl queries the systemd journal: -u nginx filters by unit, -f follows new entries (like tail -f), --since "1 hour ago" filters by time, -p err restricts the priority, and -b shows only messages from the current boot. The journal lives in volatile memory by default; making it persistent requires creating /var/log/journal and restarting systemd-journald, after which retention can be controlled with SystemMaxUse= and MaxRetentionSec= in /etc/systemd/journald.conf. For recurring tasks, systemd timer units (*.timer) can replace cron. A timer pairs with a service whose ExecStart runs the actual work; the timer's [Timer] section uses OnCalendar=*-*-* 02:00:00 for absolute times and Persistent=true to catch up after downtime, with commands like systemctl list-timers and enable managing activation. When cron is still preferred, crontab -e edits a per-user schedule whose columns encode minute, hour, day of month, month, and day of week — for example, 0 2 * * * /backup.sh runs nightly at 02:00, while */15 * * * * /check.sh fires every fifteen minutes, and the @reboot shortcut executes once at startup. For one-shot future work, at queues a job specified inline (e.g., at 14:30 followed by commands and Ctrl+D), with atq listing pending jobs, atrm removing them, and the atd service managing execution.

User and group administration underpins access control. New accounts are created with useradd -m -s /bin/bash john (the -m flag provisions a home directory), passwords set with passwd john, group memberships extended with usermod -aG sudo john, and accounts removed with userdel -r john (which deletes the home directory too). The commands id john and whoami reveal UID, GID, group memberships, and the current user. Information about accounts lives in /etc/passwd, encrypted passwords in /etc/shadow, and groups in /etc/group (with /etc/gshadow storing group passwords). Each user has exactly one primary group and may belong to many supplementary groups; groupadd, groupdel, usermod -aG, gpasswd -d, and groups manipulate these.

Privilege escalation is mediated by sudo, configured through /etc/sudoers — always edited via visudo, which validates the syntax. Entries follow user host=(runas) commands: root ALL=(ALL:ALL) ALL is the default, %sudo ALL=(ALL:ALL) ALL grants every member of the sudo group full access, and alice ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart nginx lets alice restart nginx without re-typing her password. Customizations are best kept in drop-in files inside /etc/sudoers.d/. Defaults like Defaults env_reset, timestamp_timeout=5, and log_input, log_output shape caching and auditing, while audits live in the journal (journalctl -u sudo) or /var/log/auth.log. Beyond the root-or-not binary, Linux capabilities split superuser privileges into roughly forty discrete powers (CAP_NET_BIND_SERVICE, CAP_SYS_ADMIN, CAP_DAC_OVERRIDE, and so on) that can be granted to a binary with setcap cap_net_bind_service=+ep /usr/bin/python3, so an unprivileged program can bind port 80 without running as root. getcap and getpcaps inspect them.

Mandatory access control adds an extra layer on top of standard Unix permissions. SELinux, dominant on RHEL-family systems, assigns every process and file a security context and enforces type-enforcement rules: it can run in enforcing (deny by default), permissive (log only), or disabled. Status is checked with getenforce and sestatus; modes are switched with setenforce 0|1 and made persistent in /etc/selinux/config. File contexts appear as -Z on ls and ps, are changed with chcon -t httpd_sys_content_t /var/www/html/file, and reset to the policy default with restorecon -R. Denial analysis uses sealert -a /var/log/audit/audit.log, while audit2why and audit2allow generate policy from observed violations. AppArmor, default on Ubuntu, is path-based and simpler: profiles live in /etc/apparmor.d/, with commands like aa-status, aa-complain, aa-enforce, and apparmor_parser -r managing enforcement.

The boot sequence ties everything together. After POST, the firmware (BIOS or UEFI) hands off to the bootloader — GRUB reads /boot/grub/grub.cfg on legacy BIOS or /boot/efi/EFI/... on UEFI, presenting a menu where e edits a kernel line and c opens a GRUB shell. The kernel loads an initramfs (a compressed cpio archive in /boot) that contains just enough modules to mount the real root, assemble software RAID, activate LVM, or unlock LUKS, then hands control to PID 1 — on modern systems, systemd — which resolves target dependencies, mounts filesystems from /etc/fstab, brings up networking, and starts services before getty spawns login prompts. The initramfs can be regenerated with update-initramfs -u on Debian/Ubuntu or dracut -f on RHEL, and an installed image can be listed with lsinitramfs. When the system fails to reach multi-user.target, systemctl --failed highlights the broken unit, journalctl -xb -p err shows error-level boot logs, and appending systemd.unit=rescue.target or systemd.unit=emergency.target at the GRUB prompt drops to progressively more minimal shells for repair. GRUB itself can be reinstalled from a Live USB by chrooting into the mounted root and running grub-install /dev/sda followed by grub-mkconfig -o /boot/grub/grub.cfg.

Modern Linux configures networking through the ip suite, which supersedes the older ifconfig and route. ip addr show lists addresses and their interfaces, ip link show enumerates link-layer state, ip route show prints the routing table, ip addr add 10.0.0.5/24 dev eth0 attaches an address, ip link set eth0 up activates an interface, and ip neigh show exposes the ARP cache. Socket statistics are read with ss rather than the legacy netstat: ss -tuln shows TCP and UDP listeners with numeric ports, ss -tp adds owning processes, and ss -s prints a summary. To locate the process behind a specific port, ss -tulnp 'sport = :80', lsof -i :80, or fuser 80/tcp all work, with ss reading kernel data via netlink and being the fastest of the three.

For diagnostics, three small utilities cover most needs. ping -c 4 host measures reachability and round-trip latency, traceroute host lists the routers hops traverse on the way to the destination, and dig example.com queries DNS — dig +short condenses the answer, dig @8.8.8.8 example.com targets a specific resolver, and dig example.com MX selects mail records, while nslookup remains a simpler interactive alternative. DNS resolution order is dictated by /etc/nsswitch.conf (typically hosts: files dns), and configuration is recorded in /etc/resolv.conf with nameserver, search, and options lines. On systemd-equipped hosts, systemd-resolved often intercepts these calls: it listens on 127.0.0.53, may stub /etc/resolv.conf with a symlink, and provides resolvectl status, resolvectl query, and resolvectl dns eth0 8.8.8.8 for management. Split DNS is supported through the Domains=~internal setting in /etc/systemd/resolved.conf, which routes *.internal queries to a chosen upstream. Bridges and VLANs layer atop ip link: ip link add br0 type bridge builds a virtual switch, ip link set eth0 master br0 attaches an interface, and ip link add link eth0 name eth0.10 type vlan id 10 creates an 802.1Q-tagged subinterface — a building block when configuring VMs or containers. To turn a host into a router, enable net.ipv4.ip_forward=1 via sysctl, assign an interface per subnet, and add a masquerading NAT rule with nftables.

SSH is the de facto remote administration tool. A connection starts with ssh user@hostname (with -p 2222 if the daemon listens on a non-default port), but the recommended authentication path is public-key cryptography: ssh-keygen -t ed25519 generates a key pair, ssh-copy-id user@host installs the public key into the remote ~/.ssh/authorized_keys, and an entry in ~/.ssh/config can alias the host with its hostname, user, and identity file. The ssh-agent holds decrypted private keys in memory after ssh-add, and ssh -A forwards that agent, though forwarding to an untrusted bastion is risky — the safer alternative is the modern ProxyJump form ssh -J jumphost target (also writable in ~/.ssh/config). Common tunneling patterns include local port forwarding (ssh -L 8080:internal:80 bastion), remote forwarding (ssh -R 8080:localhost:80 bastion), and dynamic SOCKS proxies (ssh -D 1080 host). Host keys, generated at sshd startup and stored under /etc/ssh/ in ssh_host_ed25519_key, ssh_host_ecdsa_key, and ssh_host_rsa_key files, identify the server to the client; a change triggers a man-in-the-middle warning, cleared with ssh-keygen -R hostname on the client.

Hardening sshd begins in /etc/ssh/sshd_config and finishes with systemctl restart sshd. PermitRootLogin no, PasswordAuthentication no, PubkeyAuthentication yes, AllowUsers alice bob or AllowGroups sshusers, MaxAuthTries 3, LoginGraceTime 30, and a non-default Port 2222 are typical. Idle sessions can be cut off with ClientAliveInterval 300 and ClientAliveCountMax 2; AllowTcpForwarding no and X11Forwarding no close off features that aren't needed; sshd -t validates the configuration safely. To protect against brute force, fail2ban scans logs (typically /var/log/auth.log or /var/log/nginx/error.log) and bans offending IPs at the firewall; configuration lives in /etc/fail2ban/jail.local, with enabled = true, maxretry = 3, findtime = 600, and bantime = 3600, and operations are performed through fail2ban-client status, ... status sshd, and unban <ip>.

The firewall itself is nftables on modern Linux, with two friendly front ends. nft builds a ruleset as tables and chains: nft add table inet filter plus an input chain with policy drop creates a default-deny posture, while selective accepts loopback traffic, ct state established,related connections, and chosen ports (e.g., tcp dport 22 accept); nft list ruleset > /etc/nftables.conf persists the configuration, and enabling nftables.service ensures it survives reboots. Ubuntu ships ufw as a simpler layer, with ufw allow 22/tcp, ufw default deny incoming, and ufw enable for the common cases; rules land in /etc/ufw/. RHEL provides firewalld, which organizes the firewall around zones (public, internal, dmz, trusted, block, drop) assigned to interfaces, with commands like firewall-cmd --zone=public --add-port=80/tcp --permanent and --reload for runtime change vs persistent storage. For deeper inspection, tcpdump -i eth0 -nn -w cap.pcap captures raw packets — viewable in Wireshark — and openssl s_client -connect host:443 -servername host examines TLS handshakes, which depend on certificates issued by certbot --nginx -d example.com from Let's Encrypt and stored under /etc/letsencrypt/live/.

File transfers over SSH prefer rsync over scp because rsync only sends changed blocks, supports resumes, and handles complex flags efficiently. The canonical form rsync -avz -e ssh /src/ user@host:/dst/ walks /src/, preserves permissions and timestamps, and compresses during transit; --progress shows a status line, --delete mirrors deletions from the destination, --exclude='*.tmp' filters patterns, --dry-run previews actions, and --bwlimit=5000 caps bandwidth in KB/s. --link-dest=/prev enables hard-link snapshot backups by referring unchanged files to a previous run. The trailing slash on the source matters: /src/ copies directory contents while /src copies the directory itself. By contrast, scp file user@host:/path/ (or scp -r dir/ user@host:/path/) performs straightforward secure copies without delta logic. For purely outbound downloads, curl -O URL supports a wide range of protocols and HTTP verbs (curl -X POST -d '...' URL, curl -I URL for headers), while wget -r URL specializes in recursive, resuable, mirror-style downloads (wget -c URL resumes a partial file).

Disk management starts with discovery. lsblk lists block devices in a tree (disks, partitions, and their mount points), and lsblk -f adds the filesystem type and UUID. df -h reports free space across mounted filesystems in human-readable units, df -T includes the filesystem type, and df -i exposes inode consumption. The companion du -sh /path summarizes a directory, du -h --max-depth=1 / sizes every top-level entry, and du -ah | sort -rh | head -20 hunts for the largest files; the interactive ncdu browser provides the same data with arrow-key navigation. Mounting is performed with mount /dev/sdb1 /mnt/data (optionally specifying a type via -t ext4) and undone with umount; persistent mounts belong in /etc/fstab, whose columns are device specifier, mount point, filesystem type, options, dump flag, and fsck pass (1 for root, 2 for other, 0 to skip). The device specifier is best given as UUID= (found with blkid) because it survives drive reordering, and a syntax test runs via mount -a.

Partition tables divide a disk. MBR permits four primary partitions and tops out near 2 TiB, while GPT (GUID Partition Table) supports 128 partitions, spans 8 ZiB, includes a backup header at the disk's tail, and is required for UEFI booting. The legacy fdisk (and the modern gdisk dedicated to GPT) handles both; parted offers scriptable partitioning (parted /dev/sdb mklabel gpt followed by mkpart primary ext4 0% 100%). A fresh kernel view follows partprobe, after which mkfs.ext4, mkfs.xfs, or mkfs.btrfs formats the partition. UEFI installs additionally require an ESP (EFI System Partition) — a FAT32 partition of roughly 256–512 MB flagged as esp. When problems surface, fsck /dev/sda1 checks ext-family filesystems (with the family-specific fsck.ext4 for explicitness), xfs_repair handles XFS, and btrfs scrub start /mnt verifies all data on a btrfs filesystem. A corrupted ext4 superblock can be recovered by listing alternates with mkfs.ext4 -n /dev/sda1 and then running fsck.ext4 -b 32768 /dev/sda1 against one of them.

Three mature filesystems dominate Linux. ext4 is the default on many distributions, supports extents and journaling, can be resized online, and is maintained with e4defrag and tune2fs -l. XFS, default on RHEL, is a 64-bit high-performance journaling filesystem that excels at large files and parallel I/O; it can be grown online but cannot be shrunk. btrfs brings copy-on-write, subvolumes (btrfs subvolume create), snapshots, compression, and an integrated btrfs scrub data verifier. All three can be snapshotted independently through LVM, which abstracts physical storage into a stack of physical volumes (PVs), volume groups (VGs), and logical volumes (LVs). Setup runs pvcreate /dev/sdb, vgcreate vg0 /dev/sdb, lvcreate -L 50G -n data vg0, and finally mkfs.ext4 /dev/vg0/data; later changes include lvextend -L +10G -r /dev/vg0/data (the -r flag resizes the filesystem in lock-step), vgextend/vgreduce to add or remove disks, and status checks with pvs, vgs, and lvs. RAID complements LVM for redundancy: mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc builds a mirror, RAID 5 tolerates one disk failure with single parity, RAID 6 with double parity, RAID 10 stripes mirrors for both speed and safety, and cat /proc/mdstat plus mdadm --detail /dev/md0 report status; mdadm --detail --scan >> /etc/mdadm/mdadm.conf persists the array layout.

Inodes track file metadata — permissions, timestamps, and block pointers — separate from filenames. df -i checks inode usage, an entirely independent constraint from blocks. When df -h shows free space yet new files cannot be created, inodes are exhausted, typically by millions of empty files in caches or session directories. Locate them with find / -xdev -type d -exec sh -c 'echo $(find "\$1" | wc -l) "\$1"' _ {} \; | sort -rn | head. Adjacent to inodes, two RAM-based filesystems offer alternatives to disk. tmpfs uses both RAM and swap, has a configurable size (mount -t tmpfs -o size=512m tmpfs /mnt/tmp), honors ownership, and shows up in df; common examples are /run, /tmp, and /dev/shm. ramfs has no size cap at all — it grows until memory is exhausted — so it must be used only where unbounded growth is genuinely safe.

Swap extends virtual memory onto disk and serves a second purpose on laptops: enabling suspend to disk. A swap file is created by allocating space (fallocate -l 2G /swapfile), restricting permissions (chmod 600), formatting (mkswap /swapfile), and activating (swapon /swapfile), with persistence added through /etc/fstab. swapon --show and free -h report current usage. The kernel's vm.swappiness tunable (read/written via /proc/sys/vm/swappiness, default 60) biases the kernel toward retaining pages in RAM or pushing them out aggressively; on servers with abundant memory, lower values are usually preferable. When memory is genuinely exhausted, the kernel relies on overcommit heuristics (vm.overcommit_memory=0) and, in the worst case, activates the OOM killer, which scores every process by oom_score_adj (range -1000 to 1000 in /proc/[pid]/oom_score_adj) and delivers SIGKILL to the worst offender; logs of the event surface in dmesg | grep -i oom or journalctl -k | grep -i oom.

Bash scripts glue the system's utilities together, and mastering variables, control flow, and exit handling unlocks real automation. Variables are assigned without spaces around = (e.g., NAME="Linux") and read with a leading dollar sign (\$NAME or the safer "\${NAME}"); readonly PI=3.14 creates a constant. Special positional and status variables are critical: \$0 holds the script name, \$1, \$2, ... are positional arguments, \$# counts them, \$@ expands them all, \$? exposes the last command's exit code, and \$\$ returns the current PID. Environment variables, set per session with export VAR="value" or for a single command with VAR=value command, are visible across the process tree; env and printenv list them. The most important is PATH, a colon-separated list of executable directories; append export PATH="\$PATH:/new/dir" for the current shell, persist it by editing ~/.bashrc or ~/.profile, and then run source ~/.bashrc to activate. Common built-ins like HOME, USER, SHELL, and LANG are populated automatically.

Control flow lets scripts react to data. The for loop iterates over a list (for i in 1 2 3; do echo \$i; done), over a glob (for f in *.txt; do wc -l "\$f"; done), or with a C-style counter (for ((i=0; i<10; i++)); do echo \$i; done). The while loop runs as long as a condition holds — a count loop (while [ \$count -lt 10 ]; do echo \$count; ((count++)); done) or a file consumer (while IFS= read -r line; do echo "\$line"; done < file.txt) — and conditional logic uses if/elif/else with the classic test command ([) and operators like -eq, -f file, -d dir, -z for empty string, -n for non-empty, and = for string equality; the modern [[ ]] supports advanced pattern matching. Functions defined as greet() { echo "Hello, \$1!"; return 0; } take arguments through \$1, \$2, return status codes in 0–255 via return, and benefit from local variables for scoping; their output is captured with command substitution as result=\$(greet "World"), and a function must be defined before it is called.

Exit codes transform scripts into reliable building blocks. A zero status means success, anything else flags failure, and \$? exposes the most recent status. Scripts exit explicitly with exit 1, and pipelines can short-circuit with cmd1 && cmd2 (run cmd2 only if cmd1 succeeded) or cmd1 || cmd2 (run cmd2 only on failure). Defensive practice includes set -e (exit on first error) and set -o pipefail (catch failures anywhere in a pipeline). Redirection complements errors: > file and >> file handle stdout, 2> captures stderr, 2>&1 merges stderr into stdout, and command > /dev/null 2>&1 silences everything — a common idiom for "run but don't pollute the console."

Package management differs by family. Debian and Ubuntu systems use apt: apt update refreshes the package index, apt upgrade applies upgrades, apt install nginx installs a package, apt remove nginx keeps configuration while removing binaries, and apt purge nginx removes configuration too. apt search keyword finds packages by name or description, apt list --installed audits the system, and dpkg -i package.deb installs a local archive. RHEL-family distributions use dnf (the successor to yum): dnf install httpd, dnf remove httpd, dnf update, dnf search keyword, dnf info httpd, and dnf list installed cover the same workflow, with rpm -ivh package.rpm for local files and rpm -qa for installed-package inventory.

Observability starts with what the kernel exposes. The traditional ring buffer holds boot and runtime messages that dmesg can dump (dmesg -T for human timestamps, dmesg -w to follow new entries, and dmesg -l err to filter by severity emerg→debug). On systemd hosts the same data appears under journalctl -k; if persistence is desired, /var/log/dmesg plus the systemd journal in /var/log/journal cover most needs. The /proc filesystem surfaces CPU details (/proc/cpuinfo), memory (/proc/meminfo), load averages (/proc/loadavg), and a per-process directory at /proc/[PID]. The /sys virtual filesystem (sysfs) exposes kernel objects — devices, drivers, buses — and modern tools like lsblk and ip read it directly. Writing to certain paths under /sys/class/ alters kernel state, though the kernel provides no undo, so changes should be deliberate. Persistent logs are managed with logrotate: per-service drop-ins in /etc/logrotate.d/ specify directives such as daily, rotate 14, compress, missingok, and a postrotate block to signal services (e.g., kill -USR1 \$(cat /var/run/nginx.pid)). The logrotate -d flag runs in debug mode without applying changes; logrotate -f forces execution; the daily cron job lives at /etc/cron.daily/logrotate.

CPU and memory pressure reveal themselves through uptime, top, and /proc/loadavg, which report load averages over 1, 5, and 15 minutes. On a system with \(N\) CPU cores, a load of \(N\) means full utilization and a load above \(N\) means a queue is building; uninterruptible (state D) tasks — usually blocked on I/O — also count toward load, so a rising load with stable CPU usage typically signals disk or memory pressure rather than CPU saturation. Three tools from the sysstat package quantify this: vmstat 1 5 samples virtual memory, CPU, and I/O (showing runnable r, blocked b, swap in/out si/so, and CPU time split between user, system, idle, wait, and steal); iostat -xz 1 tallies per-device disk activity; mpstat -P ALL 1 reports per-CPU statistics. The systemd-cgtop command gives a top-like view of control-group resource usage.

When a process misbehaves, deeper introspection arrives through tracers and profilers. strace -p <pid> attaches to a running process and prints every system call it makes — invaluable for "why is this hanging?" — while strace -f -o trace.log cmd traces a new command and follows forks, -e openat,read,write filters calls, -T measures each call's duration, and -c summarizes counts. ltrace does the same for library calls; perf trace provides higher-throughput equivalents. perf itself samples performance counters: perf stat cmd reports aggregate counters, perf record -F 99 -a -g cmd samples at 99 Hz with a call graph for later perf report analysis, perf top provides a live top-like view, and perf record -b base; perf record -b patched; perf diff compares two builds. pidstat -u 1 and pidstat -d 1 produce per-process CPU and I/O statistics, iotop ranks processes by disk activity, nethogs attributes bandwidth to processes, and atop records historical system state for retrospective analysis.

Runtime kernel parameters are managed through sysctl, which reads and writes /proc/sys/. sysctl -a lists every parameter, sysctl net.ipv4.ip_forward reads one, and sysctl -w net.ipv4.ip_forward=1 writes it on the fly. Persistent changes belong in files under /etc/sysctl.d/, applied by sysctl --system or sysctl -p /etc/sysctl.conf; common tunings include net.ipv4.tcp_tw_reuse=1, vm.swappiness=10, and fs.file-max=100000. Accurate time is equally foundational: hwclock -r reads the hardware clock, timedatectl shows the current state, timedatectl set-ntp true enables systemd-timesyncd, and chronyc tracking / chronyc sources reveal the chrony client's view; chrony is generally preferred on virtual machines and laptops with intermittent connectivity.

Cgroups and namespaces together form the substrate of modern containers. Cgroups (control groups) organize processes into hierarchies and impose resource limits — systemd creates one per service at /sys/fs/cgroup/systemd.slice/<name>.service/, and unit files can declare CPUQuota=50%, MemoryMax=512M, IOWeight=500, and TasksMax=100; cgroups v2 (the unified hierarchy) is the default on modern kernels. Namespaces provide isolation across several dimensions: pid for process IDs, net for the network stack, mount for mount points, uts for hostname, ipc for inter-process communication, user for UID mapping, and cgroup for cgroup visibility. Each container receives a private PID, network, mount, and user namespace plus cgroup-enforced CPU and memory limits. Operators can experiment by hand: unshare --pid --fork bash opens a new PID namespace in a shell, while nsenter -t 1 -m -u -i -n -p bash enters existing namespaces — useful for troubleshooting a process that has detached itself. Container runtimes such as runc, crun, and containerd orchestrate these primitives; inspect any running container with cat /proc/1/cgroup and ls -l /proc/1/ns/.