System Design

This deck walks through the foundational building blocks of system design, from scaling strategies and load balancing to caching, databases, and content delivery. Each card targets a core concept—like the difference between horizontal and vertical scaling, how sharding differs from replication, or what makes Layer 4 and Layer 7 load balancers distinct—so you can build a clear mental map of how large-scale systems are structured.

It's a great fit if you're preparing for a software engineering interview, starting a new project that needs to scale, or simply want to strengthen your intuition for how distributed systems fit together. Beginners will find approachable definitions, while more experienced engineers can use the cards to refresh terminology and compare related approaches side by side.

To get the most out of these flashcards, try connecting each concept to a real-world example as you review—think of a website or app you use and imagine how it might apply load balancing, caching, or sharding. Space out your study sessions over a few days rather than cramming, since these ideas build on each other and benefit from repeated exposure. When you get a card wrong, take a moment to explain the answer out loud in your own words before moving on.

Scaling is the practice of increasing a system's capacity to handle growing demand. The two fundamental approaches are vertical scaling, which adds more power (CPU, RAM, disk) to an existing machine, and horizontal scaling, which adds more machines to the resource pool. Vertical scaling is simpler because it avoids distributed coordination and requires no code changes, but it hits a hardware ceiling, creates a single point of failure, and often involves downtime during upgrades. Horizontal scaling, by contrast, offers near-infinite scalability, better fault tolerance since one server failing does not take down the system, and the ability to use commodity hardware. The trade-off is increased complexity in data consistency and distributed coordination.

When a system runs across multiple servers, a load balancer becomes essential. A load balancer distributes incoming network traffic across backend servers so that no single server is overwhelmed. Sitting between clients and servers, it improves availability by routing around failed servers, increases throughput by enabling parallel processing, and reduces latency by directing traffic to the least-loaded server. Common implementations include NGINX, HAProxy, and AWS ALB or NLB.

Load balancers use various algorithms to decide where to send each request. Round robin distributes requests sequentially across servers, while weighted round robin assigns more requests to higher-capacity machines. Least connections routes to the server with the fewest active connections, IP hash provides session stickiness by routing based on client IP, and least response time sends traffic to the fastest-responding server. Random selection is occasionally used when simplicity matters most.

Load balancers also differ in the OSI layer at which they operate. Layer 4 (transport) load balancers route based on IP addresses and TCP or UDP ports without inspecting packet contents, making them fast but less flexible. Layer 7 (application) load balancers inspect HTTP headers, cookies, and URLs to make smarter routing decisions, enabling content-based routing, SSL termination, and request modification. To know which servers are healthy, load balancers perform periodic probes through health checks, which can be active (sending HTTP or TCP requests at intervals) or passive (monitoring real traffic for errors). Configurable parameters include interval, timeout, unhealthy threshold, and healthy threshold.

Caching accelerates data access by storing frequently used information closer to where it is needed. Redis (Remote Dictionary Server) is a popular open-source, in-memory key-value data store that delivers sub-millisecond latency and supports rich data structures including strings, hashes, lists, sets, sorted sets, and streams. Redis offers persistence options through RDB snapshots and the Append Only File (AOF), built-in replication, Lua scripting, pub/sub messaging, and a cluster mode for horizontal scaling.

Memcached is another distributed in-memory caching system, but it is designed for simplicity and raw speed. Compared to Redis, Memcached supports only simple key-value strings, has no persistence, is multi-threaded (versus Redis's traditionally single-threaded model), and lacks built-in replication. Memcached shines for simple caching workloads, while Redis is better suited to complex use cases that benefit from diverse data types and durability.

At the edge of the network, content delivery networks (CDNs) cache static and dynamic content on geographically distributed edge servers. When a user requests content, it is routed to the nearest edge server; if cached, it is served immediately with low latency, and if not cached, it is fetched from the origin, stored, then delivered. CDNs reduce latency, lower origin server load, and provide DDoS protection. CloudFlare, AWS CloudFront, and Akamai are prominent examples.

Keeping caches fresh requires deliberate invalidation strategies. Time-to-live (TTL) expiration is the simplest approach. Event-based invalidation triggers updates on write events, while write-through caching writes simultaneously to both cache and database, and write-behind (write-back) caching writes to the cache first and updates the database asynchronously in batches. Manual purges explicitly delete specific cache keys. The cache-aside (lazy-loading) pattern places the application in charge of the cache: on a read miss, the application queries the database and populates the cache, and on writes it invalidates the cache entry. Read-through caching shifts this responsibility to the cache itself, which loads missing data transparently. Each approach balances latency, consistency, and complexity in different ways.

As data volumes grow, distributing them across multiple database nodes becomes necessary. Sharding is a horizontal partitioning strategy that splits data across multiple database instances, with each shard holding a subset of total data. A shard key, such as user_id modulo the number of shards, determines where each record lives. Sharding enables horizontal scalability and better performance, but cross-shard queries become expensive, joins across shards are difficult, and rebalancing shards is complex.

Replication copies data from a primary database server to one or more replicas, improving read scalability, high availability, and disaster recovery. In synchronous replication, the primary waits for replica acknowledgment before confirming a write, providing stronger consistency. Asynchronous replication lets the primary proceed without waiting, giving better performance at the cost of eventual consistency. Semi-synchronous replication waits for at least one replica. Replication topologies include single-leader setups, multi-leader configurations (useful across datacenters but requiring conflict resolution), and leaderless designs where any node can accept reads or writes as long as a quorum is met. Common conflict resolution strategies include last-write-wins, vector clocks, and CRDTs.

Partitioning divides a large table into smaller pieces for easier management. Horizontal partitioning splits rows across partitions, such as orders divided by date range, while vertical partitioning splits columns, separating frequently accessed fields from rarely used ones. Common partitioning strategies include range partitioning by value ranges, hash partitioning by hash of a key, and list partitioning by explicit value lists.

Within a database, indexes speed up data retrieval at the cost of additional storage and slower writes. Most indexes use B-tree or B+ tree data structures, and indexes can be primary (on the primary key), secondary (on non-primary columns), composite (on multiple columns), or covering (containing all columns needed by a query). Choosing between SQL and NoSQL databases also shapes system design: SQL databases like PostgreSQL and MySQL are relational, use fixed schemas, offer ACID compliance, and scale vertically; NoSQL databases like MongoDB, Cassandra, and DynamoDB offer flexible schemas, horizontal scalability, and typically eventual consistency. Finally, connection pooling maintains a pool of reusable database connections, avoiding the overhead of establishing new connections for every request. Pools start with a minimum number of connections, grow up to a maximum under load, and are commonly implemented by tools like HikariCP, pgBouncer, and SQLAlchemy.

Monolithic architectures package all application functionality into a single deployable unit, which is simpler to develop and deploy initially but harder to scale individual components and ties the entire system to a single codebase and deployment. Microservices decompose the application into small, independently deployable services that can scale, deploy, and use different technologies independently, with better fault isolation as a key benefit. The trade-off is increased operational complexity in networking, observability, and data consistency.

In a microservices environment, instances come and go dynamically, so services need a way to locate each other. Service discovery provides this mechanism. In client-side discovery, the client queries a service registry such as Netflix Eureka, Consul, etcd, or ZooKeeper, and selects an instance directly. In server-side discovery, the client sends a request to a load balancer or router (such as AWS ALB or Kubernetes kube-proxy), which queries the registry on the client's behalf.

An API gateway acts as a single entry point for all client requests to backend microservices. It handles request routing, authentication and authorization, rate limiting and throttling, request and response transformation, SSL termination, caching, logging, and monitoring. By centralizing these cross-cutting concerns, an API gateway simplifies client interactions. Kong, AWS API Gateway, NGINX, and Apigee are popular implementations.

A reverse proxy similarly sits in front of backend servers and forwards client requests to them, but its focus is on behalf of servers rather than clients, since a forward proxy acts on behalf of clients. Reverse proxies provide security by hiding backend server IPs, perform SSL termination, distribute traffic as a load balancer, serve cached responses, and compress payloads. NGINX, HAProxy, and Traefik are common examples. Underpinning all of this is DNS, the Domain Name System that translates human-readable domain names into IP addresses. Resolution flows from the browser's local cache to a recursive resolver, then through root nameservers, TLD nameservers, and finally authoritative nameservers, with results cached according to their TTL. Common record types include A (IPv4), AAAA (IPv6), CNAME (alias), MX (mail), NS (nameserver), and TXT.

Rate limiting controls the number of requests a client can make to a service within a given time window, protecting against abuse and ensuring fair resource usage. When limits are exceeded, services typically respond with HTTP 429 Too Many Requests along with a Retry-After header. Rate limits can be enforced per user, per API key, per IP address, or globally for an entire service, and can be implemented at the API gateway, load balancer, or application layer.

Several algorithms implement rate limiting. The token bucket algorithm holds tokens in a bucket up to a burst capacity, refills tokens at a fixed rate per second, and consumes one token per request. If the bucket is empty, requests are rejected or queued. Token buckets allow short bursts of traffic up to bucket size, are simple to implement, and smooth out traffic over time, which is why AWS, Stripe, and many other API providers use them.

The sliding window algorithm tracks requests in a continuously moving time window. The sliding window log stores timestamps of all requests, removes timestamps older than the window on each new request, and rejects the request if the count exceeds the limit. The sliding window counter is a hybrid that combines the current and previous fixed window counts, weighting by overlap. Sliding windows are more accurate than fixed windows and avoid the boundary burst problem.

Beyond request throttling, applications often need real-time communication between client and server. WebSockets provide a full-duplex, persistent communication channel over a single TCP connection, starting with an HTTP upgrade handshake and then allowing both sides to send messages at any time. They are ideal for real-time chat, live dashboards, multiplayer games, and collaborative editing, and use the ws or wss (encrypted) protocols. Long polling is an alternative where the client sends an HTTP request and the server holds it open until new data is available or a timeout occurs; the client then immediately reopens the connection. Long polling works everywhere HTTP works without special protocols, but it carries higher latency than WebSockets and forces the server to hold many open connections. Server-Sent Events (SSE) provide a unidirectional push channel from server to client over a single, long-lived HTTP connection using the text/event-stream content type, with built-in browser auto-reconnection. SSE is best for server-push scenarios like live feeds and notifications, while WebSockets are preferable when bidirectional communication is needed.

Asynchronous messaging decouples services and enables reliable communication at scale. Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming applications. Its core concepts include topics (named feeds of messages), partitions (which split topics for parallelism), producers (which publish messages), consumers (which read messages through consumer groups), and brokers (the Kafka servers in a cluster). Kafka delivers high throughput, durable storage by persisting messages to disk, exactly-once semantics, and replay capability.

RabbitMQ is an open-source message broker implementing the AMQP protocol. Producers send messages to exchanges, which route them to queues based on bindings and routing keys, where consumers read them. Exchange types include direct, fanout, topic, and headers. Compared to Kafka, RabbitMQ excels at complex routing and task queues with delivery guarantees, while Kafka is better suited for high-throughput event streaming and log aggregation.

The publish/subscribe (pub/sub) messaging pattern underlies many of these systems. Publishers send messages to a topic without knowing who will receive them, and subscribers receive messages from topics they care about. This pattern provides loose coupling between producers and consumers, supports one-to-many communication, and allows subscribers to be added without modifying publishers. Implementations include Kafka topics, Redis Pub/Sub, AWS SNS, and Google Pub/Sub.

These messaging primitives enable event-driven architecture (EDA), in which components communicate by producing and consuming events that represent state changes. Event producers emit events, an event broker (such as Kafka or RabbitMQ) routes them, and event consumers react accordingly. EDA offers loose coupling, scalability, real-time processing, and auditability, but introduces challenges around eventual consistency, event ordering, and debugging. Command Query Responsibility Segregation (CQRS) takes this further by separating read and write operations into different models: the command side handles writes optimized for validation and business logic, while the query side handles reads optimized for fast queries and projections. This enables independent scaling of reads and writes and tailored data models. CQRS is often paired with event sourcing, where state changes are stored as an immutable, append-only sequence of domain events in an event store. Current state is then derived by folding all events over an initial state, providing a complete audit trail and temporal queries at the cost of event versioning, storage growth, and rebuild time. EventStoreDB and Kafka used as an event store are common implementations.

Distributed systems face inherent trade-offs that shape every design decision. The CAP theorem states that a distributed data store can guarantee at most two of three properties simultaneously: consistency (every read receives the most recent write), availability (every request receives a response), and partition tolerance (the system operates despite network partitions). Since network partitions are inevitable in real systems, the practical choice is between CP systems (such as HBase and MongoDB) and AP systems (such as Cassandra and DynamoDB). Consistency models describe how quickly a system converges after this trade-off is made. With strong consistency, every read after a write returns the updated value, and every node sees the same data at the same time, at the cost of higher latency and lower availability. With eventual consistency, reads may temporarily return stale data, but all replicas will converge given enough time without new updates, prioritizing availability and lower latency. Conflicts in eventually consistent systems are resolved through timestamps, vector clocks, or application logic. Examples of strong consistency include Spanner and CockroachDB; examples of eventual consistency include DynamoDB and Cassandra.

Traditional relational databases favor ACID guarantees: atomicity (transactions are all-or-nothing), consistency (transactions move the database from one valid state to another), isolation (concurrent transactions do not interfere), and durability (committed changes survive crashes). NoSQL and distributed systems often embrace BASE instead: basically available, soft state, and eventual consistency. BASE trades strong consistency for availability and partition tolerance, which is appropriate for the scale and flexibility demands of distributed systems.

Maintaining correctness across distributed transactions is challenging. Two-phase commit (2PC) is a protocol where a coordinator first asks all participants to vote on whether they can commit, then sends commit or abort based on the votes. While 2PC ensures atomicity, it is blocking and makes the coordinator a single point of failure. The saga pattern offers an alternative, breaking distributed transactions into a sequence of local transactions, each paired with a compensating action for rollback. In choreography, each service publishes events that trigger the next step; in orchestration, a central orchestrator coordinates the saga steps. Idempotency is crucial in distributed systems: an operation is idempotent if performing it multiple times produces the same result as performing it once. Idempotency keys, unique IDs sent with each request, allow servers to detect and discard duplicate requests arising from network retries or at-least-once delivery from message queues. In HTTP, GET, PUT, and DELETE are idempotent, while POST is not.

Resilience patterns help systems tolerate failures gracefully. The circuit breaker pattern prevents cascading failures by stopping calls to a failing service. In the closed state, requests flow normally and failures are counted; once a threshold is reached, the circuit opens and all requests are immediately rejected so the system fails fast. After a timeout, the circuit becomes half-open and a limited number of test requests are allowed through; if they succeed, the circuit closes again, otherwise it reopens. Libraries like Resilience4j, Hystrix, and Polly implement this pattern. Back-pressure complements circuit breakers by allowing downstream components to signal upstream components to slow down when they cannot keep up, using strategies such as buffering in bounded queues, dropping excess messages, blocking the producer, or explicitly rate limiting throughput. Without back-pressure, systems suffer memory exhaustion and cascading failures. Finally, consistent hashing distributes data across nodes so that adding or removing a node only requires remapping \(K/N\) keys (where K is the total number of keys and N is the number of nodes). Nodes and keys are mapped onto a hash ring, and each key is assigned to the first node clockwise from its position; virtual nodes, which map each physical node to multiple ring positions, improve balance. Consistent hashing is used by DynamoDB, Cassandra, and Memcached to scale smoothly.