Distributed systems face inherent trade-offs that shape every design decision. The CAP theorem states that a distributed data store can guarantee at most two of three properties simultaneously: consistency (every read receives the most recent write), availability (every request receives a response), and partition tolerance (the system operates despite network partitions). Since network partitions are inevitable in real systems, the practical choice is between CP systems (such as HBase and MongoDB) and AP systems (such as Cassandra and DynamoDB). Consistency models describe how quickly a system converges after this trade-off is made. With strong consistency, every read after a write returns the updated value, and every node sees the same data at the same time, at the cost of higher latency and lower availability. With eventual consistency, reads may temporarily return stale data, but all replicas will converge given enough time without new updates, prioritizing availability and lower latency. Conflicts in eventually consistent systems are resolved through timestamps, vector clocks, or application logic. Examples of strong consistency include Spanner and CockroachDB; examples of eventual consistency include DynamoDB and Cassandra.
Traditional relational databases favor ACID guarantees: atomicity (transactions are all-or-nothing), consistency (transactions move the database from one valid state to another), isolation (concurrent transactions do not interfere), and durability (committed changes survive crashes). NoSQL and distributed systems often embrace BASE instead: basically available, soft state, and eventual consistency. BASE trades strong consistency for availability and partition tolerance, which is appropriate for the scale and flexibility demands of distributed systems.
Maintaining correctness across distributed transactions is challenging. Two-phase commit (2PC) is a protocol where a coordinator first asks all participants to vote on whether they can commit, then sends commit or abort based on the votes. While 2PC ensures atomicity, it is blocking and makes the coordinator a single point of failure. The saga pattern offers an alternative, breaking distributed transactions into a sequence of local transactions, each paired with a compensating action for rollback. In choreography, each service publishes events that trigger the next step; in orchestration, a central orchestrator coordinates the saga steps. Idempotency is crucial in distributed systems: an operation is idempotent if performing it multiple times produces the same result as performing it once. Idempotency keys, unique IDs sent with each request, allow servers to detect and discard duplicate requests arising from network retries or at-least-once delivery from message queues. In HTTP, GET, PUT, and DELETE are idempotent, while POST is not.
Resilience patterns help systems tolerate failures gracefully. The circuit breaker pattern prevents cascading failures by stopping calls to a failing service. In the closed state, requests flow normally and failures are counted; once a threshold is reached, the circuit opens and all requests are immediately rejected so the system fails fast. After a timeout, the circuit becomes half-open and a limited number of test requests are allowed through; if they succeed, the circuit closes again, otherwise it reopens. Libraries like Resilience4j, Hystrix, and Polly implement this pattern. Back-pressure complements circuit breakers by allowing downstream components to signal upstream components to slow down when they cannot keep up, using strategies such as buffering in bounded queues, dropping excess messages, blocking the producer, or explicitly rate limiting throughput. Without back-pressure, systems suffer memory exhaustion and cascading failures. Finally, consistent hashing distributes data across nodes so that adding or removing a node only requires remapping \(K/N\) keys (where K is the total number of keys and N is the number of nodes). Nodes and keys are mapped onto a hash ring, and each key is assigned to the first node clockwise from its position; virtual nodes, which map each physical node to multiple ring positions, improve balance. Consistent hashing is used by DynamoDB, Cassandra, and Memcached to scale smoothly.