Apache Kafka is a distributed event streaming platform used for building real-time data pipelines. It operates on a publish-subscribe model in which producers write messages to topics and consumers read from them, with high throughput, fault tolerance through replication, and message durability. A topic is a named category or feed of messages, divided into partitions—ordered, immutable sequences of messages that enable parallelism and scalability. Each message in a partition is assigned a sequential offset, and partitions allow multiple consumers to read concurrently.
Consumers organize themselves into consumer groups, in which each partition is assigned to exactly one consumer, enabling parallel processing within the group. If a consumer fails, Kafka rebalances the partitions among the remaining consumers, and different consumer groups independently consume the same messages—useful for fanning out the same stream to multiple downstream systems. The ISR (in-sync replica) set is the collection of replicas for a partition that are caught up with the leader; the leader waits for acknowledgments from replicas in the ISR before considering a write durable. The acks setting on producers controls durability: acks=0 means fire-and-forget (lowest latency), acks=1 waits only for the leader, and acks=all waits for all in-sync replicas for the strongest guarantee.
Kafka offers two distinct retention mechanisms. Time- or size-based retention discards old messages regardless of content, while log compaction retains only the latest value for each key, making it ideal for change logs and slowly changing state. Retention is evaluated at segment boundaries, where log.segment.bytes controls the maximum size of an individual segment file on disk; old messages may briefly exceed the retention time until the segment closes. Kafka's transactional producer uses a transaction coordinator and an __transaction_state topic to atomically commit reads and writes across partitions, providing exactly-once delivery so that either all messages in a transaction are visible to read-committed consumers or none are.
Delivery guarantees exist on a spectrum. At-most-once delivery may lose messages but never duplicates them. At-least-once delivery never loses messages but may duplicate them. Exactly-once delivery processes each event exactly one time, requiring idempotent sinks or transactional output plus checkpointing of source offsets. A dead-letter queue (DLQ) is a special destination for messages that cannot be processed successfully after a number of retries, isolating poison messages so they do not block the main pipeline. When choosing a streaming sink (Kafka, S3, a database), key considerations are delivery guarantees, throughput, latency, and schema stability; sinks that support transactions (Kafka, Delta Lake) enable end-to-end exactly-once, while generic sinks usually require idempotency at the destination. Apache Flink and Kafka Streams both provide tools for stream processing: Kafka Streams is a client library that runs inside a Java application tightly coupled to Kafka topics, while Flink is a standalone distributed engine supporting many sources and sinks with more advanced state management.