Apache Spark is a distributed computing framework for large-scale data processing with APIs in Python (PySpark), Scala, Java, and R. It supports batch processing, stream processing, machine learning (via MLlib), and graph processing. Spark's key advantage is in-memory computation, which makes it significantly faster than Hadoop MapReduce for many workloads. The Spark DataFrame is a distributed collection of data organized into named columns with a high-level API for transformations and actions, allowing complex pipelines to be expressed concisely in Python and Scala.
Spark executes transformations in two flavors. Narrow transformations, such as map and filter, operate within a single partition without network shuffling. Wide transformations, such as groupBy or a join with different keys, require data to be reshuffled across partitions. Shuffles are the most expensive operation in Spark because they write data to local disk and transfer data between executors, so minimizing wide transformations is the primary lever for performance. When one side of a join is small enough to fit in memory, a broadcast join copies it to every executor and avoids shuffling the large side. Data skew—where some keys hold disproportionately more data—causes stragglers and OOM errors; common mitigations include broadcast joins, salting keys with a random prefix to distribute hot keys across partitions, and choosing partitioning keys with high cardinality.
Apache Flink is a distributed stream processing framework designed for stateful, real-time processing with low latency. Unlike Spark's micro-batch approach, Flink processes events one at a time with true streaming semantics, supporting event-time processing, exactly-once guarantees, and complex event processing. Apache Beam provides a unified programming model that can run on multiple engines, including Flink, Spark, and Google Cloud Dataflow, making it attractive when portability matters. Stateful stream processing operators maintain information across events—running counts, last-seen values, or session windows—and that state is checkpointed to durable storage for fault tolerance. Window processing groups streaming events into finite sets for aggregation, with common window types being tumbling (fixed, non-overlapping), sliding (overlapping intervals), and session (dynamic, activity-gap-based).
A core distinction in stream processing is between event time and processing time. Event time is the timestamp embedded in the data, while processing time is the wall-clock time at which the system processes the event; ingestion time is the time the event enters the pipeline. Watermarks are lower-bound timestamps that indicate how long the engine is willing to wait for late-arriving data, enabling deterministic window evaluation. Late-arriving data can be dropped, used to update results, or appended to a separate "late" table for later correction. Backpressure is a flow-control mechanism in which a slow consumer signals upstream producers to slow down when its buffer fills, preventing OOM errors. Checkpoints are durable snapshots of state and source offsets written periodically to reliable storage, allowing recovery from the last snapshot; a write-ahead log (WAL) instead logs every change before it is applied. Many systems use both: WAL for ongoing durability and checkpoints to bound recovery time. A transaction is a single unit of work with ACID guarantees, while a micro-batch is a small group of records processed together at fixed intervals, trading latency for simpler exactly-once semantics. In Spark Structured Streaming, a trigger controls how often a streaming query produces micro-batches, with options like processingTime, once, or continuous.