Several architectural patterns have emerged to address the scale, diversity, and real-time demands of modern data. The medallion architecture, popularized by Databricks, organizes a data lake into three layers: Bronze holds raw, immutable data exactly as it arrived from sources (the system of record); Silver contains cleaned, enriched, deduplicated, and conformed data that answers "what is the current state of the business?"; and Gold holds business-level aggregates and curated datasets ready for BI, ML, and reverse ETL. Each layer applies progressively more transformation, providing clear separation of concerns and traceability.
Data mesh is a decentralized architecture paradigm proposed by Zhamak Dehghani. It treats data as a product owned by domain teams rather than centralized data teams, organized around four principles: domain-oriented ownership, data as a product, self-serve data infrastructure, and federated computational governance. Data mesh pushes responsibility for data quality and discoverability to the teams closest to the data, addressing the bottlenecks of centralized data teams. Reverse ETL moves data from a warehouse back into operational systems like CRM, marketing, and support platforms, putting warehouse-curated customer, product, and event data into the tools frontline teams use daily. Tools like Hightouch and Census specialize in this pattern. A feature store is a centralized system that stores, versions, and serves ML features for both training and real-time inference, providing low-latency online serving and point-in-time correct offline retrieval to prevent training-serving skew.
Event-driven architectures underpin many of these patterns. In a request-response architecture, services synchronously call each other and wait for replies, coupling, latency, and failure modes compound. In an event-driven architecture, services publish events to a broker (Kafka, SNS) and react asynchronously, decoupling producers from consumers and enabling independent scaling. The outbox pattern solves dual-write consistency between a database and a message broker by writing the message to an outbox table in the same transaction; a separate process (such as Debezium) tails the outbox and publishes to Kafka, ensuring at-least-once delivery. The saga pattern manages distributed transactions across services by sequencing local transactions with compensating actions, in either orchestration (a central coordinator) or choreography (services react to events) style. CQRS (Command Query Responsibility Segregation) splits a system into separate write and read models, often paired with event sourcing, which persists the sequence of state changes as events rather than storing current state—offering full audit trail and time travel at the cost of query complexity.
Specialized analytical databases round out the modern stack. Apache Druid is a real-time analytics database optimized for sub-second OLAP queries on high-cardinality time-series and event data, with streaming ingestion from Kafka and pre-aggregation via roll-ups. Apache Pinot is similar, designed for low-latency user-facing analytics with upserts and star-tree indexes. ClickHouse is an open-source columnar OLAP database delivering extremely fast analytical queries on petabyte-scale data using vectorized execution and MergeTree storage engines, popular for observability, ad analytics, and real-time reporting. TimescaleDB turns Postgres into a time-series database via hypertables and continuous aggregates, fully SQL-compatible. InfluxDB is a purpose-built time-series database for high-ingest metrics and IoT data with a custom line protocol and TSI indexes. The fundamental choice between a database, a data warehouse, and a lakehouse comes down to access patterns and data variety: databases optimize for OLTP, warehouses for OLAP over structured data, and lakehouses for flexibility with warehouse-style features. Finally, Google Cloud Dataflow provides pre-built pipeline templates that can be launched repeatedly with different parameters, encapsulating best-practice patterns like Pub/Sub to BigQuery for standard ingestion.