Data quality is the foundation of trustworthy analytics, encompassing accuracy, completeness, consistency, timeliness, and validity. Poor data quality leads to incorrect analytics, bad business decisions, and compliance risks. Key practices include data validation, profiling, anomaly detection, and quality checks at each pipeline stage using tools like Great Expectations and dbt tests. Data profiling examines a dataset to collect statistics—data types, null counts, unique values, distributions, min/max values, and patterns—and is an essential first step to identify quality issues before building pipelines. Great Expectations is an open-source Python library that lets you define declarative assertions (expectations) about data, generate quality reports, and integrate with orchestrators like Airflow to catch issues before bad data reaches downstream systems.
Data observability extends these ideas by monitoring a data system's health across five pillars: freshness, volume, schema, quality, and lineage. Tools like Monte Carlo, Great Expectations, and Datafold detect anomalies, broken pipelines, and silent data corruption. Data downtime is the period during which data is incomplete, inaccurate, missing, or stale—analogous to website downtime but often invisible until wrong dashboards are noticed. A data SLI (Service Level Indicator) is a quantitative measure of pipeline health (such as "freshness" or "% of nulls in column X"), and a data SLO (Service Level Objective) is a target for that indicator (such as "99% of dashboards updated within 1 hour"), making data reliability a first-class engineering concern.
Governance and discoverability round out the picture. Data governance is a framework of policies, processes, and standards covering data quality, security, privacy, compliance (such as GDPR), access control, and metadata management. Data lineage tracks the origin, movement, and transformation of data throughout its lifecycle, answering "where did this data come from?" and "what transformations were applied?"; it is essential for debugging, impact analysis, and regulatory compliance. Tools like OpenLineage and dbt provide lineage capabilities. A data catalog is an organized inventory of data assets providing metadata, lineage, descriptions, and search capabilities, helping engineers and analysts discover and trust datasets. Popular tools include Apache Atlas, AWS Glue Data Catalog, Alation, and DataHub. A data contract is an explicit agreement between data producers and consumers about the schema, semantics, SLAs, and ownership of a dataset, the data equivalent of an API contract for data.
A schema registry is a central service that stores, versions, and enforces schemas for data streams, especially Kafka topics. The Confluent Schema Registry supports Avro, JSON Schema, and Protobuf with backward, forward, and full compatibility checks, preventing breaking schema changes. Avro is row-based, dynamically typed, and rich in schema evolution, while Protobuf is binary, strongly typed, more compact, and faster to encode but with more rigid schema evolution. Choose Avro for flexibility and broad tooling, Protobuf for performance and typed APIs. The final layer is decision support: a dashboard is an interactive, real-time view of metrics designed for ongoing monitoring and exploration, while a report is a static, periodic deliverable summarizing data over a period. A semantic layer is a centrally defined, metric-aware layer between raw data and BI tools that defines business concepts (such as "active customer" or "net revenue") as code, ensuring every dashboard uses the same definition; tools include dbt Semantic Layer, Cube, and LookML. A metric store catalogs, versions, and serves canonical business KPIs with logic, dimensions, and time grain, complementing the semantic layer with first-class metric governance.