Data observability and its five pillars -- freshness, volume, distribution, schema, and lineage -- plus how it differs from testing and monitoring.
Data observability is not data testing. Tests are assertions you write in advance about properties you expect to hold. Observability is the ability to ask "what happened to my data and why" after a failure occurs -- including failures you did not predict. The distinction matters because test coverage has a ceiling; observability does not.
The five pillars framework (freshness, volume, distribution, schema, lineage) is the right starting point for any data observability program. Freshness and volume are easy to instrument and catch the most common failures. Distribution monitoring (tracking statistical properties of columns over time) catches silent data drift. Schema monitoring catches upstream changes that break downstream dependencies. Lineage is the hardest and most valuable: when something breaks, lineage tells you what else is affected and why.
The connection to AI systems is direct. LLM and ML pipelines are downstream consumers of data. A freshness failure means a model inference runs on stale features. A distribution shift means the model is operating out of its training distribution. A schema change means a feature pipeline throws at runtime or silently returns nulls. Observability on the data layer is the early warning system for AI system reliability.
The build-vs-buy decision for observability tooling is clearer than it is for most data engineering tools. Open-source options (Elementary, Marquez, Great Expectations) cover three to four of the five pillars adequately. Paid tools (Monte Carlo, Metaplane, Acceldata) add the fifth pillar (lineage at scale) and polished alerting UX. The right threshold: if you have more than five critical pipelines and an on-call rotation, the paid tier ROI is positive. Below that, the open-source stack is sufficient.
The five pillars framework, originated by Monte Carlo and now widely adopted across the industry, breaks data health into five independently measurable dimensions. Freshness tracks whether data arrived on schedule -- if a table that updates hourly has not received new rows in four hours, something is wrong upstream. Volume monitors row counts and byte sizes against historical baselines -- a 40 percent drop in daily records usually indicates a pipeline failure or an upstream schema change that silently dropped records.
Distribution observability tracks statistical properties of columns over time: mean, median, null rate, cardinality. When a column that was 5 percent null suddenly becomes 30 percent null, you have a data quality problem even if no schema changed and no job explicitly failed. Schema observability monitors column additions, removals, renames, and type changes -- the most common source of silent downstream failures where a pipeline keeps running but produces wrong results. Lineage, the fifth and hardest pillar, maps the dependency graph of every data asset: which tables feed which models, which models feed which dashboards, which dashboards feed which business decisions.
The five pillars are not equally hard to instrument. Freshness and volume monitoring require only metadata -- last-modified timestamps and row counts -- which every warehouse exposes natively at near-zero cost. Distribution monitoring requires scanning actual column values, which adds compute cost proportional to table size. Schema monitoring is nearly free if you are already running schema checks in CI or using dbt model contracts. Lineage is the hardest: it requires emitting lineage events from every tool in your stack (Airflow, Spark, dbt, your ingestion layer) and storing them in a compatible backend such as Marquez or a paid observability platform.
Testing, monitoring, and observability solve the same problem at different levels of generality. Testing is pre-defined assertions evaluated at a point in time: not_null on the order_id column, accepted_values on the status field, row count above a threshold. Testing answers the question "Does the data satisfy the properties I know to expect?" It has a coverage ceiling: tests cannot catch failures you did not anticipate, and new tables added without tests have zero coverage.
Monitoring adds the time dimension: you measure a metric continuously and alert when it deviates from a baseline. A volume monitor that alerts when row count drops more than 20 percent below the 7-day moving average catches failures that no static test would. The limitation of monitoring is that you still need to predefine what to measure. A new table added without a monitor attached to it has no coverage, and a monitor calibrated to the wrong baseline fires on every seasonality swing.
Observability is the ability to understand the state of your data system from its external outputs, without predefined instrumentation for the specific failure mode you are investigating. In practice, this means collecting broad telemetry (metadata, column statistics, lineage events) across the entire data platform and making it queryable after the fact. When an unexplained anomaly surfaces at 2am, observability is what lets you diagnose it by querying historical telemetry rather than reading log lines. Testing and monitoring prevent problems you anticipated; observability is your diagnostic tool for the problems you did not.
For data engineering teams managing pipelines that feed AI and ML systems, observability is especially critical because the feedback loop is longer. A monitoring gap on a feature table that a model depends on means you may not detect that a feature drifted until the model degrades in production -- potentially days after the source changed. Observability across the full lineage graph, from raw source to model input, is the only way to catch this class of failure at the data layer before it propagates to model quality.
Elementary Data, installed as a dbt package, instruments the metadata that dbt already produces and turns it into an observability layer covering four of the five pillars: freshness (via dbt source freshness checks), volume and distribution (via Elementary anomaly detection tests on row counts and column statistics), and schema (via dbt model contracts and test failures). For dbt-native teams, this covers the majority of observability needs with no additional infrastructure. The lineage pillar requires a separate tool.
Marquez implements the OpenLineage specification -- an open standard for lineage metadata supported natively by Apache Airflow, Apache Spark, and dbt. Adding the Airflow OpenLineage provider instruments every task automatically, emitting lineage events at zero marginal cost beyond the setup. Marquez stores and serves the lineage graph, which you can query to answer "what upstream jobs wrote to this table?" and "what downstream models read from this column?" The open-source combination of Elementary plus Marquez covers all five pillars for teams willing to operate the infrastructure.
Paid tools -- Monte Carlo, Metaplane, Acceldata -- add operationalization on top of coverage. Their primary advantages are automated baseline calibration that reduces false-positive alerts during seasonality, noise-reduced alerting with root-cause suggestions, column-level lineage at the scale of enterprise warehouses (which self-hosted Marquez struggles with above a certain graph size), and integrations with incident management and on-call tooling. The ROI threshold maps roughly to the cost of a data incident: for teams where a four-hour pipeline outage costs thousands of dollars in downstream impact, SaaS pricing pays for itself in avoided incidents. Below that threshold, the open-source stack is the right starting point.
The clearest explanation of where testing, monitoring, and observability differ and when each is the right investment. Read before buying any tooling.
Freshness, volume, distribution, schema, lineage -- the five dimensions that together describe data health. Understanding this framework makes every other data quality tool easier to evaluate.
Marquez implements the OpenLineage spec for lineage collection. Pairs with Airflow and Spark to give you the lineage pillar of data observability without a paid tool.
Built on top of dbt, Elementary turns your dbt test results and run metadata into an observability layer. The free tier covers the four non-lineage pillars for dbt-native teams.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.