Data observability is the ability to understand the health of your data at any point in time. How it differs from testing and monitoring, and how to build it.
Data observability is not data testing. Tests are assertions you write in advance about properties you expect to hold. Observability is the ability to ask "what happened to my data and why" after a failure occurs -- including failures you did not predict. The distinction matters because test coverage has a ceiling; observability does not.
The five pillars framework (freshness, volume, distribution, schema, lineage) is the right starting point for any data observability program. Freshness and volume are easy to instrument and catch the most common failures. Distribution monitoring (tracking statistical properties of columns over time) catches silent data drift. Schema monitoring catches upstream changes that break downstream dependencies. Lineage is the hardest and most valuable: when something breaks, lineage tells you what else is affected and why.
The connection to AI systems is direct. LLM and ML pipelines are downstream consumers of data. A freshness failure means a model inference runs on stale features. A distribution shift means the model is operating out of its training distribution. A schema change means a feature pipeline throws at runtime or silently returns nulls. Observability on the data layer is the early warning system for AI system reliability.
The build-vs-buy decision for observability tooling is clearer than it is for most data engineering tools. Open-source options (Elementary, Marquez, Great Expectations) cover three to four of the five pillars adequately. Paid tools (Monte Carlo, Metaplane, Acceldata) add the fifth pillar (lineage at scale) and polished alerting UX. The right threshold: if you have more than five critical pipelines and an on-call rotation, the paid tier ROI is positive. Below that, the open-source stack is sufficient.
The clearest explanation of where testing, monitoring, and observability differ and when each is the right investment. Read before buying any tooling.
Freshness, volume, distribution, schema, lineage -- the five dimensions that together describe data health. Understanding this framework makes every other data quality tool easier to evaluate.
Marquez implements the OpenLineage spec for lineage collection. Pairs with Airflow and Spark to give you the lineage pillar of data observability without a paid tool.
Built on top of dbt, Elementary turns your dbt test results and run metadata into an observability layer. The free tier covers the four non-lineage pillars for dbt-native teams.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.