How data engineers measure, monitor, and enforce data quality at scale -- and why "data downtime" is the right mental model for the problem.
Data quality is not a cleanliness metric -- it is a reliability problem. The right analogy is not "clean data vs. dirty data" but "uptime vs. downtime." A pipeline that produces wrong numbers for four hours is experiencing data downtime. Frame it that way and the engineering response becomes obvious: SLAs, alerting, incident response, and root-cause analysis.
The five dimensions of data quality that matter operationally are freshness, volume, distribution, schema, and lineage. A freshness problem means data stopped arriving. A volume problem means it arrived but at the wrong rate. A distribution problem means values shifted outside expected ranges. A schema problem means column names or types changed. A lineage problem means you cannot trace which upstream change caused the downstream failure. Each requires a different detection strategy.
Most data quality tools measure the wrong thing. They check whether data satisfies rules you already know to write. What they cannot catch is what you do not know to check for -- silent shifts in distributions, subtle joins that started returning duplicates, upstream model changes that changed a field's semantics without changing its type. Statistical profiling (tracking value distributions over time) catches a broader class of failures than rule-based checks alone.
For AI systems, data quality is load-bearing in a way it is not for dashboards. A dashboard with wrong numbers gets corrected when someone notices. A model trained on wrong numbers bakes the error into weights -- the damage is hidden and multiplied. The minimum viable data quality investment for any ML pipeline is: freshness monitoring on every source, schema change detection on every upstream table, and statistical drift detection on every feature used in training.
The most complete treatment of data quality as an engineering discipline, not a checklist. Read the chapters on data downtime and reliability before picking any tooling.
Reframes data quality as an uptime/SLA problem rather than a governance problem. This mental model unlocks the right engineering responses.
The fastest path from zero to tested data. Four built-in tests (not_null, unique, accepted_values, relationships) catch 80% of production data quality issues.
Browse hundreds of pre-built expectations to understand what is measurable. Useful even if you choose Soda or dbt tests instead.
Some course links above are affiliate links. If you enroll, we may earn a small commission at no extra cost to you.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.