How schema changes silently break downstream pipelines, and how data contracts prevent the problem by making schema a versioned, tested interface.
Schema drift is the most common source of silent pipeline failures. An upstream team renames a column, changes a type from string to integer, or drops a field that was rarely used in dashboards but was load-bearing in a feature pipeline. The downstream job fails -- or worse, silently coerces the value and produces wrong results. Without a contract, there is no warning before the failure and no accountability after.
The data contracts pattern solves this by making schema a versioned, tested interface -- the same discipline software engineering applies to APIs. A contract specifies the expected shape of a table or dataset: column names, types, nullability, uniqueness constraints, value ranges. Breaking the contract requires a versioned change and an explicit deprecation window, not a silent Tuesday afternoon ALTER TABLE.
Implementation does not require a new platform. dbt model contracts enforce schema at build time in SQL. Soda schema checks enforce it at load time. A JSON Schema file committed alongside an event schema enforces it at the producer. The tooling is less important than the discipline: schema changes go through a PR, consumers are notified, and the contract version is bumped.
For AI systems, schema drift is especially dangerous at the feature layer. A feature pipeline reads a column that changed type. The pipeline may cast silently, pass type-checking, and produce features that are numerically off by an order of magnitude. The model consumes them without complaint. The resulting prediction errors are real but invisible until a human notices. Data contracts on feature-producing tables -- with CI enforcement -- are the only reliable defense.
The essay that crystallized the data contracts movement. Sanderson makes the case that treating data as a product with a versioned schema interface -- not an internal implementation detail -- is the only scalable solution to schema drift.
dbt's native contract enforcement: declare expected column names and types in YAML; the build fails if a model violates its own contract. The easiest entry point for teams already on dbt.
SodaCL schema checks validate column presence and types at ingestion or transformation time. Catches drift before it reaches downstream consumers.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.