Column-level lineage, OpenLineage, and how to use lineage graphs to accelerate root-cause analysis when pipelines break.
Lineage is the answer to "what broke and what does it affect?" Table-level lineage answers the first question: trace the failure upstream to find the source. Column-level lineage answers the second: identify every downstream column, model, feature, and dashboard that depends on the broken field. Without lineage, a single upstream schema change can require hours of manual discovery to scope the blast radius.
OpenLineage is the right default choice for instrumenting lineage. It is an open specification backed by Airflow, Spark, dbt, and the major cloud providers. Emitting OpenLineage events costs almost nothing -- the Airflow OpenLineage provider instruments every task automatically. The backend (Marquez for self-hosted, or a vendor) is where to spend evaluation effort.
The highest-value use of lineage is impact analysis during schema migrations. Before you rename a column or change a type in a source table, run a lineage query that shows every downstream artifact that reads that column. This turns a multi-day manual audit into a five-minute query. At organizations where this is routine, schema migrations go from high-risk operations to routine ones.
For AI systems, lineage is essential for model debugging. When a model's predictions degrade, the first question is "did the features change?" Column-level lineage between the feature store and source tables answers this directly. When a model is retrained, lineage records which version of each source table was used -- critical for reproducing training runs and auditing model behavior. Lineage is not a nice-to-have for ML pipelines; it is a prerequisite for operating them responsibly.
The open standard for lineage metadata. If you instrument your Airflow DAGs and Spark jobs with the OpenLineage integration, you get lineage for free with no vendor lock-in.
The reference implementation that stores and serves OpenLineage events. Run it locally to see what lineage data looks like before committing to a vendor.
dbt's native lineage graph covers your dbt DAG end-to-end. Exposures extend it to downstream consumers (dashboards, ML models). Use this before reaching for a separate lineage tool.
Table-level lineage tells you which tables depend on which. Column-level lineage tells you which field values flow from which source columns -- necessary for impact analysis of schema changes.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.