Anomaly Detection on Pipelines

How to detect silent data failures -- volume drops, distribution shifts, and freshness gaps -- automatically, without predefining every rule.

Rule-based data quality checks have a ceiling: they catch failures you anticipated. Anomaly detection catches failures you did not. A volume drop of 40% at 3am, a null rate that doubles after a source schema change, a distribution that shifts because an upstream ETL job started filtering differently -- none of these require a prewritten rule if you are tracking statistical baselines.

The core technique is straightforward: for each metric you care about (row count, null rate, mean, p95 of a column), compute a rolling baseline over recent history. Alert when the current value deviates beyond a threshold -- typically 2 to 3 standard deviations or a fixed percentage. The hard part is not the statistics; it is choosing the right metrics and calibrating thresholds to avoid alert fatigue.

Freshness and volume are the two metrics to instrument first. They catch the broadest class of failures -- upstream table stopped loading, job silently errored, scheduler drift. Distribution monitoring comes second and is higher-signal but noisier: a column mean shifting 10% may be a bug or may be a real business change. Context from data lineage (what upstream jobs ran?) is essential for triaging distribution alerts.

For ML pipelines, anomaly detection on training data features is a precondition for model reliability. Feature drift -- where a feature's distribution shifts between training and inference -- is one of the primary causes of model degradation in production. Monitoring feature distributions at ingestion time, not just at model evaluation time, catches the drift hours earlier and gives data teams time to fix the source before a model retrain bakes in the error.

Key resources

• Data Quality Fundamentals Ch. 4 -- Anomaly Detection

  The clearest treatment of the statistical methods behind pipeline anomaly detection: z-scores, interquartile range, and time-series forecasting applied to data quality metrics. Read before evaluating any vendor.

• Elementary Data -- Anomaly Detection for dbt

  Open-source anomaly detection that plugs directly into dbt. Monitors row count, null rate, and column distributions out of the box -- without writing custom SQL.

• Monte Carlo -- How Anomaly Detection Works

  Explains the production approach: train a baseline on historical table behavior, then alert when current metrics fall outside expected bounds. Useful for calibrating your own alerting thresholds.