When dbt tests are not enough: how to use Great Expectations and Soda for rich data validation, profiling, and documentation.
Great Expectations and Soda solve a problem dbt tests do not fully address: validating data that your dbt project does not own. Raw source tables loaded by Fivetran or Airbyte, upstream data from partner teams, CSV uploads from business teams -- these need validation at ingestion, before dbt ever touches them. That is the primary use case where a dedicated validation framework beats dbt tests.
Great Expectations has the richer ecosystem -- it profiles datasets automatically, generates human-readable documentation, and has connectors for dozens of data sources. The cost is complexity: the Expectation/Suite/Checkpoint/DataSource/RuntimeBatchRequest object hierarchy is steep. Plan for a half-day to understand it before you write your first production suite. It repays the investment at scale.
Soda's design philosophy is the opposite: checks are YAML files readable by anyone on the data team, not just engineers. A SodaCL check is three lines. This makes validation visible in code reviews, easy to edit without Python expertise, and simple to explain to stakeholders. For teams where data analysts write checks alongside engineers, Soda's approach is significantly more maintainable.
The common mistake with both tools is over-validation. Teams write hundreds of expectations, many of them redundant or trivially satisfied. This creates noise: when checks fail constantly on irrelevant columns, engineers start ignoring failures. Better to have 20 checks that represent genuine data quality risks and that someone will act on than 200 checks that fail intermittently and get tuned out.
The most mature Python-native data validation library. The quickstart is the fastest way to understand the Expectation/Suite/Checkpoint model before committing.
Soda's YAML-first approach (SodaCL) is easier to read and review in pull requests than Python-based Great Expectations. Better choice for teams with strong SQL skills but lighter Python.
Auto-generated documentation from your validation runs. Shows which expectations passed/failed on each batch. Valuable for communicating data quality to non-engineers.
Complete reference for Soda's check language. More readable than SQL-based assertions, easier to version-control, and designed to be written by data analysts, not just engineers.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.