Text-to-SQL Benchmarks & Evaluation

How to evaluate text-to-SQL systems: what academic benchmarks measure, why they are insufficient, and how to build your own eval dataset.

Spider and BIRD are calibration tools, not production benchmarks. They tell you how a model performs on a standardized schema with standardized queries -- not on your schema with your users' questions. A model that scores 85% on BIRD may score 40% on your internal queries, or 90%, depending almost entirely on how well your schema is documented and how complex your typical questions are. Use benchmarks to compare models; use your own eval to measure production quality.

Execution-match accuracy is the metric that matters in production. Exact-match (does the generated SQL string match the reference SQL string?) is too strict -- there are multiple correct SQL queries for any question. Execution-match (does the query return the same result as the reference query?) is what users experience. Build your eval infrastructure around execution match from the start.

Building your internal text-to-SQL eval dataset is data engineering work. You need: 50 to 200 real user questions drawn from actual usage or user research, paired with correct SQL written by someone who knows the schema, and a snapshot of the expected result set. Treat this dataset like a first-class data product -- version it, maintain it as the schema evolves, and run every model or prompt change through it. Without it, you are flying blind.

Key resources

• BIRD Benchmark -- Big Bench for Large-Scale Database Grounded Text-to-SQL

  The current standard academic benchmark. Uses real databases with dirty data and external knowledge requirements -- closer to production conditions than Spider. The leaderboard shows what state-of-the-art systems actually achieve.

• Spider 1.0 -- Cross-Domain Text-to-SQL Benchmark (Yale)

  The dataset that defined the field. Still useful for understanding the structure of a rigorous text-to-SQL eval: diverse domains, graduated difficulty levels, and both exact-match and execution-match metrics.

• sql-eval -- Open-Source Text-to-SQL Evaluation Framework (Defog)

  A practical evaluation framework you can run against your own schema and queries. Supports execution-based evaluation (does the query return the right result?) rather than just string matching. This is the approach to copy for your internal eval.