Skip to content

evaldata

Evaluate AI-generated SQL with pytest.

evaldata runs text-to-SQL evals in your existing test suite.

It checks semantic equivalence of SQL queries, diffs result sets in your warehouse, and uses an LLM judge for ambiguous cases.

Why evaldata

  • Semantic equivalence. Parse both queries, normalize their ASTs, and compare canonical forms. No execution, no LLM — when it can't confirm, it returns unknown.
  • Execution in your warehouse. Run the query on DuckDB, Postgres, Databricks, Snowflake, or BigQuery and compare the results, accounting for row order, NULLs, float tolerance, and types.
  • It's just pytest. Every eval is a test, run in your suite and your CI on every PR. No new runner, notebook, or dashboard.
  • An LLM judge when you need one. For ambiguous questions, missing reference answers, or explanations to grade, use a grader model with explicit criteria.

Install

uv add evaldata                # core (includes the DuckDB adapter)
uv add "evaldata[postgres]"    # + Postgres adapter
uv add "evaldata[databricks]"  # + Databricks adapter
uv add "evaldata[snowflake]"   # + Snowflake adapter
uv add "evaldata[bigquery]"    # + BigQuery adapter
uv add "evaldata[cortex]"      # + Snowflake Cortex Analyst solver
uv add "evaldata[litellm]"     # + litellm, to call a model from PromptSolver

DuckDB, Postgres, Databricks, Snowflake, and BigQuery are the adapters available today.

More use cases

Where to go next