Skip to content

Reproduce dbt's Semantic Layer benchmark

dbt Labs published a benchmark of LLM-generated Semantic Layer queries on an ACME Insurance dataset, run through dbt Cloud. evaldata reproduces the dataset, questions, and model locally on DuckDB as pytest tests. It scores them with resolve-and-compare, run-and-compare, and an optional grader.

The result

gpt-5.3-codex answers each question at the model's default temperature (many reasoning models do not accept temperature=0), so each question is run 10 times and the pass rate reported. gpt-4o-mini runs once at temperature 0.

Corpus Model Accuracy
ACME (dbt's suite) openai/gpt-5.3-codex 96.4% (106/110)
ACME (dbt's suite) openai/gpt-4o-mini 45.5% (5/11)
jaffle (authored) openai/gpt-5.3-codex 100.0% (320/320)
jaffle (authored) openai/gpt-4o-mini 31.2% (10/32)

Of the 110 ACME runs, 44 were decided by the resolve-and-compare tier and 62 by run-and-compare; the judge was never needed. --no-judge yields the same number with no grader call.

How the reproduction is built

  • Dataset and questions. dbt's ACME project (dbt-labs/semantic-layer-llm-benchmarking) is ported to dbt-duckdb, and its exact 11-question suite (dbt-labs/dbt-llm-sl-bench) is committed as acme_bench.yml. Both are Apache-2.0 (see the fixture's NOTICE.md).
  • Gold-query check. Each question's gold MetricFlow query returns the same rows as dbt's gold SQL on the same warehouse; the e2e asserts this row for row.
  • Result comparison. The run-and-compare tier aligns columns by value, compares numbers within a tolerance, and accepts a redundant extra grouping column. A correct answer with a different metric label or number format still passes.

Run it yourself

From a clone of the repository, with an OpenAI key in the environment:

# Build the ACME fixture (seeds -> models -> semantic manifest).
uv run --group fixtures bash tests/dbt/fixtures/acme_insurance/regen.sh
uv run --group fixtures dbt build \
  --project-dir tests/dbt/fixtures/acme_insurance \
  --profiles-dir tests/dbt/fixtures/acme_insurance

# Score the suite. Reasoning models need --temperature 1.
uv run --all-extras --group fixtures evaldata sl-bench tests/dbt/fixtures/acme_insurance \
  --model openai/gpt-5.3-codex --temperature 1 \
  --cases tests/dbt/fixtures/acme_insurance/acme_bench.yml \
  --json acme.json

sl-bench runs the suite once. The table above reports the mean over 10 runs, so a single run can vary by a few points at this temperature.

Next steps