Evals & cases

core

Core orchestration: the runner, the pytest-facing assert_eval, and run_benchmark.

BenchmarkSummary

Bases: BaseModel

The aggregate outcome of running a set of cases: pass count and accuracy.

CaseEvaluation dataclass

CaseEvaluation(
    report: CaseReport,
    output: SolverOutput,
    result: ExecutionResult | None,
    failures: list[ScoreResult],
)

The outcome of running one case through a solver + platform + scorers, without raising.

Attributes:

Name	Type	Description
report	CaseReport	The case outcome (identity, pass/fail, per-scorer results or a solver error).
output	SolverOutput	The solver output, carrying either the SQL or a typed SolverError.
result	ExecutionResult | None	The executed model result, or None when the solver itself failed.
failures	list[ScoreResult]	The failing scorer results (empty when the case passed or the solver failed).

assert_eval

assert_eval(
    case: EvalCase,
    solver: Solver,
    *,
    scorers: Sequence[Scorer],
    adapter: PlatformAdapter | None = None,
) -> None

Run case through solver + a platform adapter + scorers; raise on any failure.

Records the case outcome to the run accumulator, then raises if the solver failed or any scorer failed. Raising is pytest's failure protocol.

Parameters:

Name	Type	Description	Default
case	EvalCase	The eval case to run.	required
solver	Solver	The solver that produces SQL for the case.	required
scorers	Sequence[Scorer]	Scorers applied to the execution result; all must pass.	required
adapter	PlatformAdapter | None	A platform adapter to execute against. If omitted, one is resolved and session-cached from case.platform.	None

Raises:

Type	Description
AssertionError	If the solver fails or any scorer fails, carrying a composed diagnostic.

evaluate_case

evaluate_case(
    case: EvalCase,
    solver: Solver,
    *,
    scorers: Sequence[Scorer],
    adapter: PlatformAdapter | None = None,
) -> CaseEvaluation

Run case through solver + a platform adapter + scorers, returning the outcome.

Solves the case, executes the produced SQL, and scores the result with each scorer. The adapter is the explicitly passed adapter if given, otherwise resolved (and session-cached) from case.platform. Execution is bounded by case.cost_budget's max_seconds: an overrunning query is cancelled and scored as an execution failure. Does not raise on failure and does not record to the run accumulator — callers decide how to surface the result.

Parameters:

Name	Type	Description	Default
case	EvalCase	The eval case to run.	required
solver	Solver	The solver that produces SQL for the case.	required
scorers	Sequence[Scorer]	Scorers applied to the execution result; all must pass for the case to pass.	required
adapter	PlatformAdapter | None	A platform adapter to execute against. If omitted, one is resolved and session-cached from case.platform.	None

Returns:

Type	Description
CaseEvaluation	A CaseEvaluation carrying the case report, the solver output, the execution result
CaseEvaluation	(or None on solver error), and any failing scorer results.

Raises:

Type	Description
AssertionError	If the solver returns neither output nor error (unreachable — the SolverOutput validator guarantees exactly one is set).

run_benchmark

run_benchmark(
    cases: Iterable[EvalCase],
    solver: Solver,
    *,
    scorers: Sequence[Scorer],
    limit: int | None = None,
) -> BenchmarkSummary

Run cases through solver + scorers and return aggregate accuracy.

Each case runs through evaluate_case (so a solver error or any failing scorer marks the case failed), and adapters resolve per case.platform through the session cache. Unlike assert_eval, this neither raises nor records to the run accumulator — it returns the aggregate for the caller to print or persist.

Parameters:

Name	Type	Description	Default
cases	Iterable[EvalCase]	The eval cases to run, in order.	required
solver	Solver	The solver under test.	required
scorers	Sequence[Scorer]	Scorers applied to each case; all must pass for the case to count as passed.	required
limit	int | None	Run at most this many cases, or None for all of them.	None

Returns:

Type	Description
BenchmarkSummary	A BenchmarkSummary with the total, the passed count, the accuracy
BenchmarkSummary	(passed / total, or 0.0 when no cases ran), and every case report.

loaders

Loaders: build EvalCases from authoring surfaces (Python decorator, benchmark datasets).

load_bird

load_bird(
    root: str | Path,
    *,
    split: str = "dev",
    include_evidence: bool = True,
) -> Iterator[EvalCase]

Yield BIRD cases from an unzipped BIRD dataset directory.

Reads <root>/<split>.json (records with db_id, question, evidence, SQL, and on the dev split question_id/difficulty) and pairs each question with its db_id's SQLite database at <root>/<split>_databases/<db_id>/<db_id>.sqlite. The gold SQL becomes the case's GoldQuery, scored with ExecutionAccuracy. The dataset is not redistributed; download it from the BIRD project and pass its directory here.

Parameters:

Name	Type	Description	Default
root	str | Path	Path to the unzipped BIRD dataset directory.	required
split	str	The split file stem to read ("dev" reads dev.json and dev_databases/).	'dev'
include_evidence	bool	When True (default), fold each record's external-knowledge evidence into the question input. The raw evidence is kept in metadata either way.	True

Yields:

Type	Description
EvalCase	One EvalCase per question, in file order, on the sqlite platform.

load_spider

load_spider(
    root: str | Path, *, split: str = "dev"
) -> Iterator[EvalCase]

Yield Spider cases from an unzipped Spider dataset directory.

Reads <root>/<split>.json (a list of {db_id, question, query} records) and pairs each question with its db_id's SQLite database at <root>/database/<db_id>/<db_id>.sqlite. The gold query becomes the case's GoldQuery, scored with ExecutionAccuracy. The dataset is not redistributed; download it from the Spider project and pass its directory here.

Parameters:

Name	Type	Description	Default
root	str | Path	Path to the unzipped Spider dataset directory.	required
split	str	The split file stem to read ("dev" reads dev.json).	'dev'

Yields:

Type	Description
EvalCase	One EvalCase per question, in file order, on the sqlite platform.

eval_case

eval_case(
    *,
    input: str,
    expected: dict[str, Any] | Expected,
    platform: PlatformRef,
    id: str | None = None,
    metadata: dict[str, Any] | None = None,
    comparison: ComparisonConfig | None = None,
    cost_budget: CostBudget | None = None,
) -> Callable[[_TestFn], _TestFn]

Attach an EvalCase to a test function for the case fixture to inject.

Parameters:

Name	Type	Description	Default
input	str	The natural-language question / instruction under test.	required
expected	dict[str, Any] | Expected	The expected outcome — a typed Expected or a dict coerced to one.	required
platform	PlatformRef	A PlatformRef (build one with duckdb_platform / postgres_platform).	required
id	str | None	Case identifier; defaults to the decorated function's name.	None
metadata	dict[str, Any] | None	Optional free-form tags/owner/source metadata.	None
comparison	ComparisonConfig | None	Optional result-set comparison rules; defaults to ComparisonConfig().	None
cost_budget	CostBudget | None	Optional ceiling on platform resource consumption for the case.	None

Returns:

Type	Description
Callable[[_TestFn], _TestFn]	A decorator that records the case and returns the function unchanged.

read_eval_case

read_eval_case(func: Callable[..., Any]) -> EvalCase | None

Return the EvalCase attached to func by @eval_case, or None.