Scorers¶

scorers ¶

Scorers: pluggable pass/fail checks.

Ships ResultSetEquivalence, ExpectationSuiteScorer, SemanticEquivalence, and the LLM-as-judge LlmJudge, plus the FirstDecisive combinator and the equivalence-preset compositions it powers.

Scorer ¶

Bases: Protocol

Produces a ScoreResult from a case, its solver output, and the execution result.

score ¶

score(
    case: EvalCase,
    output: SolverOutput,
    result: ExecutionResult,
    *,
    context: ScoreContext,
) -> ScoreResult

Decide pass/fail with diagnostics for case given output, result, and context.

FirstDecisive ¶

FirstDecisive(scorers: Sequence[Scorer])

Runs member scorers in order; the first that decides wins, else the last result stands.

The combinator continues only while a member is inconclusive; the first member to return a decisive verdict (pass or fail) is returned immediately, so a later member cannot override an earlier one's decision. If every member is inconclusive, the last member's result is returned, so its diagnostics (e.g. a diff) surface.

Bind the combinator to an ordered list of member scorers.

Parameters:

Name	Type	Description	Default
`scorers`	`Sequence[Scorer]`	The member scorers, in priority order.	required

Raises:

Type	Description
`ValueError`	If `scorers` is empty.

score ¶

score(
    case: EvalCase,
    output: SolverOutput,
    result: ExecutionResult,
    *,
    context: ScoreContext,
) -> ScoreResult

Run members in order, returning the first decisive result (later members not consulted), else the last.

The returned result carries a metadata["first_decisive"] trail of {"scorer", "passed", "verdict"} for each member that actually ran.

Parameters:

Name	Type	Description	Default
`case`	`EvalCase`	The eval case, forwarded to each member.	required
`output`	`SolverOutput`	The solver output, forwarded to each member.	required
`result`	`ExecutionResult`	The executed model result, forwarded to each member.	required
`context`	`ScoreContext`	The score context, forwarded to each member.	required

Returns:

Type	Description
`ScoreResult`	The first decisive member's `ScoreResult` (verdict `pass` or `fail`), or the last
`ScoreResult`	member's result when every member is `inconclusive`, with the `"first_decisive"` trail
`ScoreResult`	merged into its metadata.

ScoreContext `dataclass` ¶

ScoreContext(queries: QueryRunner)

Per-case capabilities injected into Scorer.score.

Attributes:

Name	Type	Description
`queries`	`QueryRunner`	The budget-aware runner for derived SQL against the case's platform.

ExecutionAccuracy ¶

ExecutionAccuracy(
    *,
    row_order: Literal[
        "when_ordered", "ignore"
    ] = "when_ordered",
    multiplicity: Literal["multiset", "set"] = "multiset",
    column_alignment: Literal[
        "by_position", "by_value"
    ] = "by_position",
)

Scores a case by execution accuracy: does the model's SQL return the gold query's rows?

Configure the comparison semantics.

Parameters:

Name	Type	Description	Default
`row_order`	`Literal['when_ordered', 'ignore']`	`"when_ordered"` (default) makes the comparison order-sensitive iff the gold query carries a top-level `ORDER BY`; `"ignore"` always compares order-insensitively.	`'when_ordered'`
`multiplicity`	`Literal['multiset', 'set']`	`"multiset"` (default) requires duplicate rows to match by count (bag semantics); `"set"` compares distinct rows (set equality).	`'multiset'`
`column_alignment`	`Literal['by_position', 'by_value']`	`"by_position"` (default) compares columns positionally; `"by_value"` ignores column order by searching for a column permutation whose values make the result sets match. Use `"by_value"` when the gold and model queries label columns differently, so columns can only be aligned by content, not by name; it requires both result sets to have the same number of columns.	`'by_position'`

score ¶

score(
    case: EvalCase,
    output: SolverOutput,
    result: ExecutionResult,
    *,
    context: ScoreContext,
) -> ScoreResult

Compare the model result against the gold query's executed rows.

Parameters:

Name	Type	Description	Default
`case`	`EvalCase`	The eval case; its `expected` must be a `GoldQuery`.	required
`output`	`SolverOutput`	The solver output (part of the `Scorer` protocol; unused here).	required
`result`	`ExecutionResult`	The executed model result to score.	required
`context`	`ScoreContext`	The score context, carrying the budget-aware `QueryRunner`.	required

Returns:

Type	Description
`ScoreResult`	A passing `ScoreResult` (`basis="observed"`) when the result sets match under the
`ScoreResult`	configured semantics, else a failing one carrying a `ResultSetDiff`. A failed model
`ScoreResult`	query, or a failed gold query (`metadata["gold_query_failed"]`), yields a failing
`ScoreResult`	result with an explanation.

Raises:

Type	Description
`TypeError`	If `case.expected` is not a `GoldQuery`.

ExpectationSuiteScorer ¶

Scores a case by checking its executed result against each Expectation in its suite.

score ¶

score(
    case: EvalCase,
    output: SolverOutput,
    result: ExecutionResult,
    *,
    context: ScoreContext,
) -> ScoreResult

Evaluate every expectation in the suite; pass iff all hold.

Row-level checks (row_count, not_null, unique) are pushed into the platform as SQL over the model's query via context.queries; only counts and bounded failing-row samples are read back. Schema checks (column_presence, column_type) read the result's schema metadata and run no query.

Parameters:

Name	Type	Description	Default
`case`	`EvalCase`	The eval case, carrying the `ExpectationSuite`.	required
`output`	`SolverOutput`	The solver output (part of the `Scorer` protocol; unused here).	required
`result`	`ExecutionResult`	The executed result; its schema is used by the schema checks.	required
`context`	`ScoreContext`	The score context, carrying the budget-aware `QueryRunner`.	required

Returns:

Type	Description
`ScoreResult`	A `ScoreResult` that passes when all expectations hold. `outcomes` carries one
`ScoreResult`	`ExpectationOutcome` per expectation (passing and failing alike); on failure
`ScoreResult`	`explanation` lists each unmet expectation, derived from those outcomes. A failed
`ScoreResult`	model query yields a failing result with an explanation and no outcomes; a failed
`ScoreResult`	derived query fails only its own expectation's outcome.

Raises:

Type	Description
`TypeError`	If `case.expected` is not an `ExpectationSuite`.

JudgeExample ¶

Bases: BaseModel

A few-shot anchor: a graded output and the score it should receive.

input and expected_output are optional context; only the fields that are present are rendered into the prompt.

LlmJudge ¶

LlmJudge(
    *,
    model: str | Llm,
    criteria: str,
    steps: Sequence[str] | None = None,
    examples: Sequence[JudgeExample] | None = None,
    rubric: Sequence[RubricBand] | None = None,
    threshold: float = 0.5,
    temperature: float | None = 0.0,
    timeout: float | None = None,
    show: Sequence[JudgeField] | None = None,
)

LLM-as-judge Scorer: a grader model scores the case against authored criteria.

The grader's 0-1 score is compared to a threshold for the pass/fail verdict; the score and rationale are recorded. A provider failure or a malformed reply yields an inconclusive result.

Configure the judge.

Parameters:

Name	Type	Description	Default
`model`	`str \| Llm`	A litellm grader-model identifier (separate from any solver model), or an `Llm` to use directly. `temperature` and `timeout` apply only to the model-string path.	required
`criteria`	`str`	The natural-language standard the grader scores the case against.	required
`steps`	`Sequence[str] \| None`	Ordered evaluation steps the grader should work through, rendered as a numbered block. Omitted from the prompt when absent.	`None`
`examples`	`Sequence[JudgeExample] \| None`	Few-shot anchors mapping graded outputs to scores. Omitted from the prompt when absent.	`None`
`rubric`	`Sequence[RubricBand] \| None`	Scoring bands that describe what each score range means. Omitted from the prompt when absent.	`None`
`threshold`	`float`	The minimum score (inclusive) for a passing verdict. Defaults to `0.5`.	`0.5`
`temperature`	`float \| None`	Sampling temperature; `None` leaves the provider default. Defaults to `0.0` for deterministic grading.	`0.0`
`timeout`	`float \| None`	Per-request timeout in seconds.	`None`
`show`	`Sequence[JudgeField] \| None`	The case fields to offer the grader, each included only when available. Defaults to all of `input`, `actual_output`, and `expected_output`.	`None`

score ¶

score(
    case: EvalCase,
    output: SolverOutput,
    result: ExecutionResult,
    *,
    context: ScoreContext,
) -> ScoreResult

Grade case with the grader model and return a graded ScoreResult.

Builds a prompt from the criteria and the selected available fields, calls the grader, and maps its score to a verdict against the threshold.

Parameters:

Name	Type	Description	Default
`case`	`EvalCase`	The eval case, supplying the input and (optionally) the expected output.	required
`output`	`SolverOutput`	The solver output (part of the `Scorer` protocol; unused here).	required
`result`	`ExecutionResult`	The executed model result (part of the `Scorer` protocol; unused here).	required
`context`	`ScoreContext`	The score context, supplying the model's SQL.	required

Returns:

Type	Description
`ScoreResult`	A `ScoreResult` whose verdict is pass or fail with the graded score and rationale,
`ScoreResult`	or inconclusive when no verdict could be reached.

RubricBand ¶

Bases: BaseModel

A scoring band: a [min_score, max_score] range and what it describes.

QueryRunner ¶

QueryRunner(
    adapter: PlatformAdapter,
    model_sql: Sql,
    dialect: Dialect,
    budget: float | None,
)

Runs derived SQL against a case's platform, drawing on a shared cost budget.

Holds a live adapter, the model's SQL, the case dialect, and a remaining-time pool seeded from the case budget. Each completed query decrements the pool by its latency_seconds; once the pool is exhausted, further runs short-circuit to an ExecutionResult carrying an error without touching the adapter. A None budget means the pool is unbounded.

Bind the runner to a platform and seed its budget pool.

Parameters:

Name	Type	Description	Default
`adapter`	`PlatformAdapter`	The platform adapter derived queries execute against.	required
`model_sql`	`Sql`	The model's SQL.	required
`dialect`	`Dialect`	The SQLGlot dialect derived queries are built and rendered in.	required
`budget`	`float \| None`	The shared remaining-time pool in seconds, or `None` for unbounded.	required

model_sql `property` ¶

model_sql: Sql

The model's SQL.

dialect `property` ¶

dialect: Dialect

The dialect derived queries are built and rendered in.

run ¶

run(sql: Sql) -> ExecutionResult

Run sql within the remaining budget, decrementing the pool by its latency.

Parameters:

Name	Type	Description	Default
`sql`	`Sql`	The SQL statement to execute.	required

Returns:

Type	Description
`ExecutionResult`	The adapter's `ExecutionResult`, or an `ExecutionResult` with `error` set when
`ExecutionResult`	the budget pool is already exhausted (the adapter is not invoked in that case).

scalar ¶

scalar(sql: Sql) -> ScalarResult

Run sql and read back its single cell, or an error.

Delegates to run, so the budget pool is drawn exactly as for any derived query. An underlying error is propagated; a result that is not exactly one row by one column is itself an error.

Parameters:

Name	Type	Description	Default
`sql`	`Sql`	The SQL statement to execute; expected to return one row and one column.	required

Returns:

Type	Description
`ScalarResult`	A `ScalarResult` carrying the single cell on success, else `error`.

resolved_schema ¶

resolved_schema(
    base: TypedSchema, sql: Sql
) -> TypedSchema | ExecutionError

Return base with its column types resolved to the platform's precise types.

Backends that already report precise types return base unchanged; otherwise the adapter's type probe runs through this runner (drawing on the same budget) and its types align to base's columns by position, preserving names and nullability.

Parameters:

Name	Type	Description	Default
`base`	`TypedSchema`	The schema whose column types to resolve, as `execute` reported them.	required
`sql`	`Sql`	The statement that produced `base`, re-probed for precise types.	required

Returns:

Type	Description
`TypedSchema \| ExecutionError`	`base`, a new `TypedSchema` with precise types, or an `ExecutionError`.

ScalarResult `dataclass` ¶

ScalarResult(
    value: Any | None,
    error: ExecutionError | None,
    latency_seconds: float,
)

The single cell returned by a derived query, or an error.

Attributes:

Name	Type	Description
`value`	`Any \| None`	The single cell value, or `None` when `error` is set.
`error`	`ExecutionError \| None`	The failure, or `None` on success.
`latency_seconds`	`float`	Wall-clock time the underlying query took.

ResultSetEquivalence ¶

Scores a case by diffing its executed result set against its expected result set in SQL.

score ¶

score(
    case: EvalCase,
    output: SolverOutput,
    result: ExecutionResult,
    *,
    context: ScoreContext,
) -> ScoreResult

Compare result against case.expected; pass iff the engine finds them equivalent.

Column reconciliation and type comparison run in Python; row equivalence is pushed into the platform. For authored rows the expected side is materialised as typed literals; for a GoldQuery it is the reference query embedded as a subquery, whose schema is discovered with a zero-row execution and whose rows never reach Python. With an empty match_key, two EXCEPT ALL diffs compute the bag difference (and null_equality="distinct" is rejected). With a non-empty match_key, a FULL OUTER JOIN aligns rows on the key and compares per column — supporting null_equality="distinct", an exact tolerance band, and per-column mismatch counts. Only mismatch counts and bounded samples are read back.

Parameters:

Name	Type	Description	Default
`case`	`EvalCase`	The eval case, carrying the expected result set, comparison config, and platform.	required
`output`	`SolverOutput`	The solver output (part of the `Scorer` protocol; unused here).	required
`result`	`ExecutionResult`	The executed result to compare against the expectation.	required
`context`	`ScoreContext`	The score context, carrying the budget-aware `QueryRunner`.	required

Returns:

Type	Description
`ScoreResult`	A `ScoreResult` that passes when the result set matches the expectation. A failed
`ScoreResult`	model query, a failed gold query, a failed derived query, a non-unique or absent
`ScoreResult`	`match_key`, or (keyless) `null_equality="distinct"` each yield a failing result
`ScoreResult`	with an explanation.

Raises:

Type	Description
`TypeError`	If `case.expected` is not one of `UntypedResultSet`, `TypedResultSet`, or `GoldQuery`.

AstEquivalence ¶

Confirms equivalence when both queries normalize to the same SQLGlot syntax tree.

Matching normalized trees yield "equivalent"; anything else (trees differ, a parse failure, or input that is not exactly one statement) yields "unknown", never a refutation. The normalization is schema-free: it fully reassociates commutative arithmetic (+/*), boolean/bitwise chains, and IN-list order; other unhandled equivalences fall through as "unknown".

judge ¶

judge(
    case: EvalCase,
    output: SolverOutput,
    result: ExecutionResult,
    *,
    context: ScoreContext,
) -> SemanticVerdict

Compare the model and gold queries' normalized syntax trees.

Parameters:

Name	Type	Description	Default
`case`	`EvalCase`	The eval case; `expected` must be a `GoldQuery` to compare against.	required
`output`	`SolverOutput`	The solver output (unused).	required
`result`	`ExecutionResult`	The executed model result (unused; this check touches no data).	required
`context`	`ScoreContext`	The score context, supplying the model SQL and dialect.	required

Returns:

Type	Description
`SemanticVerdict`	`"equivalent"` when the normalized trees match, else `"unknown"`.

EquivalenceCheck ¶

Bases: Protocol

One way of confirming equivalence by comparing the queries, not their results; never raises.

A check confirms equivalence ("equivalent") or returns "unknown"; it never refutes, so it cannot falsely reject a correct query.

judge ¶

judge(
    case: EvalCase,
    output: SolverOutput,
    result: ExecutionResult,
    *,
    context: ScoreContext,
) -> SemanticVerdict

Judge whether the model's query is equivalent to the case's gold query.

Parameters:

Name	Type	Description	Default
`case`	`EvalCase`	The eval case, whose `expected` carries the gold query.	required
`output`	`SolverOutput`	The solver output.	required
`result`	`ExecutionResult`	The already-executed model result (reused by execution-based checks).	required
`context`	`ScoreContext`	The score context, carrying the model SQL, dialect, and query runner.	required

Returns:

Type	Description
`SemanticVerdict`	A `SemanticVerdict` describing the equivalence decision for this check.

SemanticEquivalence ¶

SemanticEquivalence(
    checks: Sequence[EquivalenceCheck] | None = None,
)

Scores a gold-query case with checks that compare the queries themselves.

It never runs a query and never refutes, so it confirms equivalence or is undecided. The first check that confirms yields a passing result; if none confirm, the result is inconclusive.

Bind the scorer to an ordered list of checks.

Parameters:

Name	Type	Description	Default
`checks`	`Sequence[EquivalenceCheck] \| None`	The checks to run, in priority order; the first that confirms wins. Defaults to `default_equivalence_checks()` when omitted.	`None`

score ¶

score(
    case: EvalCase,
    output: SolverOutput,
    result: ExecutionResult,
    *,
    context: ScoreContext,
) -> ScoreResult

Run the checks in order, stopping at the first that confirms equivalence.

Parameters:

Name	Type	Description	Default
`case`	`EvalCase`	The eval case; `expected` must be a `GoldQuery` (query-vs-query comparison).	required
`output`	`SolverOutput`	The solver output, passed through to each check.	required
`result`	`ExecutionResult`	The executed model result, passed through to each check.	required
`context`	`ScoreContext`	The score context, carrying the model SQL, dialect, and query runner.	required

Returns:

Type	Description
`ScoreResult`	A passing `ScoreResult` when a check confirms equivalence, else an inconclusive
`ScoreResult`	result (no check could confirm).

Raises:

Type	Description
`TypeError`	If `case.expected` is not a `GoldQuery`.

judged_equivalence ¶

judged_equivalence(model: str | Llm) -> FirstDecisive

The query-vs-query check that confirms by structure, else asks an LLM judge.

The case's expected must be a GoldQuery.

Parameters:

Name	Type	Description	Default
`model`	`str \| Llm`	A litellm grader-model identifier, or an `Llm` to use directly.	required

Returns:

Type	Description
`FirstDecisive`	A `FirstDecisive` cascade: `SemanticEquivalence` compares the queries and confirms
`FirstDecisive`	equivalence first; when it cannot, the SQL-equivalence judge decides without running
`FirstDecisive`	either query.

observed_equivalence ¶

observed_equivalence() -> FirstDecisive

The query-vs-query check that confirms by structure, else by running both queries.

The case's expected must be a GoldQuery.

Returns:

Type	Description
`FirstDecisive`	A `FirstDecisive` cascade: `SemanticEquivalence` compares the queries and confirms
`FirstDecisive`	equivalence first; when it cannot, `ResultSetEquivalence` runs both queries and decides
`FirstDecisive`	by diffing their results.

sql_equivalence_judge ¶

sql_equivalence_judge(model: str | Llm) -> LlmJudge

An LlmJudge pre-configured to grade whether two SQL queries are equivalent.

Parameters:

Name	Type	Description	Default
`model`	`str \| Llm`	A litellm grader-model identifier, or an `Llm` to use directly.	required

Returns:

Type	Description
`LlmJudge`	An `LlmJudge` with SQL-equivalence criteria and few-shot examples.

default_equivalence_checks ¶

default_equivalence_checks() -> list[EquivalenceCheck]

The default checks, cheapest and most portable first.

Returns:

Type	Description
`list[EquivalenceCheck]`	A fresh list of the default `EquivalenceCheck`s: just `AstEquivalence`.

Scorers¶

scorers ¶

Scorer ¶

score ¶

FirstDecisive ¶

score ¶

ScoreContext dataclass ¶

ExecutionAccuracy ¶

score ¶

ExpectationSuiteScorer ¶

score ¶

JudgeExample ¶

LlmJudge ¶

score ¶

RubricBand ¶

QueryRunner ¶

model_sql property ¶

dialect property ¶

run ¶

scalar ¶

resolved_schema ¶

ScalarResult dataclass ¶

ResultSetEquivalence ¶

score ¶

AstEquivalence ¶

judge ¶

EquivalenceCheck ¶

judge ¶

SemanticEquivalence ¶

score ¶

judged_equivalence ¶

observed_equivalence ¶

sql_equivalence_judge ¶

default_equivalence_checks ¶

ScoreContext `dataclass` ¶

model_sql `property` ¶

dialect `property` ¶

ScalarResult `dataclass` ¶