Pydantic Evals¶

evaluator ¶

SqlEquivalence: a pydantic-evals Evaluator that scores generated SQL by execution.

SqlEquivalence `dataclass` ¶

SqlEquivalence(platform: PlatformRef)

Bases: Evaluator[Any, str, Any]

Score generated SQL by executing it against a warehouse and checking equivalence.

Reads the case's generated SQL from ctx.output and its reference from ctx.expected_output (a gold-SQL str, or a GoldQuery/UntypedResultSet/ TypedResultSet), runs both against platform, and returns an EvaluationReason whose value is the pass/fail and whose reason explains it. Invalid generated SQL scores as a failure rather than raising; a missing or unusable case contract raises ValueError.

Runs safely under Pydantic Evals concurrency: each case acquires its own platform session for the duration of scoring, so max_concurrency parallelizes warehouse execution up to the platform's per-name pool size.

Attributes:

Name	Type	Description
`platform`	`PlatformRef`	The platform to execute the generated and reference SQL against.

evaluate_async `async` ¶

evaluate_async(
    ctx: EvaluatorContext[Any, str, Any],
) -> EvaluationReason

Score ctx on a worker thread so Pydantic Evals runs cases in parallel.

Parameters:

Name	Type	Description	Default
`ctx`	`EvaluatorContext[Any, str, Any]`	The evaluation context passed straight to `evaluate`.	required

Returns:

Type	Description
`EvaluationReason`	The `EvaluationReason` produced by `evaluate`.

evaluate ¶

evaluate(
    ctx: EvaluatorContext[Any, str, Any],
) -> EvaluationReason

Score ctx.output against ctx.expected_output by executing both on platform.