Solvers¶

base ¶

Solver Protocol: the contract for the AI system under test (EvalCase -> SolverOutput).

Solver ¶

Bases: Protocol

Produces a SolverOutput for an EvalCase.

solve ¶

solve(case: EvalCase) -> SolverOutput

Produce output (for SQL solvers, the executable SQL) for case.

callable ¶

CallableSolver: adapt a plain function into a Solver (no-LLM, deterministic).

CallableSolver ¶

CallableSolver(fn: Callable[[EvalCase], str])

Wraps a function (EvalCase) -> sql as a Solver.

Store the SQL-producing function fn.

solve ¶

solve(case: EvalCase) -> SolverOutput

Call the wrapped function and return its SQL as SolverOutput.output.

Parameters:

Name	Type	Description	Default
`case`	`EvalCase`	The eval case to solve.	required

Returns:

Type	Description
`SolverOutput`	A `SolverOutput` carrying the SQL produced by the wrapped function.

prompt ¶

PromptSolver: a single-prompt, LLM-backed Solver over the Llm seam.

PromptSolver ¶

PromptSolver(
    model: str | Llm,
    prompt_template: str = DEFAULT_PROMPT_TEMPLATE,
    timeout: float | None = None,
    temperature: float | None = None,
)

Single-prompt LLM Solver: question -> SQL via the Llm seam.

Configure the solver.

Parameters:

Name	Type	Description	Default
`model`	`str \| Llm`	A litellm model identifier (e.g. `"openai/gpt-4o-mini"`), or an `Llm` to use directly. `timeout` and `temperature` apply only to the model-string path.	required
`prompt_template`	`str`	A `str.format_map` template with `{dialect}`, `{input}`, and optional `{schema}` fields. Defaults to `DEFAULT_PROMPT_TEMPLATE`; pass `SCHEMA_PROMPT_TEMPLATE` to inject `case.metadata["schema_ddl"]`.	`DEFAULT_PROMPT_TEMPLATE`
`timeout`	`float \| None`	Per-request timeout in seconds.	`None`
`temperature`	`float \| None`	Sampling temperature; `None` leaves the provider default. Use `0` for deterministic output.	`None`

solve ¶

solve(case: EvalCase) -> SolverOutput

Produce SQL for case, returning a success or a typed SolverError.

Renders the prompt, calls the model, and extracts the SQL. Expected provider failures are mapped to a SolverError in SolverOutput.error.

Parameters:

Name	Type	Description	Default
`case`	`EvalCase`	The eval case to solve.	required

Returns:

Type	Description
`SolverOutput`	A `SolverOutput` carrying either the extracted SQL plus token/latency/cost
`SolverOutput`	telemetry on success, or a typed `SolverError` on an expected failure.