Types¶
Core public Pydantic types for evaldata.
Bases: BaseModel
Serializable reference to a configured data platform connection.
Bases: BaseModel
A SQL column type, canonicalised through SQLGlot for safe equality.
raw is the native string the platform or author produced (truthful, for diffs).
canonical is the dialect-neutral SQLGlot rendering used for comparison, or None
when SQLGlot cannot parse the type. Equality compares canonical when both sides
have one, else falls back to raw.
Accepts a plain string as authoring shorthand: "BIGINT" becomes a SqlType with
raw="BIGINT" and canonical=None. Canonicalisation happens eagerly at the
ingestion boundary (adapters and the EvalCase validator), never at compare time.
classmethod
¶
parse(raw: str, dialect: SQLDialect) -> SqlType
Build a SqlType, canonicalising raw in dialect.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw
|
str
|
The native SQL type string. |
required |
dialect
|
SQLDialect
|
The SQLGlot dialect to parse |
required |
Returns:
| Type | Description |
|---|---|
SqlType
|
A |
SqlType
|
|
Bases: BaseModel
A result-set column: name, SQL type, and tri-state nullability.
Bases: RootModel[list[Column]]
An ordered, duplicate-faithful sequence of typed result-set columns.
Wraps a list[Column] and serialises as a plain JSON array. Name lookup is not
offered: result sets may repeat column names, so the convenience accessors are
positional and duplicate-safe. Engines that report no result-column types produce an
UntypedSchema instead.
Bases: RootModel[list[str]]
An ordered sequence of result-column names with no type information.
Produced by engines whose driver reports no result-column types (e.g. SQLite). Carries
only names; type comparison against an UntypedSchema abstains rather than refuting.
Bases: BaseModel
Expected outcome as concrete rows without column types: value comparison only.
Bases: BaseModel
Expected outcome as concrete rows plus a schema: value and type comparison.
Bases: BaseModel
Expected outcome as a gold/reference query whose executed result IS the expected answer.
The expected answer is whatever sql returns when executed, not literal rows authored
up front.
Bases: BaseModel
The result set must contain exactly this many rows.
Bases: BaseModel
The result set must contain at least these columns.
Bases: BaseModel
A named column must have the given SQL type.
Bases: BaseModel
A named column must contain no NULL values.
Bases: BaseModel
A named column's values must be distinct.
Bases: BaseModel
Expected outcome specified as a suite of expectations the result set must satisfy.
Bases: BaseModel
Rules for deciding whether two result sets are equivalent.
A non-empty match_key selects the keyed FULL OUTER JOIN comparison: rows are
aligned on the key columns and compared per remaining column, enabling
null_equality="distinct", an exact abs(actual - expected) <= float_tolerance
band, and per-column mismatch counts. An empty match_key uses the keyless bag
(EXCEPT ALL) comparison.
Bases: BaseModel
Per-eval-case ceiling on platform resource consumption.
Bases: BaseModel
One AI-evaluation case: an input with an expected outcome and a platform to run against.
Bases: BaseModel
Base for the typed error types.
Holds the fields every typed error shares. Subclasses add a kind discriminator and any
domain-specific structured fields (an ExecutionError's sqlstate, a SolverError's
provider). cause keeps the original exception — and its traceback — for in-process
debugging and logging; it is excluded from serialization, so reports carry only the
structured surface.
Bases: BaseModel
A Solver's output: either a successful output artifact or an error.
Exactly one of output/error is set. For SQL solvers, output is the SQL to run.
Bases: Error
A typed failure from running SQL against a platform.
condition carries the driver's error class/code string (e.g. Spark's
TABLE_OR_VIEW_NOT_FOUND) when it exposes one, since not every engine reports SQLSTATE.
Bases: BaseModel
The result of running SQL against a platform: returned rows plus execution measurements.
Bases: BaseModel
A column whose actual type in the result set differs from the expected type.
Bases: BaseModel
Per-column count of rows whose value in the actual result set differs from the expected value.
Bases: BaseModel
Structured difference between an actual result set and an expected result set.
Bases: BaseModel
The result of checking one Expectation against an executed result.
expected/actual carry the compared scalars (a row count, a column's type
raw); count carries the number of offending elements (NULL or duplicate
values); sample_rows carries a bounded sample of the offending rows (empty
unless the expectation failed and the kind produces one); detail is the
human-readable failure message, None when the expectation holds. Which fields
are populated depends on the expectation kind.
Bases: BaseModel
The outcome of running a Scorer against an EvalCase: a verdict plus diagnostics.
score and basis must be absent when verdict is "inconclusive" — an undecided result
carries no evidence.
Bases: BaseModel
One equivalence check's judgment on whether two queries are equivalent.
A verdict never carries a diff; a refutation surfaces as a result-set ScoreResult.diff.