Skip to content

Score with an LLM judge

Not every answer can be checked by comparing rows or normalizing SQL ASTs. Does the SQL read clearly? Does it follow your team's style? Do two queries return the same thing when you have no warehouse to run them on? LlmJudge answers questions like these: it asks a grader model to score the case against criteria you write, and turns that score into a pass/fail verdict.

What the judge does

You give the judge a standard to grade against, written as plain text in criteria. It builds a prompt from that standard and the case, asks the grader model to reason about how well the case meets it and give a score from 0.0 to 1.0, then compares that score to a threshold to decide the verdict:

from evaldata import LlmJudge

judge = LlmJudge(
    model="openai/gpt-4o-mini",
    criteria="The SQL answers the question, reads clearly, and uses no SELECT *.",
    threshold=0.7,
)

The grader's score is in result.score and its rationale is in result.explanation. The result has basis="judged", which marks it as a probabilistic judgment rather than a proof.

Pass it to assert_eval like any other scorer:

assert_eval(case, solver, scorers=[judge])

Inconclusive, not a guess

A judge can fail to reach a verdict if the provider call errors or the response doesn't parse. The result is then inconclusive (verdict="inconclusive") and carries the failure in its explanation. Inconclusive means the judge couldn't decide; it doesn't mean the answer is wrong.

Configure the judge

criteria alone is often enough. Four optional arguments control the grading prompt and pass threshold:

  • steps: an ordered checklist the grader works through before scoring, rendered as a numbered block.
  • rubric: score bands that describe each range.
  • examples: few-shot anchors, each a graded output with its score and reason.
  • threshold: the minimum score (inclusive) for a pass. Defaults to 0.5.
from evaldata import JudgeExample, LlmJudge, RubricBand

judge = LlmJudge(
    model="openai/gpt-4o-mini",
    criteria="The SQL answers the question and is idiomatic for an analytics team.",
    steps=[
        "Check the query returns the columns the question asks for.",
        "Check it filters and aggregates correctly.",
        "Judge readability: aliases, no SELECT *, sensible formatting.",
    ],
    rubric=[
        RubricBand(min_score=0.0, max_score=0.4, description="Wrong result or unreadable."),
        RubricBand(min_score=0.4, max_score=0.8, description="Correct result, rough style."),
        RubricBand(min_score=0.8, max_score=1.0, description="Correct and idiomatic."),
    ],
    examples=[
        JudgeExample(
            actual_output="SELECT name FROM customers WHERE country = 'US'",
            score=1.0,
            reason="Correct filter, named column, clear.",
        ),
    ],
    threshold=0.8,
)

Unset rubric and examples values are not included in the prompt.

Choose what the grader sees

By default, the judge shows the grader these case fields when they are available:

  • input: the case's question.
  • actual_output: the SQL the solver produced.
  • expected_output: the gold query's SQL, when the case has a GoldQuery expected.

Use show to hide fields that would bias or distract the grader. For example, grade the output without revealing the reference answer:

judge = LlmJudge(
    model="openai/gpt-4o-mini",
    criteria="The SQL answers the question.",
    show=["input", "actual_output"],   # withhold the expected query
)

Choosing a grader model

The grader is separate from the solver. model takes any litellm identifier (or an Llm to use directly), so the grader can run on a different provider from the model being evaluated.

The grader must support structured output: the judge requests a JSON {score, reason} response and treats a malformed one as inconclusive. Grading defaults to temperature=0.0 for repeatable scores; pass temperature=None to leave the provider default, or a timeout to bound the call.

Use fixed judge responses in CI

A judged eval calls a model, which costs money and can vary between runs. Add a conftest.py next to your test that returns the structured {score, reason} response the judge expects:

"""Mocked grader replies for the LLM-judge example, so it runs without a key or network."""

import json
from typing import Any

import litellm
import pytest

_EQUIVALENT = json.dumps({"reason": "the CTE wraps the same filter", "score": 1.0})
_NOT_EQUIVALENT = json.dumps({"reason": "country = 'GB' selects different customers", "score": 0.0})


def _mock_reply(messages: list[dict[str, Any]]) -> str:
    """Pick the grader reply for a request by matching the actual SQL under judgement.

    Args:
        messages: The chat messages of the request, whose prompt carries the actual SQL.

    Returns:
        A `JudgeReply`-shaped `{"reason", "score"}` JSON string.
    """
    prompt = " ".join(m.get("content", "") for m in messages)
    return _NOT_EQUIVALENT if "country = 'GB'" in prompt else _EQUIVALENT


@pytest.fixture(autouse=True)
def _mock_completion(monkeypatch: pytest.MonkeyPatch) -> None:
    """Patch grader calls to return a deterministic structured reply per case, with no network."""
    real_completion = litellm.completion

    def fake(**kwargs: Any) -> Any:
        return real_completion(**kwargs, mock_response=_mock_reply(kwargs["messages"]))

    monkeypatch.setattr("litellm.completion", fake)
    monkeypatch.setattr("litellm.supports_response_schema", lambda **_: True)

conftest.py makes the eval use fixed judge responses:

"""LLM-judge example evals: `judged_equivalence` has an LLM grade the SQL the syntax check can't confirm."""

import os
import tempfile
from collections.abc import Iterator
from pathlib import Path

import duckdb
import pytest

from evaldata import CallableSolver, EvalCase, eval_case, judged_equivalence
from evaldata.platforms import duckdb_platform, resolve
from evaldata.scorers import QueryRunner, ScoreContext, Scorer
from evaldata.types import ScoreResult, SolverSuccess

_DB_PATH = Path(tempfile.mkdtemp(prefix="evaldata_ex05_")) / "shop.duckdb"
_PLATFORM = duckdb_platform(name="examples-llm-judge", path=str(_DB_PATH))
_MODEL = os.getenv("EVALDATA_HOSTED_MODEL", "openai/gpt-4o-mini")


@pytest.fixture(scope="module", autouse=True)
def _seed_db() -> Iterator[None]:
    con = duckdb.connect(str(_DB_PATH))
    con.execute("CREATE TABLE customers (id INTEGER, name VARCHAR, country VARCHAR)")
    con.execute("INSERT INTO customers VALUES (1, 'Ada', 'GB'), (2, 'Bo', 'US'), (3, 'Cy', 'US')")
    con.close()
    yield


def _score(case: EvalCase, model_sql: str, scorer: Scorer) -> ScoreResult:
    """Score `model_sql` against the case's gold query with `scorer`.

    Args:
        case: The eval case, carrying the gold query and platform.
        model_sql: The model's SQL to judge against the gold query.
        scorer: The scorer to run.

    Returns:
        The `ScoreResult`.
    """
    solver = CallableSolver(lambda _case: model_sql)
    output = solver.solve(case)
    assert isinstance(output, SolverSuccess)
    sql = output.output
    dialect = case.platform.dialect or case.platform.kind
    runner = QueryRunner(resolve(case.platform), sql, dialect, None)
    context = ScoreContext(queries=runner)
    return scorer.score(case, output, runner.run(sql), context=context)


def _trail(result: ScoreResult) -> list[tuple[str, bool]]:
    """The `(scorer, passed)` of each member that ran, in order.

    Args:
        result: A `judged_equivalence` score result.

    Returns:
        One `(scorer, passed)` pair per member that ran.
    """
    return [(entry["scorer"], entry["passed"]) for entry in result.metadata["first_decisive"]]


@eval_case(
    input="Name the US customers.",
    expected={"kind": "gold_query", "sql": "SELECT name FROM customers WHERE country = 'US'"},
    platform=_PLATFORM,
)
def test_judge_confirms_when_ast_is_inconclusive(case: EvalCase) -> None:
    """A CTE the syntax check can't match; the judge confirms it without running either query."""
    result = _score(
        case,
        "WITH us AS (SELECT * FROM customers WHERE country = 'US') SELECT name FROM us",
        judged_equivalence(_MODEL),
    )
    assert result.passed
    assert _trail(result) == [("semantic_equivalence", False), ("llm_judge", True)]


@eval_case(
    input="Name the US customers.",
    expected={"kind": "gold_query", "sql": "SELECT name FROM customers WHERE country = 'US'"},
    platform=_PLATFORM,
)
def test_judge_refutes_wrong_filter(case: EvalCase) -> None:
    """A wrong filter the syntax check can't match; the judge refutes it without running either query."""
    result = _score(
        case,
        "WITH gb AS (SELECT * FROM customers WHERE country = 'GB') SELECT name FROM gb",
        judged_equivalence(_MODEL),
    )
    assert not result.passed
    assert _trail(result) == [("semantic_equivalence", False), ("llm_judge", False)]

uv run pytest test_judged_equivalence.py -q

Remove conftest.py to grade with the model.

Run it from a clone

This is the bundled examples/05_llm_judge/ example. If you've cloned the repo, run it with uv run pytest examples/05_llm_judge. It includes fixed judge responses.

A ready-made judge: SQL equivalence

sql_equivalence_judge(model) is an LlmJudge pre-loaded with criteria and examples for one common question: do two SQL queries return the same rows? It grades actual_output against expected_output (the gold query), forgiving differences that never change the result and penalising those that do.

from evaldata import sql_equivalence_judge

judge = sql_equivalence_judge("openai/gpt-4o-mini")

Most often you don't reach for it directly. judged_equivalence(model) first tries to confirm equivalence by normalizing and comparing SQL ASTs, and only falls back to this judge when that check is inconclusive. Use it when you have no warehouse to run the queries against. See Check semantic equivalence for that cascade.

Next steps