Skip to content

Score with an LLM judge

Not every answer can be checked by comparing rows or matching syntax trees. Does the SQL read clearly? Does it follow your team's style? Do two queries return the same thing when you have no warehouse to run them on? LlmJudge answers questions like these: it asks a grader model to score the case against criteria you write, and turns that score into a pass/fail verdict.

What the judge does

You give the judge a standard to grade against, written as plain text in criteria. It builds a prompt from that standard and the case, asks the grader model to reason about how well the case meets it and give a score from 0.0 to 1.0, then compares that score to a threshold to decide the verdict:

from evaldata import LlmJudge

judge = LlmJudge(
    model="openai/gpt-4o-mini",
    criteria="The SQL answers the question, reads clearly, and uses no SELECT *.",
    threshold=0.7,
)

The grader's score lands in result.score and its rationale in result.explanation; the result is stamped basis="judged" to mark it a probabilistic judgment, not a proof.

Pass it to assert_eval like any other scorer:

assert_eval(case, solver, scorers=[judge])

Inconclusive, not a guess

A judge can fail to reach a verdict — the provider call errors, or the reply doesn't parse. When that happens the result is inconclusive (verdict="inconclusive"), carrying the failure in its explanation, rather than a fail. Inconclusive means the judge couldn't decide; it doesn't mean the answer is wrong.

Shape the grader's judgment

criteria alone is often enough. Four optional arguments make the grade more consistent when it isn't:

  • steps — an ordered checklist the grader works through before scoring, rendered as a numbered block.
  • rubric — score bands that say what each range means, so the number is anchored.
  • examples — few-shot anchors, each a graded output with its score and the reason for it.
  • threshold — the minimum score (inclusive) for a pass. Defaults to 0.5.
from evaldata import JudgeExample, LlmJudge, RubricBand

judge = LlmJudge(
    model="openai/gpt-4o-mini",
    criteria="The SQL answers the question and is idiomatic for an analytics team.",
    steps=[
        "Check the query returns the columns the question asks for.",
        "Check it filters and aggregates correctly.",
        "Judge readability: aliases, no SELECT *, sensible formatting.",
    ],
    rubric=[
        RubricBand(min_score=0.0, max_score=0.4, description="Wrong result or unreadable."),
        RubricBand(min_score=0.4, max_score=0.8, description="Correct result, rough style."),
        RubricBand(min_score=0.8, max_score=1.0, description="Correct and idiomatic."),
    ],
    examples=[
        JudgeExample(
            actual_output="SELECT name FROM customers WHERE country = 'US'",
            score=1.0,
            reason="Correct filter, named column, clear.",
        ),
    ],
    threshold=0.8,
)

Only the arguments you set appear in the prompt — an unset rubric or examples is simply omitted.

Choose what the grader sees

By default the judge shows the grader three case fields, each only when it is available:

  • input — the case's question.
  • actual_output — the SQL the solver produced.
  • expected_output — the gold query's SQL, when the case has a GoldQuery expected.

Restrict this with show when a field would bias or distract the grade — for example, grade the output on its own merits without revealing the reference answer:

judge = LlmJudge(
    model="openai/gpt-4o-mini",
    criteria="The SQL answers the question.",
    show=["input", "actual_output"],   # withhold the expected query
)

Choosing a grader model

The grader is separate from the solver — the model under test and the model grading it are usually different. model takes any litellm identifier (or an Llm to use directly), so the grader can run on a different provider from the solver.

The grader must support structured output: the judge requests a JSON {score, reason} reply and treats a malformed one as inconclusive. Grading defaults to temperature=0.0 for repeatable scores; pass temperature=None to leave the provider default, or a timeout to bound the call.

Run it deterministically in CI

A judged eval calls a live model, which costs money and varies run to run. To run the eval as a plumbing check in CI — exercising the full path except the network — mock the grader reply. Add a conftest.py next to your test that returns the structured {score, reason} the judge expects:

"""Mocked grader replies for the LLM-judge example, so it runs without a key or network."""

import json
from typing import Any

import litellm
import pytest

_EQUIVALENT = json.dumps({"reason": "the CTE wraps the same filter", "score": 1.0})
_NOT_EQUIVALENT = json.dumps({"reason": "country = 'GB' selects different customers", "score": 0.0})


def _mock_reply(messages: list[dict[str, Any]]) -> str:
    """Pick the grader reply for a request by matching the actual SQL under judgement.

    Args:
        messages: The chat messages of the request, whose prompt carries the actual SQL.

    Returns:
        A `JudgeReply`-shaped `{"reason", "score"}` JSON string.
    """
    prompt = " ".join(m.get("content", "") for m in messages)
    return _NOT_EQUIVALENT if "country = 'GB'" in prompt else _EQUIVALENT


@pytest.fixture(autouse=True)
def _mock_completion(monkeypatch: pytest.MonkeyPatch) -> None:
    """Patch grader calls to return a deterministic structured reply per case, with no network."""
    real_completion = litellm.completion

    def fake(**kwargs: Any) -> Any:
        return real_completion(**kwargs, mock_response=_mock_reply(kwargs["messages"]))

    monkeypatch.setattr("litellm.completion", fake)
    monkeypatch.setattr("litellm.supports_response_schema", lambda **_: True)

With the mock in place the eval runs offline, with no key:

"""LLM-judge example evals: `judged_equivalence` has an LLM grade the SQL the syntax check can't confirm."""

import os
import tempfile
from collections.abc import Iterator
from pathlib import Path

import duckdb
import pytest

from evaldata import CallableSolver, EvalCase, eval_case, judged_equivalence
from evaldata.platforms import duckdb_platform, resolve
from evaldata.scorers import QueryRunner, ScoreContext, Scorer
from evaldata.types import ScoreResult

_DB_PATH = Path(tempfile.mkdtemp(prefix="evaldata_ex05_")) / "shop.duckdb"
_PLATFORM = duckdb_platform(name="examples-llm-judge", path=str(_DB_PATH))
_MODEL = os.getenv("EVALDATA_HOSTED_MODEL", "openai/gpt-4o-mini")


@pytest.fixture(scope="module", autouse=True)
def _seed_db() -> Iterator[None]:
    con = duckdb.connect(str(_DB_PATH))
    con.execute("CREATE TABLE customers (id INTEGER, name VARCHAR, country VARCHAR)")
    con.execute("INSERT INTO customers VALUES (1, 'Ada', 'GB'), (2, 'Bo', 'US'), (3, 'Cy', 'US')")
    con.close()
    yield


def _score(case: EvalCase, model_sql: str, scorer: Scorer) -> ScoreResult:
    """Score `model_sql` against the case's gold query with `scorer`.

    Args:
        case: The eval case, carrying the gold query and platform.
        model_sql: The model's SQL to judge against the gold query.
        scorer: The scorer to run.

    Returns:
        The `ScoreResult`.
    """
    solver = CallableSolver(lambda _case: model_sql)
    output = solver.solve(case)
    sql = output.output
    assert sql is not None  # CallableSolver always produces SQL here
    dialect = case.platform.dialect or case.platform.kind
    runner = QueryRunner(resolve(case.platform), sql, dialect, None)
    context = ScoreContext(queries=runner)
    return scorer.score(case, output, runner.run(sql), context=context)


def _trail(result: ScoreResult) -> list[tuple[str, bool]]:
    """The `(scorer, passed)` of each member that ran, in order.

    Args:
        result: A `judged_equivalence` score result.

    Returns:
        One `(scorer, passed)` pair per member that ran.
    """
    return [(entry["scorer"], entry["passed"]) for entry in result.metadata["first_decisive"]]


@eval_case(
    input="Name the US customers.",
    expected={"kind": "gold_query", "sql": "SELECT name FROM customers WHERE country = 'US'"},
    platform=_PLATFORM,
)
def test_judge_confirms_when_ast_is_inconclusive(case: EvalCase) -> None:
    """A CTE the syntax check can't match; the judge confirms it without running either query."""
    result = _score(
        case,
        "WITH us AS (SELECT * FROM customers WHERE country = 'US') SELECT name FROM us",
        judged_equivalence(_MODEL),
    )
    assert result.passed
    assert _trail(result) == [("semantic_equivalence", False), ("llm_judge", True)]


@eval_case(
    input="Name the US customers.",
    expected={"kind": "gold_query", "sql": "SELECT name FROM customers WHERE country = 'US'"},
    platform=_PLATFORM,
)
def test_judge_refutes_wrong_filter(case: EvalCase) -> None:
    """A wrong filter the syntax check can't match; the judge refutes it without running either query."""
    result = _score(
        case,
        "WITH gb AS (SELECT * FROM customers WHERE country = 'GB') SELECT name FROM gb",
        judged_equivalence(_MODEL),
    )
    assert not result.passed
    assert _trail(result) == [("semantic_equivalence", False), ("llm_judge", False)]

uv run pytest test_judged_equivalence.py -q

Remove the conftest.py (and set the provider key, e.g. OPENAI_API_KEY) to grade against the live model instead.

Run it from a clone

This is the bundled examples/05_llm_judge/ example. If you've cloned the repo, run it directly with uv run pytest examples/05_llm_judge — it runs mocked, with no key needed.

A ready-made judge: SQL equivalence

sql_equivalence_judge(model) is an LlmJudge pre-loaded with criteria and examples for one common question: do two SQL queries return the same rows? It grades actual_output against expected_output (the gold query), forgiving differences that never change the result and penalising those that do.

from evaldata import sql_equivalence_judge

judge = sql_equivalence_judge("openai/gpt-4o-mini")

Most often you don't reach for it directly. judged_equivalence(model) first tries to confirm equivalence by comparing the queries' structure, and only falls back to this judge when the structure is inconclusive — handy when you have no warehouse to run the queries against. See Check semantic equivalence for that cascade.

Next steps