LLM as a judge: evaluating answers

You need: a fetch_... key (create one), a balance (50 questions ≈ 10 cents), Python 3.10+, 20–100 real customer questions with references, short correct answers.

1. 1
   Collect the questions

   Take real questions from your support chat or inbox, easy and tricky ones. For each, write a reference: what the answer must contain.

2. 2
   Pick the judge

   The judge is stronger than the candidates and not one of them: claude-sonnet-5-5 or gpt-6.1-sol.

3. 3
   Check the rubric

   It lives in GRADE_SYSTEM: correctness, completeness, clarity on a 1–5 scale. If "never promise what the rules don't say" matters to your business, add it as a separate criterion.

4. 4
   Run it

   Copy the code into judge.py and replace RULES and DATASET with your own. Run pip install openai pydantic, then python judge.py.

5. 5
   Check the judge against yourself

   Open judge_results.jsonl and grade 10–20 answers yourself. If you often disagree with the judge, refine the rubric or the references instead of trusting the table.

1Code

The example compares deepseek-v4.1-flash and claude-haiku-5-5 as a store support agent. The judge is claude-sonnet-5-5.

"""
LLM as a judge: which of two models answers store customers better.
pip install openai pydantic
python judge.py
"""
import asyncio
import json
import os
import statistics
from typing import Literal

from openai import AsyncOpenAI
from pydantic import BaseModel, Field, ValidationError

You'll get a table like this (the numbers depend on the models and questions):

Model                  Score   Corr  Compl  Clar    Wins     Losses   Ties
deepseek-v4.1-flash     4.20   4.40   4.00  4.60       1          2      2
claude-haiku-5-5        4.40   4.60   4.20  4.60       2          1      2

The judge changed its mind when the answers were swapped: 1 of 5

2Reading the table

• The judge compares each pair twice, with the answers swapped. A win counts only if it picked the same candidate both times.
• If the judge changed its mind on more than a third of the questions, the candidates are too close or the rubric is vague.
• On 5 questions a 0.2-point gap is noise. For a decision use 30–50 questions and look at pairwise wins.

3Keep in mind

• X-Zukko-Cache: true: identical requests are served from the cache for free for 7 days. Change the rubric and you pay only for the new judge requests. Need fresh candidate answers? Set CACHE = {}.
• Don't set temperature: newer Claude models may reject non-default values.

4If it doesn't work