How to test prompts: a simple eval harness in Python

A new prompt looks better until you run it on twenty examples and see where it fails. An eval harness is a small prompt-testing script: it runs a set of examples through a prompt and a model, scores the answers and totals the results. Below is a working Python version you can run on any Zukko model with a single key.

Step 1. The test set

Create test_cases.jsonl with one JSON object per line:

{"input": "Love this product, recommend it to everyone", "expected": "positive"}
{"input": "Terrible quality, returned it the next day", "expected": "negative"}
{"input": "It's fine, nothing special", "expected": "neutral"}
{"input": "Best purchase of the year!", "expected": "positive"}
{"input": "Waste of money", "expected": "negative"}
{"input": "Arrived on time, box intact", "expected": "neutral"}

How to build a good set:

• use real inputs, not invented ones; invented examples tend to be too easy;
• add hard and edge cases: sarcasm, mixed opinions, typos;
• start with 20–50 examples and add one every time you find an error in production.

Step 2. The harness

We use the OpenAI SDK: through https://api.zukko.pro/v1 it can call any model in the catalog, so the same code works for Claude, GPT and GLM.

import json
import time
from dataclasses import dataclass, asdict

from openai import OpenAI

client = OpenAI(base_url="https://api.zukko.pro/v1", api_key="fetch_your_key")

PRICES = {  # $ per 1M tokens: (input, output)
    "claude-sonnet-5": (0.30, 0.30),
    "claude-opus-5-5": (0.30, 0.30),
    "gpt-5.6-sol": (0.30, 0.30),
    "gpt-5.6-luna": (0.10, 0.10),
    "glm-5.3-flash": (0.04, 0.04),

Cost uses the formula (input × input_price + output × output_price) / 1,000,000 with prices from the dict. The model set changes: check the dict against GET https://api.zukko.pro/v1/models or the "Models" page in the dashboard.

max_tokens has headroom here: for some models the limit may also cover reasoning tokens, and too small a value will cut the answer off.

Step 3. Two prompt versions

PROMPT_V1 = """Classify the sentiment of the review. Answer with one word: positive, negative or neutral.

Review: {input}"""

PROMPT_V2 = """You classify customer reviews.
- positive: the customer is satisfied or recommends the product;
- negative: the customer is unhappy, complains or returned the product;
- neutral: facts without a clear opinion.
Answer with exactly one word from the list, no period, no explanation.

Review: {input}"""

Step 4. Comparison

def summary(results: list[EvalResult]) -> str:
    passed = sum(r.passed for r in results)
    cost = sum(r.cost_usd for r in results)
    return f"{passed}/{len(results)} ({passed / len(results):.0%}), ${cost:.4f}"

cases = load_cases("test_cases.jsonl")
all_results = []
for model in ["claude-sonnet-5", "gpt-5.6-luna"]:
    for version, prompt in [("v1", PROMPT_V1), ("v2", PROMPT_V2)]:
        res = run_eval(prompt, version, cases, model)
        all_results += res
        print(f"== {model} {version}: {summary(res)}")

The script prints a line per example and a summary for each "model + prompt version" pair: pass rate and run cost. Always look at the failures one by one too: often it is not the prompt failing but an inaccurate label in the set.

How to read the results

• Mind the set size. With 20 examples, one error is 5 percentage points. A one- or two-example gap between versions may be noise. The bigger the set, the more confident the conclusion.
• Run twice. Model output is not fully deterministic, and newer Claude models do not let you tune temperature. Run the set two or three times: if the result jumps around, grow the set or tighten the answer format.
• Compare cost. A longer prompt costs more on every call. If v2 scores the same as v1 but is longer, keep v1.

Scoring without exact match: an LLM judge

Generated text never matches exactly. In that case another model scores the answer against your criteria:

JUDGE_PROMPT = """Score the answer against these criteria:
1) it answers the question; 2) no invented facts; 3) no longer than 3 sentences.
Return JSON: {{"score": a number from 0 to 3, "reason": "brief"}}

Question: {question}
Reference: {expected}
Answer: {answer}"""

def judge(question: str, expected: str, answer: str, model: str = "claude-sonnet-5") -> dict:
    r = client.chat.completions.create(
        model=model,
        max_tokens=300,
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(
            question=question, expected=expected, answer=answer)}],
    )
    text = r.choices[0].message.content or ""
    try:
        start, end = text.index("{"), text.rindex("}") + 1
        return json.loads(text[start:end])
    except ValueError:
        return {"score": None, "reason": "could not parse the judge's answer"}

json.JSONDecodeError is a subclass of ValueError, so except ValueError catches both missing braces and invalid JSON.

Before trusting the judge, check it: grade 20 answers by hand and compare with its scores. If it often disagrees with you, refine the criteria. Make criteria checkable ("no longer than 3 sentences"), not vague ("a good answer").

Saving results

import csv
from datetime import date

def save_csv(results: list[EvalResult], path: str):
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(asdict(results[0]).keys()))
        writer.writeheader()
        for r in results:
            writer.writerow(asdict(r))

save_csv(all_results, f"eval_{date.today()}.csv")

A date in the file name lets you compare runs over time: open two CSVs and you can see which examples started passing and which broke.

Team workflow

1. Prompts live in the repository next to the code, each with a version.
2. Every prompt change comes with an eval run.
3. A new version ships only if it is not worse than the current one on the whole set.
4. Errors from production are added to the set as new examples.

It is handy to have a separate key with a daily limit for eval runs; create one under API keys↗. More on calls in the Chat Completions reference↗.

FAQ

How many examples do I need?

For a first check, 20–50. For a release decision on an important prompt, 100 or more: on a small set random variation is comparable to the effect.

Why do answers to the same example differ between runs?

Generation is not fully deterministic. Tighten the answer format (one word, JSON with a schema) and look at the total across several runs rather than a single answer.

Should I test on several models?

If you are choosing a model or might switch, yes. The same prompt can behave differently on different models; the harness above checks that with one line in the model list.

What does a run cost?

Use the formula. For example, 50 examples × 2 prompt versions on claude-sonnet-5, averaging 400 input and 20 output tokens per request: 100 requests × 420 tokens = 42,000 tokens; 42,000 × $0.30 / 1,000,000 = $0.0126. The same run on gpt-5.6-luna: 42,000 × $0.10 / 1,000,000 = $0.0042.