Data labeling with an LLM: a 50,000-row dataset on a cheap model

Labeling tens of thousands of rows by hand takes weeks or a contract with annotators. Data labeling with an LLM comes down to a script: a cheap model reads each row according to your instruction and returns a label as JSON, and a human checks a random sample. Below: the prompt, Python code, a way to check quality, and the cost of 50,000 rows on three models.

When a model can do the labeling

Tasks with a fixed set of answers work well:

• classification against a label list: ticket topic, review sentiment, product category;
• yes/no flags: does the text contain contact details, is it spam, does it mention a competitor;
• extracting short fields: city, order number, amount.

Tasks where people themselves disagree on the right answer work poorly, as do tasks where the label needs knowledge that isn't in the row text.

A simple rule: if you can write an instruction that makes two different people label the same way, you can give that instruction to a model. If you can't, the model will be guessing too.

The instruction: a prompt that gives stable answers

One instruction covers the whole dataset and goes into the system message. It should contain:

1. The label list with a one- or two-sentence definition of each.
2. Rules for edge cases: what to do when a ticket has two topics.
3. Three to five "text → label" examples.
4. A separate label for unclear cases (unclear) so the model doesn't guess.
5. A strict output format.

You are labeling customer tickets for an online store.
Pick exactly one label:
- billing — payment, refunds, receipts, charges;
- delivery — timing, courier, pickup point, tracking number;
- quality — defects, wrong item, missing parts;
- other — anything else that is clear from the text;
- unclear — the text is empty, meaningless, or the topic can't be determined.
If there are several topics, pick the one the customer starts with.

Examples:
"I was charged twice" -> billing
"The courier never came, tracking isn't updating" -> delivery

Reply with JSON only, no explanations: {"label": "<label>"}

Why JSON rather than a single word: the answer is parsed with one line of code, any extra text immediately shows up as an error, and you can easily add a second field later, such as an extracted order number.

The labeling script in Python

The script reads data.csv with id and text columns, writes results line by line to labels.jsonl, and skips rows that are already labeled when restarted. It needs Python 3.9+ and the openai package.

import csv
import json
import os

from openai import OpenAI

client = OpenAI(base_url="https://api.zukko.pro/v1", api_key=os.environ["ZUKKO_API_KEY"])
MODEL = "deepseek-v4.1-flash"
LABELS = {"billing", "delivery", "quality", "other", "unclear"}
with open("instruction.txt", encoding="utf-8") as f:
    INSTRUCTION = f.read()


def label_row(text: str) -> str:

What matters here:

• temperature=0 reduces variation between repeated runs.
• The X-Zukko-Cache: true header turns on Zukko's response cache: an identical request (same model, same body) is served free the second time. Duplicate rows in the dataset and a rerun after a crash cost nothing. Details are in the key limits and cache docs↗.
• The __parse_error__ and __bad_label__ labels aren't lost: those rows go to manual review or to a rerun with a fixed instruction.

A single thread over 50,000 rows takes a long time, so in practice requests are parallelized. How to do that without hitting provider limits is covered in the article on bulk request processing↗.

Human review of a sample

Without a check you don't know how many errors the labels contain. The check has two steps.

A pilot before the main run

Label 200–300 rows by hand — that's your gold set. Run two or three models over it with the same instruction and compute each model's agreement with the gold set. Look at which labels the models get wrong and fix the instruction: errors usually cluster around one or two edge cases.

The pilot costs pennies even on a pricier model. For 300 rows at 550 input and 40 output tokens on claude-sonnet-5 ($0.2 / $1.013 per 1M): 300 × 550 = 165,000 input tokens × 0.2 / 1,000,000 = $0.033, plus 300 × 40 = 12,000 output tokens × 1.013 / 1,000,000 ≈ $0.012. About $0.045 in total.

A sample after the run

Take random rows from the finished labels and check them by hand. The required size comes from the standard formula for a proportion: n = z² × p × (1 − p) / e². For 95% confidence (z = 1.96), the worst case p = 0.5, and a margin of e = 5 percentage points: 1.96² × 0.25 / 0.05² ≈ 384. About 400 random rows give you the error rate within ±5 points. Check every row labeled unclear and every parse error separately; they are not part of the random sample.

What labeling 50,000 rows costs

Assumptions: the instruction is 400 tokens, an average row is 150 tokens, and the answer {"label": "delivery"} is up to 40 tokens including overhead.

• Input: 50,000 × (400 + 150) = 27,500,000 tokens = 27.5M.
• Output: 50,000 × 40 = 2,000,000 tokens = 2M.
• Formula: cost = 27.5 × input price + 2 × output price (prices in $ per 1M).

For comparison, the same job on claude-sonnet-5: 27.5 × 0.2 + 2 × 1.013 = 5.5 + 2.026 ≈ $7.53.

Two caveats. First, how many tokens your text takes depends on the model's tokenizer. Run 100 rows, look at resp.usage.prompt_tokens and resp.usage.completion_tokens, and plug your own averages into the formula; you can also count tokens before a run with count_tokens↗. Second, prompt caching: for DeepSeek and GLM a cache read costs the same as input, so the repeated instruction doesn't get cheaper. For Gemini 3.8 Flash a cache read is cheaper than input (0.013 vs 0.025), but whether the instruction actually lands in the cache and whether the provider reports the read in its response depends on the provider — the real split shows up in your usage history. These models have no separate cache-write charge.

The intelligence index in the table is Artificial Analysis's aggregate score on general benchmarks, not on your labeling task. Don't pick the cheapest model; pick the cheapest one that passed your pilot with the agreement rate you need. If no cheap model is good enough, hard rows can go to a stronger one — the pattern is covered in the article on model cascades↗. Current models and prices: GET https://api.zukko.pro/v1/models or the Models page in the dashboard.

For labeling it's convenient to create a separate key with a daily limit: if the script loops, it stops at the cap, not at a zero balance. Keys are created under API keys↗ in the dashboard.

FAQ

Can I ask the model to report its confidence?

You can add a field, but the model's self-assessment hasn't been validated on your task. Rely on agreement in the pilot and in the random sample, not on a number the model wrote about itself.

What if the model doesn't return JSON?

The script above strips a triple-backtick wrapper and marks everything else as __parse_error__. If more than a couple of percent of rows end up there, tighten the format wording in the instruction and add an example answer.

How many rows should I check by hand?

About 400 random rows estimate the error rate within ±5 percentage points at 95% confidence. Plus every row labeled unclear and every parse error.

How do I avoid paying twice on a restart?

The script skips ids that are already written, and the X-Zukko-Cache: true header serves identical requests from cache for free. The cache keeps successful responses for up to 7 days.