Generating test data with an AI model: schema, validation and fixtures

Generating test data with an AI model pays off where random strings aren't enough: support requests, reviews, product descriptions, profiles with consistent fields. Below is a Python script that gets such records through the API, validates each one with pydantic, removes duplicates and saves the result as a test fixture. Separately: how to make the data reproducible and what it costs.

AI model or Faker

Faker and similar libraries are fast, free and deterministic with a fixed seed. But they produce values independently of each other: the city has nothing to do with the name, and a support request is a bag of random words.

A practical option is a hybrid: your code generates technical fields (id, dates, amounts), and the model is responsible only for the meaningful ones. That way IDs are guaranteed unique and texts look natural.

Schema, request, validation, fixture

1. Describe the record with a pydantic model: types, ranges, allowed values.
2. Ask the model to return a JSON array with these fields.
3. Extract the array from the answer and validate every record; drop invalid ones.
4. Remove duplicates, assign IDs in your code, save to a file.

Create a fetch_... key in the Zukko dashboard↗ and put it in the ZUKKO_API_KEY variable. Dependencies: pip install openai pydantic.

import json
import os
from typing import Literal

from openai import OpenAI
from pydantic import BaseModel, Field, ValidationError

client = OpenAI(base_url="https://api.zukko.pro/v1", api_key=os.environ["ZUKKO_API_KEY"])
MODEL = os.getenv("ZUKKO_MODEL", "glm-5.3-flash")
SEGMENTS = ["students", "retirees", "small business owners", "IT specialists", "parents with kids"]


class Customer(BaseModel):
    first_name: str = Field(min_length=2, max_length=50)

What matters in this code:

• Reserved domains only. Addresses on example.com, example.org and example.net are reserved for documentation and examples, so a test email can't accidentally reach a real person. The regular expression in the model rejects everything else.
• Robust parsing. Models sometimes wrap JSON in a markdown block or add a sentence before it. extract_json_array takes the text from the first [ to the last ]; if there is no array, the batch is skipped.
• Validation of every record. Pydantic checks types, lengths, the age range and the Literal for the plan. Invalid records are dropped, the rest of the batch is kept.
• IDs and uniqueness live in code. The model doesn't know what it generated in the previous batch, so duplicate emails are removed via the seen set, and id is assigned sequentially.
• Segments. Different segments in consecutive requests make the sample more varied than ten identical requests in a row.
• A cap on attempts. The loop won't run forever if the model keeps returning garbage.

Reproducible data

The model's answer changes from run to run, and the word "seed" in the prompt won't fix that. Don't rely on sampling parameters like temperature either: they don't make the answer deterministic, and newer Claude models may reject a non-default temperature with a 400 error, so it's better not to pass it at all in examples with claude-* models.

There is only one reliable way: generate once and save. The customers.json file goes into the repository next to the tests, and the tests read it as a fixture:

import json
from pathlib import Path

import pytest

FIXTURE = Path(__file__).parent / "data" / "customers.json"


@pytest.fixture(scope="session")
def customers():
    return json.loads(FIXTURE.read_text(encoding="utf-8"))


def test_every_customer_has_ticket(customers):
    assert all(len(c["support_ticket"]) >= 20 for c in customers)

This way tests don't depend on the network or the balance, run in CI without a key, and regenerating the data becomes a deliberate action with a clear diff in review.

If you need to repeat the same request while debugging, Zukko has a response cache triggered by an explicit header: a request with X-Zukko-Cache: true and exactly the same body returns the stored answer for up to 7 days, and the response carries the X-Zukko-Cache: hit header. In the OpenAI SDK the header is passed via extra_headers, and response headers are available through with_raw_response:

raw = client.chat.completions.with_raw_response.create(
    model=MODEL,
    max_tokens=4000,
    messages=[{"role": "user", "content": PROMPT.format(count=10, segment="students")}],
    extra_headers={"X-Zukko-Cache": "true"},
)
print(raw.headers.get("X-Zukko-Cache"))
response = raw.parse()

More on the cache and key limits in the documentation↗.

Validation with jsonschema

If the schema is already described in JSON Schema (for example, it's an API contract), you can validate records with the jsonschema library instead of pydantic:

import jsonschema

for record in records:
    try:
        jsonschema.validate(instance=record, schema=schema)
    except jsonschema.ValidationError as e:
        print("Record dropped:", e.message)

It also helps to put the same schema into the prompt: an explicit field description is easier for the model to follow than a prose summary.

Cost and model choice

Zukko prices per 1M tokens (input/output): glm-5.3-flash $0.04 / $0.04, gpt-5.6-luna $0.10 / $0.10, claude-sonnet-5 $0.30 / $0.30. The current list is at GET https://api.zukko.pro/v1/models or on the "Models" page in the dashboard.

Example: 1,000 records in batches of 10 means 100 requests. Say the prompt takes 400 input tokens and an answer with ten records 1,500 output tokens.

• glm-5.3-flash: 100 × (400 + 1,500) × $0.04 / 1,000,000 = $0.0076.
• claude-sonnet-5: 100 × (400 + 1,500) × $0.30 / 1,000,000 = $0.057.

Dropped records and repeated batches add a few percent to these amounts. Test which model gives varied and correct enough data for your tests on your own tasks: generate 50 records with each of two models and see how many are rejected by validation and how similar the texts are to each other.

FAQ

How do I avoid getting real personal data?

Say "fictional" in the prompt, forbid celebrities and use only the reserved example.* domains, as in the example. For phone numbers and document numbers, generate values in your own code within ranges known to be test-only instead of asking the model.

The model returns fewer records than requested

That's normal: validation drops some records, and sometimes the model cuts the answer off at max_tokens. The loop in generate() fills the gap with further batches. If answers are regularly cut off, lower batch_size or raise max_tokens.

Can I generate images or PDFs?

There are no image generation models in the Zukko catalog. For PDFs and documents use libraries such as ReportLab, and use the model only for the text that goes into them.

How do I generate data for several related tables?

First generate parent records (customers) and assign them IDs, then pass a short list of parent IDs and key fields into the prompt for the child records (orders, tickets). Check the relationships in code after generation. The request format is described in the chat completions reference↗.