Bulk request processing without hitting the rate limit

A batch job running thousands of requests behaves differently from an interactive chat: errors accumulate, retries multiply, and a spending limit on a key that was invisible during development turns into a wall an hour into the run. Here are concrete techniques that remove that wall — not by disabling limits altogether, which isn't always possible or wise, but by working with them.

Know which 429 you actually got

The gateway has no rate limit of its own. A 429 from Zukko itself means a specific key's spending limit ran out, and the response code says so directly — key_daily_limit_reached or key_monthly_limit_reached. Any other 429 came from the model provider. These are fundamentally different situations. A provider 429 is a signal to slow down and retry with exponential backoff; it clears on its own after a while. A key limit only clears when a new day (reset at 00:00 UTC) or month starts, or after manually raising the limit in the dashboard — hammering it with retries accomplishes nothing; the worker should switch to another key or stop.

if resp.status_code == 429:
    code = resp.json().get("error", {}).get("code")
    if code in ("key_daily_limit_reached", "key_monthly_limit_reached"):
        switch_to_next_key()
    else:
        time.sleep(backoff_seconds)

Split the load across several keys

Each key's page in the dashboard sets its own daily/monthly limit and list of allowed models. For batch processing it makes sense to keep a pool of several keys and distribute requests across them — by hashing the job id or by sub-task, for instance. This doesn't add throughput (the gateway has no rate limit, and all keys draw on the same account balance), but each part of the run gets its own budget: a sub-task stuck in a loop exhausts its own key's limit, not the whole batch's budget.

Idempotent retries instead of bare ones

A response cutting off mid-batch is almost always a client or network timeout, not a token limit — worth double-checking max_tokens and the HTTP client's timeouts before touching retry logic at all. The retry itself should carry an Idempotency-Key header, keeping the same value for the same logical request:

import uuid

idempotency_key = str(uuid.uuid4())
for attempt in range(3):
    resp = requests.post(
        "https://api.zukko.pro/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {api_key}",
            "Idempotency-Key": idempotency_key,
        },
        json=payload,
        timeout=30,
    )
    if resp.status_code < 500 and resp.status_code != 429:
        break
    time.sleep(2 ** attempt)

If the original request actually reached the model and completed, retrying with the same Idempotency-Key within 24 hours returns the same response without charging again — critical specifically in batch work, where the sum of accidental double charges across thousands of requests shows up on the bill.

Cache repeated inputs

If a batch contains a lot of fully identical requests — classifying repeated values, for instance — turn on the cache with the X-Zukko-Cache: true header: that's the primary way, and it works for any model. temperature: 0 also enables the cache, but only for models that accept that value; newer Claude models (claude-sonnet-5, claude-opus-5-5) may reject a temperature other than 1 with a 400, so don't pass temperature to them. Identical requests are cached for up to 7 days (for chat), a repeat comes back instantly and free, and the response carries X-Zukko-Cache: hit. Note that the cache only hits on an exactly identical request body — "near-identical" requests and requests sharing a fixed prompt prefix (template + different data) are not served from cache. Cache hits are not written to /v1/usage. For a batch job this isn't optimization for its own sake: if 30% of the input is exact duplicates, caching genuinely removes close to a third of the run's cost.

resp = requests.post(url, headers={**headers, "X-Zukko-Cache": "true"}, json=payload)
if resp.headers.get("X-Zukko-Cache") == "hit":
    cache_hits += 1

Watch the balance during the run, not after

Every regular (non-streaming) response carries X-Zukko-Cost-Micros (that request's cost in micro-dollars, 1,000,000 = $1) and X-Fetch-Balance-Cents (the balance left after the charge — despite the name, also in micro-dollars; the exact balance in cents is at GET /v1/balance). Streamed responses (stream: true) don't carry these headers, so batch jobs are best run without streaming. On a batch of thousands of requests, that's enough to compute an average cost from the first hundred and extrapolate to the full volume before the run is even halfway done — and stop early if the balance clearly won't cover it, instead of hitting a 402 on request 6,000 of 10,000 and untangling which part of the data got processed.

cost_micros = int(resp.headers.get("X-Zukko-Cost-Micros", 0))
running_total += cost_micros

Check the catalogue before a long run instead of hardcoding a model

GET /v1/models returns the current model list (only id, object, created, owned_by), while GET /v1/models/info gives their prices and context limits. Before kicking off a batch that will run for hours, it's worth hitting these endpoints and comparing them against what's hardcoded in the config — if a model has since been switched off, better to find out ahead of time than hit 503 model_disabled partway through and lose a partially processed batch.

curl https://api.zukko.pro/v1/models/info

503 and model_disabled aren't a reason to fail the whole job

503 provider_unavailable or model_disabled means a model is temporarily off in the catalogue, not a problem with the request. In a batch it makes sense to catch these codes separately from other 5xx errors and set the corresponding queue items aside for later retry, rather than failing the whole job or spending the same retry budget as on a plain network hiccup.

Putting it together

A working batch runner ends up looking like this: a pool of keys with limits configured, an Idempotency-Key per logical request, the X-Zukko-Cache: true header where the input predictably repeats, X-Zukko-Cost-Micros read for budget forecasting, and separate handling for a keyed 429, 503/model_disabled, and plain network timeouts. None of this needs anything beyond standard HTTP headers and response codes — the whole thing is built on top of ordinary requests or httpx, no special SDK required.