Async OpenAI requests in Python: thousands of calls with retries and accounting

Async OpenAI requests in Python are what you need when thousands of texts have to go through a model: product descriptions, support tickets, database rows. A sequential loop is too slow, and "fire everything at once" runs into errors and untracked spend. Below: a ready script on asyncio and AsyncOpenAI with capped concurrency, retries on 429 and 503, token and cost accounting, and a worked batch cost.

Async OpenAI requests in Python: what the script is made of

Four parts:

1. AsyncOpenAI — the async client from the official openai package (pip install openai). It connects to Zukko like the regular one: base_url="https://api.zukko.pro/v1" and a fetch_... key.
2. asyncio.Semaphore — limits how many requests run at the same time. Without it, 10,000 tasks start at once.
3. Retries with exponential backoff — only for transient errors. Retrying key, balance or request-format errors is pointless.
4. usage accounting — count input, output and cached tokens and reconcile the cost.

The example model is deepseek-v4.1-flash, one of the cheapest in the catalog ($0.013 per 1M input, $0.038 per 1M output). Put any other model in MODEL. The current list and prices are at GET https://api.zukko.pro/v1/models or on the "Models" page in your dashboard.

Which errors to retry

Zukko errors and what to do with them in batch processing:

If the server sends a Retry-After header, wait that many seconds. Otherwise the pause grows exponentially: 1, 2, 4, 8… seconds plus random jitter so that retries from different tasks don't line up.

More on 429 in the article about error 429↗, and on failing over to a backup model on 503 in the article on error handling and fallback↗.

The full script

import asyncio
import random

import openai
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://api.zukko.pro/v1",
    api_key="fetch_...",
    max_retries=0,  # we retry ourselves to see the error codes
    timeout=120,
)

MODEL = "deepseek-v4.1-flash"

Key points explained

Why max_retries=0

By default the OpenAI SDK retries some errors itself (twice). That is handy for single calls but gets in the way in a batch: SDK retries don't see Zukko's codes and can't tell a used-up key limit from a transient 429. So built-in retries are off, and all the logic lives in call.

The semaphore

asyncio.Semaphore(CONCURRENCY) lets at most 16 requests reach the API at once; the other tasks wait. 16 is a starting value, not a provider recommendation. If you see almost no 429s, raise it; if you see many, lower it. The pause before a retry happens inside the semaphore: while a task waits it holds its slot, so a wave of errors slows the overall pace on its own.

Retries and Idempotency-Key

Each batch item gets its own Idempotency-Key header. If a request already completed and was charged but the response never reached you, a retry with the same key returns the stored response and doesn't charge again. The key lives for 24 hours and applies to successful responses. Send the retry after the first request has finished or failed, which is what the script does. The exception is a client-side timeout: if the server is still processing the first request, the retry may reach the model again. So set the timeout generously. Details are in the docs on limits and idempotency↗.

If you re-run a batch, use a different prefix in the key (a run number, say); otherwise a retry with the same key but different request text returns 422.

Usage and cost accounting

The script computes cost two ways:

• from the price list: fresh input × input price + cached × cache read price + output × output price. Fresh input is prompt_tokens minus prompt_tokens_details.cached_tokens;
• from the X-Zukko-Cost-Micros header: a regular (non-streaming) Zukko response carries the request cost in micro-dollars (1,000,000 = $1). To read headers the call goes through with_raw_response, and the response object comes from raw.parse().

The second number is more accurate: it reflects everything actually charged. Streaming responses don't have the header; check the spending history for those.

What a batch costs

The formula for a batch without cache:

Example: 10,000 requests, each with 600 input and 200 output tokens. 6M input and 2M output in total.

• deepseek-v4.1-flash: 6 × $0.013 + 2 × $0.038 = $0.078 + $0.076 = $0.154
• gpt-6.1-sol: 6 × $0.27 + 2 × $1.32 = $1.62 + $2.64 = $4.26

With cache. For GPT models the cached part of the input shows in cached_tokens. Suppose 400 of the 600 tokens are a shared instruction read from cache from the second request on. A cache write is not charged separately for GPT: the first request is billed in full at the input price. Then for gpt-6.1-sol (cache read $0.013 per 1M):

• first request: 600 × $0.27 / 1,000,000 = $0.000162;
• fresh input of the rest: 9,999 × 200 = 1,999,800 × $0.27 / 1,000,000 = $0.539946;
• cache read: 9,999 × 400 = 3,999,600 × $0.013 / 1,000,000 = $0.0519948;
• output: 2M × $1.32 = $2.64;
• total: $3.2321028, about $3.232.

That is an assumption: the provider decides how much input gets cached, and after a pause the cache can expire — then a request is billed in full at the input price again. The exact amount comes from X-Zukko-Cost-Micros and the spending history.

Before a big run, send 50–100 requests from real data and multiply their actual cost by the batch size. That beats guessing the average answer length.

When there are hundreds of thousands of requests

• Save results as you go. Write each answer to a file or database as soon as it arrives, and skip already processed item_ids on restart.
• Split into chunks. Process 5–10 thousand tasks per main call so you don't hold hundreds of thousands of objects in memory.
• A separate key with a limit. Create a pipeline key in the dashboard↗ with a daily dollar limit and an allowed-models list. A bug in the loop then can't eat the whole balance: the batch stops on 429 key_daily_limit_reached.
• Watch spend. The run's total is in the dashboard under "Usage" and in the spending history API.

Basic client examples are in the Python docs↗.

FAQ

How many requests should run in parallel?

Start with 10–20 and watch the share of 429s. There is no universal number: it depends on the model and the provider.

Should I retry 429?

Only if it isn't a key limit. With key_daily_limit_reached or key_monthly_limit_reached, retrying won't help until the limit resets (the daily one resets at 00:00 UTC).

Why not use the SDK's built-in retries?

They don't distinguish Zukko's codes: the SDK retries a key limit or a disabled model the same way as a transient failure. Your own logic lets you stop the batch right away.

How do I get the exact cost of a batch?

Sum X-Zukko-Cost-Micros across all responses, as the script does, or check the spending history in the dashboard.