Rate limits in an API: the kinds of limits and a client built for them

A rate limit for an API is a cap on how often and how much you can send per unit of time. Exceed it and the server returns 429; what happens next depends on your client. It either waits politely and retries, or floods the server with retries and makes things worse. Below: what kinds of limits LLM APIs have, how they look through a gateway, and how to write a Python client that lives with them.

The kinds of limits

LLM APIs usually have several independent restrictions, and a 429 can come from any of them:

The first three are set by the model provider and differ by model and access tier. Providers change the exact numbers often, so this article does not quote them: check the provider's documentation.

Burst versus average rate

A limit of "120 requests per minute" can be read two ways. One: a one-minute window in which you may spend all 120 requests in the first five seconds. Two: a steady rate of 2 requests per second with no spikes. The difference is called burst.

Often this is a token bucket: the bucket holds, say, 10 requests (the burst) and refills at 2 requests per second. While the bucket is not empty you can fire a batch of 10, then you hit the average rate. If you do not know how a particular provider implements it, the safe approach is to hold to the average rate and treat burst as slack for irregularities.

Planning arithmetic: at 120 RPM with a burst of 10, a thousand requests take at least (1000 - 10) / 2 = 495 seconds, about 8 minutes. If each request weighs 1,500 tokens, 120 RPM means 120 x 1,500 = 180,000 tokens per minute, so TPM may trigger before RPM does.

How it works through the Zukko gateway

Zukko itself has no general rate limit. A 429 through the gateway comes in two cases:

1. The key's daily or monthly limit is used up (key_daily_limit_reached or key_monthly_limit_reached). You set it yourself in dollars per key; details in the key limits and cache documentation↗.
2. A restriction at the model provider: the provider answered 429 and the gateway passes it on.

Telling them apart matters because the right reaction is opposite. A provider limit clears in seconds or minutes, and a paused retry helps. A key limit will not clear until the next day or month, or until you raise it in the dashboard: retrying only burns time. You can tell them apart by the code in the response body.

If key limits are your safety fuse against a runaway script, set them up front and write retries only for the provider limit. More on the error itself: what to do about 429↗.

A client with a queue and exponential backoff

Below is an async Python client. It holds the average rate with a token bucket, caps concurrency with a semaphore, and on a provider 429 waits 1, 2, 4, 8 seconds and so on with a bit of random jitter so parallel tasks do not wake up together.

import asyncio, os, random, time
from openai import AsyncOpenAI, RateLimitError, APIStatusError, APIConnectionError

client = AsyncOpenAI(
    base_url="https://api.zukko.pro/v1",
    api_key=os.environ["ZUKKO_API_KEY"],
    max_retries=0,  # we do retries ourselves
)

class Limiter:
    """Token bucket: average rate of rpm per minute, burst up to `burst`."""
    def __init__(self, rpm: int, burst: int):
        self.rate = rpm / 60.0
        self.capacity = burst

What matters here. The pause happens outside the semaphore, otherwise sleeping tasks would hold slots. Client errors other than 429 are raised immediately: retrying a malformed request is pointless. The limiter sits before every attempt, retries included, because a retry also spends a request. If the response carries a Retry-After header, prefer it over the computed pause; check whether your model's responses include it.

Practical rules

• Start below the limit. If the provider allows 120 RPM, set 80-100: headroom for clock drift and other traffic on the same key.
• Count tokens, not just requests. Heavy requests hit TPM; estimate size beforehand with count_tokens↗ or by text length.
• Cap concurrency. Often 4-8 simultaneous requests are enough to reach the speed you need.
• Log the reason for a 429. Keep the response body: it shows whether the key or the provider is the cause.
• Never retry forever. A cap on attempts and an overall task timeout are mandatory.

For big batches see the walkthrough on bulk processing without hitting the limit↗.

FAQ

Does Zukko have a general rate limit?

No, there is no general speed limit of its own. A 429 comes either from your key's daily or monthly limit or because the model provider returned 429.

How is RPM different from TPM?

RPM counts requests; TPM counts tokens, input and output together. Many small requests hit RPM, a few huge ones hit TPM.

Should I retry a 429?

If it is a provider limit, yes, with exponential pauses and jitter. If it is a key limit (key_daily_limit_reached, key_monthly_limit_reached), no: raise the limit in the dashboard or wait for the next period.

How do I avoid hitting a 429 from my own key?

Set a daily and monthly limit in dollars that matches the real budget of the task and watch spending under "Usage". You can create a key and set limits in the dashboard↗.