LLM API error handling: retries, 429, 503 and a fallback model

A bot or pipeline that dies on the first 503 has to be restarted by hand. A script that endlessly retries a request with an invalid key just heats up the server. Solid LLM API error handling comes down to three decisions: which codes to retry, how long to wait between attempts, and when to switch to another model. Below: a table of Zukko error codes and ready-to-use Python code.

Error codes and what to do about them

Zukko returns errors as a regular HTTP status with a JSON body in OpenAI format; the specific code is in the error.code field. Handle errors by status and code, not by message text.

Add network failures and timeouts: the connection dropped, no response came back — such requests are worth retrying too.

A special case: 429 from a key limit

Every key can have a daily and monthly spending cap set in the dashboard. Once it's reached, you get a 429 with code key_daily_limit_reached or key_monthly_limit_reached. Retrying a few seconds later is pointless: the limit only resets at the start of the next day or month (UTC), or when you raise it in the dashboard. More in the key limits section↗.

LLM API error handling: retry rules

1. Exponential backoff with jitter. 1, 2, 4 seconds plus a random fraction — so a hundred workers don't hit the server at the same moment.
2. The Retry-After header. If the server sends it, wait as long as it says.
3. A bounded number of attempts. Two or three attempts on one model, then fail over to a backup or raise the error.
4. Protection against double billing. If a request timed out, it may still have reached the model. Zukko accepts an Idempotency-Key header: a retry with the same value returns the stored successful response and isn't charged again. Generate one value per logical request and send it on every retry. Two concurrent requests with the same idempotency key will both go to the model, so only retry after the previous attempt has finished.
5. Turn off the SDK's built-in retries if you write your own. By default the OpenAI SDK retries 408, 409, 429 and 5xx twice on its own. Combined with your loop, that's up to nine requests instead of three.

How to build such retries into a batch of thousands of requests is covered in the article on async processing in Python↗.

A fallback model

A 503 usually means a problem at a specific provider, so it makes sense to take the backup from a different vendor. An example chain from a strong model to a cheap one (prices as of October 6, 2026, $ per 1M tokens; the intelligence index is the Artificial Analysis Intelligence Index from the Zukko catalog):

Three practical notes:

• The backup must handle what the primary does. If your requests include images, gemini-3.8-flash won't work — use something like gpt-6-sol instead.
• Add backup models to the key's allowed_models. Otherwise you'll get a 403 instead of the backup's answer.
• Test the backup in advance. Run 20–30 typical requests through it: does it keep the response format your code expects? Finding out during an outage is too late.

For the current list and prices, see GET https://api.zukko.pro/v1/models or the Models page in your dashboard. The list itself is free and needs no key — see the /v1/models reference↗.

Python code

import os
import random
import time
import uuid

import openai
from openai import OpenAI

client = OpenAI(
    base_url="https://api.zukko.pro/v1",
    api_key=os.environ["ZUKKO_API_KEY"],
    max_retries=0,  # we handle retries ourselves
    timeout=120,
)

What's going on here:

• call_model retries on the same model only for 429 (except key limits), 5xx and network failures. Everything else immediately becomes a FatalAPIError: 401, 402 and 403 won't fix themselves on another model.
• 503 model_disabled propagates up without retries, and complete checks the rest of the chain against a fresh GET /v1/models list. If nothing is left in the chain, it takes the first available model from the EMERGENCY list.
• In the OpenAI SDK, the exception's code field comes from error.code in the response body, so there's no need to match on message text.
• The function returns the name of the model that answered — log it to see how often the fallback kicks in.

Streaming and agents

In streaming mode, the 200 status arrives before the model starts answering, and an error can show up inside the stream. Retries need to wrap the whole read loop, not just the create call. If the user has already seen half an answer, resending to another model will show them different text — in a chat it's more honest to report the failure and offer a retry.

In agents with long sessions, switching models mid-task changes both cost and behavior. With Claude, also keep caching in mind: when you switch to another model the accumulated prompt cache doesn't carry over, and the next request is billed at the fresh input price.

A separate key per service with a daily limit, set on the API keys↗ page of the dashboard, limits the damage if a retry loop does run away.

FAQ

Why not just retry every error?

400, 401, 402 and 403 are caused by the request itself, the key or the balance. A retry returns the same error and fills your logs with noise that hides the real cause.

How do I tell my own key limit apart from a provider 429?

By the error.code field: key_daily_limit_reached or key_monthly_limit_reached means you've hit the cap you set for the key yourself. Zukko has no general rate limit of its own, so any other 429 came from the model provider. More on 429 in the article about error 429↗.

What should I do about 503 model_disabled?

Don't retry: the model is switched off. Pick another one from GET https://api.zukko.pro/v1/models — the code above does this automatically.

How many attempts should I allow?

Two or three per model with a 1–4 second pause, and no more than three models in the chain. If that doesn't help, return a clear error rather than keeping the user waiting for a minute.