LLM in production: a checklist before launching an AI feature

An LLM in production breaks in different places than it did in the prototype. In the prototype the question is "can the model do this". In production: what happens when the provider answers 503, when a key hits its limit at night, when customers' phone numbers leak into logs, when the model is swapped and quality drifts. Below is a checklist of seven blocks with code you can take as a base. It is written for any project on an OpenAI-compatible API; the examples are in Python with the address https://api.zukko.pro/v1.

1. Keys and limits

• A separate key per environment and service. Development, staging and production must not share a key: then leaks and spending do not mix. Keys are created in the dashboard↗, and the full key is shown once at creation.
• A daily and monthly dollar limit on every key. This is the last line of defence against looping code: when the limit is exhausted, the key gets 429 (key_daily_limit_reached or key_monthly_limit_reached) and not another cent is spent. How limits and allowed models are configured is described in the key limits documentation↗.
• A list of allowed models (allowed_models). If the feature needs one model, allow only that one: then even a stolen key cannot run expensive models. A refusal arrives as 403 model_not_allowed_for_key.
• The key in secrets, not in code. Environment variables or a secrets manager; the key is sent in the Authorization: Bearer fetch_... header. Decide in advance who reissues the key after a leak and how: a new key is created in the dashboard and the old one is revoked.

2. Timeouts and retries

A request to a model is a network call that can take seconds or tens of seconds. Without a timeout a hung call holds a thread and, as a result, the user.

Different errors need different reactions:

Important: Zukko has no general request rate limit of its own, so 429 means either your key's limit or a limit at the model provider. Determine the cause from the code in the error body. Retrying after the key limit is exhausted is pointless: nothing changes until the next day or month starts.

3. A fallback model

Models get disabled or become unavailable at the provider. In Zukko this is a 503 with the code model_disabled; the request must be redirected to another model from GET /v1/models. Choose the fallback in advance, no worse than the main one in price and context, and verify on your data that it actually handles the task.

Code with a timeout, retries and a fallback model:

import os, time, logging
import openai
from openai import OpenAI

log = logging.getLogger("llm")
client = OpenAI(
    base_url="https://api.zukko.pro/v1",
    api_key=os.environ["ZUKKO_API_KEY"],
    timeout=30.0,
    max_retries=0,          # we retry ourselves to stay in control
)

MODELS = ["claude-sonnet-5-5", "gpt-6.1-sol"]   # main, fallback
KEY_LIMIT_CODES = ("key_daily_limit_reached", "key_monthly_limit_reached")

Adapt the code to your architecture; what matters in it is the logic: what to retry, what not to, and when to switch to the fallback. The topic is covered more widely in the article on retry and a fallback model↗. If requests get repeated, also think about protecting your system against processing the same event twice.

4. Logs without personal data

A model request journal is needed for debugging, but it easily accumulates other people's phone numbers, addresses and the content of conversations. Rules:

• In ordinary logs write metadata: model, input and output token counts, response time, result code, a request identifier from your system.
• Save request and response texts only if needed for debugging, with masking of phone numbers, e-mails and document numbers and a short retention period.
• Do not pass the model unnecessary personal data. If the task needs the customer's name but not the passport number, do not send the number.
• Check personal data legislation and your company policy before launch.

A walkthrough of techniques with code: logging LLM requests without data leaks↗.

5. Cost tracking

A response contains usage, from which you can compute the cost of each call and add it to metrics per feature and customer. For the calculation use the formula: fresh input × input price + cache read × read price + cache write × write price + output × output price. GPT models have no separate cache write, and for Claude the cache discount is visible only in the Anthropic format. An example with numbers at prices as of October 18, 2026: one claude-sonnet-5-5 call with 4,000 input tokens and 600 output tokens costs 4,000 × $0.4 / 1M + 600 × $2 / 1M = $0.0016 + $0.0012 = $0.0028, and a million such calls a month cost $2,800. That is exactly why spending alerts are needed: a small price per call at scale becomes a noticeable sum.

What to set up:

1. An alert in your monitoring system at 50% and 80% of the key's monthly limit.
2. A balance alert: a 402 error in production means the feature fails.
3. Reconciliation of your metrics with the Usage section of the dashboard and with the spending history via the API (GET /v1/usage).

6. Quality checks

Before release and before every model or prompt change you need a set of test cases. At minimum: 30–100 real examples with expected results and a script that runs them through the system and compares the answers. For structured answers check automatically (valid JSON, required fields, values from an allowed list), for free text check selectively by eye or against an explicit criterion. A simple skeleton is shown in the article on prompt testing↗.

A model version change is an event of its own. Even if the name is the same, behaviour may differ: run the set at every switch. Read the list of available models and their prices dynamically via GET /v1/models instead of hard-coding them forever.

7. Abuse protection and a kill switch

• A per-user limit on your side: requests per minute and per day. Introduce your own rate limit; the gateway imposes no general one.
• Input filtering: request length and format. A huge input from one user should not cost you money.
• Prompt injection. User data and documents may contain instructions for the model. Do not give the model rights the user does not have; tools that change data require checks and confirmation.
• A disable flag. The ability to switch the feature off without a deploy, to stop spending and errors within a minute.

The final checklist

1. A separate key, daily and monthly limit, allowed_models.
2. A timeout and a retry policy by error code.
3. A fallback model verified on your data.
4. Logs without personal data, retention period set.
5. Token and money tracking, alerts on limit and balance.
6. A set of test cases and a run at every model or prompt change.
7. Per-user limits and a kill switch.

Start with the first item: create a key with a limit in the dashboard↗. The balance is topped up via SBP, MasterCard or crypto.

FAQ

Do I need to limit request rate myself?

Yes. Zukko has no general rate limit; limits exist at the model provider and at the key level (a daily and monthly dollar limit). Set the per-user limit for your application on your side.

What should I do on a 429 error?

First determine the cause from the code in the response body. If the key limit is exhausted, do not retry: raise the limit or wait for the next period. If the limit is at the model provider, retry with a growing pause or switch to a fallback model.

Which fallback model should I choose?

A model with similar context and price, preferably from a different family so one provider's outage does not affect it. Be sure to run it on your test cases: swapping "by name" is not enough.

Can I log requests in full?

Only if you cannot debug without it, with masking of personal data and a short retention period. For everyday monitoring metadata is enough: model, tokens, time, response code.