Timeouts on long generations: how to avoid them

A long response that cuts off mid-sentence or mid-JSON looks like a server-side failure. It almost never is: the cause is a client or network timeout, not a token limit. The model would have kept generating, but the connection already closed on your end or at some point in the network.

Why it isn't a token limit

A token limit is set explicitly via max_tokens, and hitting it doesn't look like a cutoff — the model simply stops at the boundary, and the response arrives complete and syntactically whole within whatever it managed to generate. A break in the middle of a structure, by contrast, almost always means the connection died before the model finished: either the client library closed it on its own timeout, or a proxy or load balancer somewhere along the path killed a long-hanging connection.

The first thing worth checking in that situation is both parameters at once: max_tokens in the request body (is it set too low, deliberately or by a library default) and the HTTP client's timeout (does it expire before the model has a chance to finish generating).

Default timeouts are rarely sized for an LLM

The timeout values that HTTP libraries and SDKs ship with by default were designed for ordinary web requests — fractions of a second, a few seconds at most. Text generation with reasoning (extended thinking) or a large amount of context is a fundamentally different kind of operation in terms of duration, and the default timeout simply isn't sized for that wait. The sane approach is not to rely on the default at all, but to set the client's timeout explicitly, with margin relative to the expected duration of that specific request:

import httpx

client = httpx.Client(timeout=httpx.Timeout(300.0, connect=10.0))
response = client.post(
    "https://api.zukko.pro/v1/messages",
    headers={
        "Authorization": "Bearer fetch_your_key",
        "anthropic-version": "2023-06-01",
    },
    json={
        "model": "claude-opus-5-5",
        "max_tokens": 8000,
        "messages": [{"role": "user", "content": "A long reasoning task"}],
    },
)

An explicit 300-second timeout for a heavy extended thinking request isn't a guarantee it covers every case, but it beats an invisible 10-to-30-second default that will drop the connection almost every time.

Tell a timeout apart from a 503

If a model is unavailable at the provider (provider_unavailable) or switched off in Zukko (model_disabled), the server responds fast — a genuine 503 arrives with a response body, and it happens almost instantly rather than after minutes of waiting. A real timeout looks different: the client simply gets no response at all within the entire configured window, the connection hangs rather than closing right away with an error code. If the error arrives quickly, it isn't a timeout — the issue is that specific model's availability at the provider. If a request hangs and only cuts off once your own timeout expires, the question is your client's settings, not the gateway.

Retrying safely without a double charge

The real danger after a timeout is not knowing whether the request actually completed server-side, and retrying blind, risking payment for the same generation twice. The risk is real: a client-side disconnect doesn't stop the charge — the gateway reads the provider's response to the end and bills the whole generation, even if you never received it. That's what the Idempotency-Key header is for: repeating a request with the same key value within 24 hours returns the same result as the first successful call, without charging again.

curl https://api.zukko.pro/v1/messages \
  -H "Authorization: Bearer fetch_your_key" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -H "Idempotency-Key: task-42" \
  -d '{"model":"claude-opus-5-5","max_tokens":8000,"messages":[{"role":"user","content":"A long task"}]}'

If the client dropped the connection on a timeout but the request had already completed server-side, retrying with the same Idempotency-Key returns the finished result instantly instead of running the whole generation again — and doesn't charge a second time. One caveat: changing the body on a retry while reusing the same key gets a 422 with idempotency_conflict, so the key needs to map to exactly one request body. The key is set once per logical request and reused on every retry of it — never generate a new one per attempt, or each attempt gets billed separately.

What to do about genuinely long tasks

For tasks that are known to take a while — large context, extended thinking, a heavy generation with a high max_tokens — stretching the timeout toward infinity isn't always practical. It's more sensible to estimate the rough duration ahead of time with smaller test requests, set a timeout with several times that margin, and make sure such requests always carry an Idempotency-Key, so that a dropped connection doesn't turn into a risk of paying twice on retry.

Timeouts on intermediate hops

The HTTP client's own timeout isn't the only bottleneck. If a request travels through a corporate proxy, a load balancer, or a CDN before reaching the gateway, each of those hops can have its own limit on how long it holds a connection open, independent of the client library's settings. The symptom is the same — a cutoff mid-response — but raising the timeout in code won't fix it if the bottleneck isn't in your code at all but in the infrastructure between the client and the gateway. If bumping the client timeout doesn't help, it's worth checking whether a proxy or VPN sits on the path with its own connection-duration limits.

Tools like Claude Code and Cursor

Ready-made tools that talk to the gateway under the hood — Claude Code, Cursor, Cline, Roo Code — usually don't expose their request timeout as a project environment variable; it's baked into the client's internal logic. If such a tool regularly cuts off long responses from heavy models like claude-opus-5-5 with extended thinking on, it often helps to stop chasing maximum quality on every single call and instead deliberately cap max_tokens for the task at hand — a short, precise answer is more reliable than a long one that might not make it through in full.