Caching LLM responses in Redis: stop paying for repeated requests

Classifying reviews, extracting fields from documents, translating similar strings — in these pipelines the same requests repeat, and without a cache you pay for every repeat. Caching LLM responses in Redis solves this on your application's side. But first it's worth understanding what the Zukko gateway already does so you don't build something unnecessary: below are the server cache, the cases where you need your own, and working redis-py code.

What Zukko caches on its own

Zukko has a content-based request cache (details in key limits, idempotency and cache↗):

• If you repeat a request with the same model and the same body at temperature: 0, the second response is served from cache for free.
• At a different temperature, enable the cache with the X-Zukko-Cache: true header; disable it in any case with X-Zukko-Cache: false.
• A hit is visible in the X-Zukko-Cache: hit response header.
• Newer Claude models (claude-sonnet-5, claude-opus-5-5) may reject a request with a temperature other than 1 with a 400 error. So don't pass temperature to them, and enable the cache explicitly with the X-Zukko-Cache: true header.
• The cache is per account, errors are not cached, streaming and regular responses are stored separately, and entries live for 7 days.

The key limitation: the whole request body must match. An extra space in the text, a different max_tokens, a different message order or a new date in the system prompt — and it's a different request you pay for.

Separately from the cache, there is the Idempotency-Key header: repeating a request with the same key returns the stored response without charging twice (the response is kept for 24 hours). This is protection against double billing on retries, not a cache.

When you need your own Redis cache

If your repeats are exact copies and 7 days is enough, the server cache will do. Your own Redis makes sense when:

• Inputs are nearly identical. Users send the same text with different spacing, case or line breaks. Normalize the input before building the key — the server cache can't do that.
• You need more than 7 days, or precise control over when an answer becomes stale.
• You need invalidation by prompt version. You changed the instruction, so old answers must stop being used.
• Latency matters. A response from a local Redis arrives without a network call to the gateway.
• You cache processed results rather than raw responses, such as already-parsed JSON.

What not to cache: creative texts where variety is expected, answers that depend on time or external data, and agent steps where context changes every turn.

How to build the cache key

The key must include everything that affects the answer: the model, all messages (including the system one), and generation parameters. Plus a version prefix you change when the prompt changes.

import hashlib
import json
import re

PREFIX = "llm:v1:"

def normalize(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip()

def cache_key(model: str, messages: list, params: dict) -> str:
    norm = [{"role": m["role"], "content": normalize(m["content"])} for m in messages]
    payload = json.dumps(
        {"model": model, "messages": norm, "params": params},
        sort_keys=True, ensure_ascii=False,
    )
    return PREFIX + hashlib.sha256(payload.encode("utf-8")).hexdigest()

sort_keys=True makes serialization stable: dictionaries with the same content produce the same string. Normalization here only collapses whitespace; if case doesn't matter for your task, add .lower(). The model still receives the original text — only the key is normalized.

Caching LLM responses in Redis: redis-py code

Install: pip install redis openai. The example uses the OpenAI SDK with the base URL https://api.zukko.pro/v1.

import json
import redis
from openai import OpenAI

r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
client = OpenAI(base_url="https://api.zukko.pro/v1", api_key="fetch_your_key")

TTL_SECONDS = 30 * 24 * 3600

def ask(messages: list, model: str = "claude-sonnet-5", **params) -> dict:
    params.setdefault("max_tokens", 512)
    key = cache_key(model, messages, params)

    cached = r.get(key)

What matters here:

• decode_responses=True makes Redis return strings instead of bytes, so json.loads works directly.
• r.set(key, value, ex=...) writes the value together with a TTL in seconds; the equivalent is r.setex(key, ttl, value).
• temperature is not passed: that's safer for claude-sonnet-5. For GPT and GLM you can add temperature=0 to the ask(...) call — it goes into both the request and the cache key.
• extra_headers={"X-Zukko-Cache": "true"} explicitly enables Zukko's server cache, and with_raw_response gives access to response headers: server_cache will contain hit if the answer came from that cache.
• The llm:stats:hit and llm:stats:miss counters show the hit rate.

The Anthropic SDK works the same way: build the key identically, call client.messages.create, read the text from resp.content[0].text, and set the base URL to https://api.zukko.pro without /v1.

Invalidating without KEYS

The KEYS command blocks Redis while it walks the whole database, so in production use SCAN to find keys — in redis-py that's scan_iter:

def invalidate(prefix: str = PREFIX) -> int:
    deleted = 0
    for key in r.scan_iter(match=prefix + "*", count=1000):
        deleted += r.delete(key)
    return deleted

Most of the time you don't need to delete anything: change PREFIX to llm:v2: and old entries simply expire by TTL. To delete a single answer, use r.delete(cache_key(model, messages, params)).

Monitoring the cache

hits = int(r.get("llm:stats:hit") or 0)
misses = int(r.get("llm:stats:miss") or 0)
total = hits + misses
print(f"Hits: {hits} of {total}")
print("Memory:", r.info("memory")["used_memory_human"])
print("Keys in DB:", r.dbsize())

If the hit rate is close to zero, your own cache isn't worth the complexity — keep only the server one. To stop Redis from growing forever, set a memory limit and an eviction policy (for example, allkeys-lru) in its configuration — see the Redis documentation.

If one Redis server isn't enough, redis-py has RedisCluster (from redis.cluster import RedisCluster); connect with the host and port of any node or a list of ClusterNode objects, and the rest of the code stays the same.

How much you can save

Example for claude-sonnet-5 ($0.3 per 1M input and output tokens). Suppose a pipeline classifies 10,000 reviews, averaging 100 input and 50 output tokens per request:

• cost per request: (100 × 0.3 + 50 × 0.3) / 1,000,000 = $0.000045;
• without a cache: 10,000 × 0.000045 = $0.45;
• if after normalization 40% of reviews match already processed ones, 6,000 requests are paid: 6,000 × 0.000045 = $0.27.

Part of that 40% (exact copies) would also be covered for free by Zukko's server cache. Your own Redis adds hits for nearly identical inputs, storage beyond 7 days and responses without a network call. Measure the real share of repeats from your logs — the whole benefit depends on it.

You can create a separate key for the pipeline with a daily limit on the API keys↗ page, and a client setup example is in the Python guide↗.

FAQ

Do I need Redis if Zukko already has a cache?

Not always. For exact repeats the server cache is enough (it turns on by itself at temperature: 0 or with the X-Zukko-Cache: true header). Redis is useful when inputs differ in small details, or you need more than 7 days, prompt versioning or minimal latency.

Can I cache responses at temperature above 0?

Technically yes, but you deliberately give up variety: the user always gets the same answer. On Zukko's side, enable it with X-Zukko-Cache: true; in your own Redis, just store the answer. For Claude models, where it's better not to pass temperature, the header is the only way to enable the server cache.

Can I use a Python dict instead of Redis?

For a script or tests, yes. But such a cache disappears on restart and isn't shared between processes or servers.

Should I cache streaming responses?

In your own Redis it's easier to store the complete answer after the stream ends. On Zukko's side, streaming and regular responses are cached separately.