Chatbot memory: how to store conversation history without paying twice

If you build a chatbot on GPT or another model, you quickly learn something unpleasant: chatbot memory does not exist in the API. The model stores nothing between requests, and the bot "remembers" exactly what you sent again in this call. Two consequences follow: you have to manage the history yourself, and the price of a dialogue grows with every message. Below we look at ways to store history and calculate what each costs.

Python examples go through an OpenAI-compatible API: base_url="https://api.zukko.pro/v1", key fetch_.... Request and response fields, including usage, are listed in the chat completions reference↗.

How a model "remembers"

Every request to a chat API contains the whole list of messages: the system prompt, earlier user messages and replies, and the new message. The model reads the list from scratch. All the "memory" is the messages array that you build and keep yourself (in process memory, a database, Redis).

The LLM context window is the maximum volume that fits into one request. For most models in the Zukko catalog it is 1M tokens, for GLM 128K, for Grok 256K (list and prices: GET https://api.zukko.pro/v1/models). But the window limit is not the main issue with dialogue. The main issue is money: you pay for every input token in every request.

How the cost of a long dialogue grows

Suppose the system prompt is 500 tokens, a user message is 100, and a bot reply is 300. At turn n the input is 600 + 400 × (n − 1) tokens: the prompt and the new message plus all the accumulated history.

Input grows quadratically: a dialogue twice as long costs nearly four times as much on input. For 20 turns on gpt-6.1-sol (prices as of October 18, 2026: $0.27 per 1M input, $1.32 per 1M output):

A few cents per dialogue, but for a bot with thousands of users the bill is the sum of such dialogues, and input makes up most of it. How a request price is built overall is explained in How a request price is calculated↗.

Method 1. A sliding window

The simplest option: send only the last N messages.

from openai import OpenAI

client = OpenAI(base_url="https://api.zukko.pro/v1", api_key="fetch_...")
SYSTEM = {"role": "system", "content": "You are a polite shop assistant."}
KEEP = 7  # the new message + 3 previous turns (2 messages each)

def answer(history: list, user_text: str) -> str:
    history.append({"role": "user", "content": user_text})
    resp = client.chat.completions.create(
        model="gpt-6.1-sol",
        messages=[SYSTEM] + history[-KEEP:],
    )
    text = resp.choices[0].message.content
    history.append({"role": "assistant", "content": text})
    return text

Recalculating the same 20 turns: at turn n the window holds at most three earlier turns, so input is 600 + 400 × min(n − 1, 3). That gives 33,600 input tokens instead of 88,000. Cost on gpt-6.1-sol: 33,600 × 0.27 / 1e6 + 0.00792 = $0.01699, against $0.03168.

The price of this approach: the bot forgets everything older than the window, including the user's name and agreements. That is why the window is almost always combined with the second method.

Method 2. Summarizing the old part

When the history exceeds a threshold, old messages are folded into a short summary, and the request carries the summary plus the latest messages.

def compress(history: list, summary: str) -> tuple[list, str]:
    old, recent = history[:-4], history[-4:]
    if not old:
        return history, summary
    text = "\n".join(f"{m['role']}: {m['content']}" for m in old)
    resp = client.chat.completions.create(
        model="deepseek-v4.1-flash",
        messages=[
            {"role": "system", "content": "Compress the dialogue into notes of 150 words: facts about the user, agreements, open questions. Invent nothing."},
            {"role": "user", "content": f"Previous notes: {summary}\n\nNew messages:\n{text}"},
        ],
    )
    return recent, resp.choices[0].message.content

Mix the summary into the system message: "Dialogue notes: ...". The summarization call itself is cheap: 3000 input tokens and 300 output tokens on deepseek-v4.1-flash is 3000 × 0.013 / 1e6 + 300 × 0.038 / 1e6 ≈ $0.00005. The downside is loss: a summary drops details, and an error in it gets locked in. For important facts (name, order number, constraints) it is more reliable to keep them as a structure in a database and insert them into the prompt.

Method 3. Prompt caching

Caching does not reduce the history, but it makes resending the same beginning of a request cheaper. Dialogue history fits perfectly: every new request starts with the previous one.

The Claude formula is in Prompt caching: reads and writes↗; in short, a cache read costs noticeably less than input and a write costs more. It matters which endpoint a Claude bot uses: only with /v1/messages does the gateway get the split into cache reads and writes and bill them at their own prices, while through /v1/chat/completions the whole Claude history is billed as ordinary input.

An example for claude-sonnet-5-5 (input 0.4, output 2, cache read 0.05, write 0.5), same 20 turns. In every request except the first, everything from the previous one is read from cache, and the new 400 tokens (the last reply and the new message) are written to cache:

Code with the official Anthropic SDK (address without /v1, the SDK appends it):

import anthropic

client = anthropic.Anthropic(base_url="https://api.zukko.pro", api_key="fetch_...")

def answer(history: list, user_text: str) -> str:
    history.append({"role": "user", "content": user_text})
    # cache mark only on the last message: a request allows a limited number of marks
    messages = history[:-1] + [{"role": "user", "content": [
        {"type": "text", "text": user_text, "cache_control": {"type": "ephemeral"}}
    ]}]
    resp = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=1024,
        system="You are a polite shop assistant.",
        messages=messages,
    )
    u = resp.usage
    print(u.input_tokens, u.cache_read_input_tokens, u.cache_creation_input_tokens)
    text = resp.content[0].text
    history.append({"role": "assistant", "content": text})
    return text

The cache_control mark on the last message asks to cache the request up to that point; put it on the last message only, since the number of marks per request is limited. A cache has a minimum prefix length and a limited lifetime (by default about five minutes at Anthropic), so in short dialogues and with long pauses the gain is smaller than calculated. Check Anthropic's current caching documentation.

For GPT models the cache is visible in both formats: tokens read from cache arrive in usage.prompt_tokens_details.cached_tokens, and there is no separate charge for writing. If the cache hit the whole accumulated part, the 20-turn calculation for gpt-6.1-sol (read 0.013) gives 79,800 × 0.013 / 1e6 + 8,200 × 0.27 / 1e6 + 0.00792 ≈ $0.01117. That is the upper bound of the gain: whether the cache hits and on what share of the request is decided by the provider, so look at the real cached_tokens values.

What to choose

Before building a complex scheme, measure: look at the key's daily spend in the dashboard and compare with the number of dialogues. You can learn a request's size in advance via the token counting endpoint. If you are choosing a model for a bot, How to choose a model for a chatbot↗ will help.

To try it, create a key in the dashboard, put it into the example above and print usage after each turn: you will see the growth of input yourself.

FAQ

Why did the model "forget" what I said last time?

Because the history was not in this request. The model stores nothing between requests. If a user returned the next day, the history has to be fetched from your database and sent again, or a summary passed instead.

Will the whole dialogue fit into the context window?

For most models with a 1M-token window, yes, and for a very long time. But fitting and being cheap are different things: the longer the history, the more you pay for input in every request.

Do I need to keep history in a database?

If the bot outlives one process or serves many users, yes. Any database keyed by dialogue id will do. Store the original messages and treat the summary as derived data.

Can summarization be combined with caching?

Yes, but note that when the summary is rebuilt the beginning of the request changes and the cache of the previous prefix no longer matches. It is better to recompute the summary rarely, in batches, rather than on every message.