Which AI model to choose for a chatbot: dialog cost, cache, testing

"Which AI model should I choose for a chatbot" comes up for anyone building a support bot, a website assistant or a Telegram helper. Comparing models by price per 1M tokens isn't enough: on every reply the bot resends its instructions and the history, and the bill grows with the conversation length. Below: the selection criteria, the cost of 1,000 dialogs on three models with and without caching, and code to measure latency yourself.

Which AI model to choose for a chatbot: five criteria

1. Dialog cost, not token price

The API doesn't remember past messages. For the bot to answer in context, on every turn you send the system prompt plus the entire history. So input grows with each turn, and most of the bill is resent instructions and history.

Cost of one request: (input × input price + output × output price) / 1,000,000. Dialog cost is the sum over all turns. Calculate it for your typical scenario: instruction length, number of turns, message length.

2. Context

All models in the Zukko catalog have a 1M-token context, except glm-5.3 and glm-5.3-flash at 128K. For ordinary chat that's plenty. Context becomes a criterion if you inject large documents into the prompt or keep a very long history. Even when it fits, long context costs money: trim old history or fold it into a short summary.

3. Intelligence index

Models in the catalog carry an intelligence index — the Artificial Analysis Intelligence Index, a composite benchmark score. Values: Claude Opus 5.5 — 58, Claude Fable 5.1 — 53, GPT-6 Astra — 53, GPT-6.1 Sol — 52, Qwen 3.8 Max — 45, GLM 5.3 — 45, GPT-5.6 Terra — 42, GLM 5.3 Flash — 42, Gemini 3.8 Flash — 41, Qwen 3.8 Flash — 40, DeepSeek V4.1 Flash — 39, GPT-6 Luna — 38, DeepSeek V4 Pro — 36. The catalog has no index value for Claude Sonnet 5.

The index is averaged over many tasks and doesn't tell you whether a model handles your particular dialog well. Use it for a first cut and decide based on tests with real conversations.

4. Latency

In a chat, what matters is how fast the first word of the answer appears. Latency depends on the model, the input and output length, and the provider's current load, so don't trust other people's numbers — measure it yourself (code below). Two rules always apply:

• Turn on streaming (stream: true): the user sees the answer being typed instead of waiting for all of it.
• Cap max_tokens: a long answer takes longer to generate and costs more.

5. Image input

If users send screenshots or photos, you need a model that accepts images. In the catalog, image input is supported by, for example, claude-sonnet-5, claude-opus-5-5, gpt-6.1-sol, qwen3.8-max, deepseek-v4-pro. Not supported: deepseek-v4.1-flash, gemini-3.8-flash, gpt-6-luna, glm-5.3, glm-5.3-flash and others marked "no" in the catalog.

System prompt caching

A bot's system prompt — instructions, tone, rules, company info — is the same in every request. Providers can cache such a repeated prefix, and reading from cache costs less than regular input. How Zukko accounts for it:

• Claude — the cache discount works in the Anthropic format (/v1/messages, Anthropic SDK). A cache write costs more than input, a read costs less. Caching is turned on with cache_control markup. Per Anthropic's docs, the cache lives 5 minutes by default and is refreshed on each use, and the minimum cacheable prefix for Claude Sonnet 5 is 1,024 tokens. In the OpenAI format, all Claude input is billed at the input price.
• GPT — OpenAI enables caching by default; the minimum prefix for GPT-5.6 and later is 1,024 tokens. Hits show up in prompt_tokens_details.cached_tokens in both formats. Cache writes aren't charged separately: the first request is billed at the input price.
• DeepSeek, GLM and others — cache reads are counted if the provider reports them in the response; writes are not. For deepseek-v4.1-flash, the catalog's cache-read price equals the input price ($0.013 per 1M tokens, prices as of October 7, 2026), so caching doesn't change the bill.

An example Claude request with system prompt caching via the Anthropic SDK:

from anthropic import Anthropic

client = Anthropic(base_url="https://api.zukko.pro", api_key="fetch_your_key")

with open("bot_instructions.txt", encoding="utf-8") as f:
    SYSTEM_PROMPT = f.read()  # about 3,000 tokens


def reply(history):
    msg = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=500,
        system=[{"type": "text", "text": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}],
        messages=history,
    )
    u = msg.usage
    print("input:", u.input_tokens, "cache write:", u.cache_creation_input_tokens,
          "cache read:", u.cache_read_input_tokens, "output:", u.output_tokens)
    return msg.content[0].text

If both cache fields are zero, the prompt wasn't cached — for example, it's shorter than the minimum prefix. More on cache pricing in the key limits and cache docs↗.

The math: 1,000 dialogs on three models

Prices as of October 7, 2026, USD per 1M tokens. Models: deepseek-v4.1-flash (index 39), gpt-6.1-sol (52) and claude-sonnet-5 (no index value in the catalog).

Dialog scenario: a 3,000-token system prompt, 6 turns, a 50-token user question, a 200-token bot answer. On turn k, input = 3,000 + 250 × (k − 1) + 50.

• Input per dialog: 6 × 3,050 + 250 × (0 + 1 + 2 + 3 + 4 + 5) = 18,300 + 3,750 = 22,050 tokens.
• Output per dialog: 6 × 200 = 1,200 tokens.

Without caching, formula for 1,000 dialogs: 1,000 × (22,050 × input + 1,200 × output) / 1,000,000.

With system prompt caching. Of the 22,050 input tokens, 6 × 3,000 = 18,000 are the repeated system prompt; the remaining 4,050 are history and questions.

• claude-sonnet-5 in the Anthropic format (cache read $0.05, write $0.5). If the cache is rewritten at the start of every dialog: 4,050 × 0.4 + 3,000 × 0.5 + 15,000 × 0.05 + 1,200 × 2 = 1,620 + 1,500 + 750 + 2,400 = 6,270 → $6.27 per 1,000 dialogs. If dialogs come close together and the cache doesn't expire, all 18,000 tokens are read from cache: 1,620 + 900 + 2,400 = 4,920 → $4.92.
• gpt-6.1-sol (cache read $0.013, writes not charged). The first turn is at the input price, the next five read the prompt from cache: (22,050 − 15,000) × 0.27 + 15,000 × 0.013 + 1,584 = 1,903.5 + 195 + 1,584 = 3,682.5 → $3.68. With a warm cache: 4,050 × 0.27 + 18,000 × 0.013 + 1,584 = 1,093.5 + 234 + 1,584 = 2,911.5 → $2.91.
• deepseek-v4.1-flash — the cache-read price equals the input price, so it stays at $0.33.

For comparison: the highest-index model, claude-opus-5-5 (58, $0.8 / $4), in the same scenario without caching costs 22,050 × 0.8 + 1,200 × 4 = 17,640 + 4,800 = 22,440 → $22.44 per 1,000 dialogs.

Takeaways from the math:

• The longer the system prompt and the dialog, the larger the share of repeated input and the bigger the effect of caching.
• For Claude in a chatbot with long instructions, the Anthropic format makes sense — otherwise there's no cache discount.
• GPT may also cache the dialog history, in which case the real cost is below the estimate; check cached_tokens.

This is a sample calculation: vendor prices change, so redo it with your own numbers. For the current list and prices, call GET https://api.zukko.pro/v1/models or open the Models page in the dashboard.

How to measure latency yourself

The script sends the same question to several models in streaming mode and measures the time to the first text and to the end of the answer:

import time

from openai import OpenAI

client = OpenAI(base_url="https://api.zukko.pro/v1", api_key="fetch_your_key")


def measure(model, question):
    start = time.perf_counter()
    first = None
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": question}],
        max_tokens=300,

The if chunk.choices check is required: at the end of the stream Zukko sends a separate chunk with usage and an empty choices. Run the measurement several times and at different times of day — a single run tells you little.

How to choose: step by step

1. Filter models by hard requirements: image input, context length.
2. Price a dialog for 2–3 candidates on your scenario, with and without caching.
3. Collect 30–50 real dialogs with correct answers and run the candidates with the same prompt.
4. Measure latency with the script above.
5. Take the cheapest model that clears your quality bar. For hard questions you can add escalation to a stronger model — the setup is described in the article on model cascades↗.

A ready bot backend with streaming and limits is covered in the article an AI chatbot for your website↗. One key is enough for such a comparison: all models are called through one endpoint, only model changes. You can create a key on the API keys↗ page.

FAQ

Why does a dialog cost more than the sum of questions and answers?

Because on every turn the model receives the system prompt and the entire history again. In the calculation above the user wrote 300 tokens and the bot 1,200, yet input added up to 22,050 tokens.

Do I need caching if the system prompt is short?

If the prompt is shorter than the minimum cacheable prefix (1,024 tokens for Claude Sonnet 5 and for GPT-5.6 and later, per the vendors' docs), caching won't kick in. For short prompts caching barely affects the bill.

Can I switch the bot's model without rewriting code?

Yes. Through Zukko any model can be called in the OpenAI or Anthropic format, so switching only requires changing model. Keep the model name in config, and remember that the Claude cache discount requires the Anthropic format.

Which model should I use for a Russian-language bot?

We have no data that would let us rank models fairly by Russian-language quality. Run 30–50 real conversations through 2–3 candidates and compare the answers.