Why Russian text tokenizes differently than English

If you've ever pasted the same prompt in Russian and English and noticed the token count in the response differ by fifty to a hundred percent against Russian, that's not a bug or some kind of billing markup. It comes down to how each model's tokenizer works, and it's worth understanding before you plan a budget for a large Russian-language project.

What actually gets split into tokens

No model in the catalog — not claude-opus-5-5, not gpt-5.6-sol, not glm-5.3 — reads text letter by letter or word by word. Under the hood sits a tokenizer (usually some variant of BPE, byte-pair encoding) that splits input text into chunks according to a vocabulary trained on a huge text corpus. That vocabulary is a finite list — for most modern models, tens or hundreds of thousands of entries. Character sequences that appear often in the training data get their own token; rare ones get broken into smaller pieces, sometimes down to individual bytes.

English text has historically dominated training corpora: whole English words, and even common phrases, land in one or two tokens for most tokenizers. Russian text occupies a much smaller share of the same corpora, and Cyrillic is also just a different Unicode range with different letter-combination frequencies. As a result, a word that fits in one token in English often breaks into three or four in Russian — sometimes down to individual letters, if that particular word form wasn't seen often enough during tokenizer training.

How to check it yourself

No need to take this on faith — you can measure it right now without spending a cent. The POST /v1/messages/count_tokens endpoint counts tokens for free, with no call to the model:

curl https://api.zukko.pro/v1/messages/count_tokens \
  -H "Authorization: Bearer fetch_your_key" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "messages": [{"role": "user", "content": "Explain how database indexing works"}]
  }'

curl https://api.zukko.pro/v1/messages/count_tokens \
  -H "Authorization: Bearer fetch_your_key" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "messages": [{"role": "user", "content": "Объясни, как работает индексация в базах данных"}]
  }'

Run both and compare the token count field in each response — the Russian version will almost always come out noticeably higher in token count for a comparably meaningful phrase, even though it's usually shorter by word count. That gap comes purely from tokenization, not from the model charging a different rate per token — the per-token price for a given model is the same regardless of the request's language.

Why this isn't a minor detail

Since billing runs on tokens, not characters or words, the same unit of meaning in Russian typically costs more on average. That shows up most clearly in:

Long system prompts. A detailed agent instruction in Russian running a couple thousand words can take up noticeably more context space than an equivalent instruction in English — pushing you closer to the context limit and leaving less room for conversation history.

RAG and document processing. When feeding Russian-language documents to a model for analysis, budget more context headroom than the file size in megabytes or word count would intuitively suggest.

Streaming and truncated responses. If a response cuts off mid-way because of max_tokens, that happens earlier in terms of "meaning conveyed" for Russian text than the English-language intuition would suggest — set max_tokens with extra headroom specifically for Russian-language output.

Different models, different tokenizers

Claude, the GPT lineup, and GLM each use different tokenizers, trained on different language proportions in their source data. So there's no single universal multiplier like "Russian always costs 2x English" — the ratio varies by model and by text (casual everyday phrases and specialized technical text full of terminology tokenize differently). The only reliable way to get an exact number for your specific text and model is to run it through count_tokens before budgeting for a large batch of requests.

What to actually do about it

If you're estimating project cost ahead of time — say, processing thousands of Russian-language user messages with one of the catalog's models — don't eyeball the token count from word or character counts. Run a representative sample of your real text through count_tokens for the model you plan to use and build the estimate from the actual numbers. It's free, doesn't touch the model, and takes far less time than figuring out later why your budget came in fifty percent over forecast.