LangChain text splitters: how to cut text into chunks correctly

LangChain text splitters cut a long document into fragments (chunks) that fit into the model context and are easy to search. For RAG and summarization both quality and price depend on this step: chunks that are too big waste tokens, chunks that are too small lose meaning. The main trap is that RecursiveCharacterTextSplitter measures size in characters by default, while the model and the bill count tokens. Below we go through the parameters, show counting in tokens and a way to check it against an exact number from the gateway. The code follows the LangChain documentation (the Text splitters section); check signatures against it.

LangChain text splitters: how RecursiveCharacterTextSplitter works

According to the LangChain documentation this is the recommended splitter for generic text. It takes a list of separators and tries to split on them in order until the pieces are small enough. The default list is ["\n\n", "\n", " ", ""]. That is, it first tries to keep paragraphs whole, then lines, then words. The parameters you will tune:

• chunk_size sets the maximum chunk size, measured by length_function (by default len, that is, in characters);
• chunk_overlap sets the target overlap between neighboring chunks so a thought at the boundary is not lost;
• length_function defines what measures the length;
• separators lets you set your own separators.

pip install -qU langchain-text-splitters

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " ", ""],
)

with open("report.txt", encoding="utf-8") as f:
    text = f.read()

parts = splitter.split_text(text)                 # list of strings
docs = splitter.create_documents([text])          # list of Document
print(len(parts), len(docs[0].page_content))

split_text returns strings, create_documents returns Document objects for vector stores, and split_documents splits ready-made documents. In the separator list we added ". ": for Russian text it is a sentence boundary, and the default list lacks it. There are specialized splitters for particular formats: for example, MarkdownHeaderTextSplitter cuts by headings, and RecursiveCharacterTextSplitter.from_language(...) respects programming language syntax.

Why characters are the wrong unit

A token is not a character. How many characters fall on a token depends on the model's tokenizer and on the language of the text, and Russian text usually has a different density than English. For example, a 1000-character chunk of English text and of Russian text takes a different number of tokens, so the same chunk_size gives different costs. That is why for RAG and summarization it is more convenient to set the size in tokens. More: why Russian text is counted differently↗.

Counting in tokens with tiktoken

The LangChain documentation describes a way to measure size with the tiktoken tokenizer: the splitters have a from_tiktoken_encoder constructor. For a hard size limit the documentation recommends exactly RecursiveCharacterTextSplitter.from_tiktoken_encoder: every piece that exceeds the size is split recursively.

pip install -qU tiktoken

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=300,
    chunk_overlap=40,
)
chunks = splitter.split_text(text)

Here chunk_size=300 is already in tokens. The documentation warns separately: using TokenTextSplitter directly can split a multi-byte character between chunks (a problem for languages such as Chinese and Japanese; it is possible for other scripts too if a character is encoded by several tokens); from_tiktoken_encoder on the recursive splitter avoids it. And there is a caveat on accuracy: tiktoken is OpenAI's tokenizer and suits OpenAI models better. For Claude, Gemini, DeepSeek or GLM it gives an estimate, not an exact number.

Checking against an exact count through the gateway

Zukko has POST /v1/messages/count_tokens: it counts the input tokens of a request in the Anthropic Messages format without generating anything. According to the reference this is a paid request (tokens are billed at the model's input price), counting is not supported for GLM, and you should pick a measurement model from the supported ones. So it makes sense not to count every chunk but to calibrate a coefficient once on a sample:

import os
import requests


def exact_tokens(text: str, model: str = "claude-sonnet-5") -> int:
    r = requests.post(
        "https://api.zukko.pro/v1/messages/count_tokens",
        headers={
            "Authorization": f"Bearer {os.environ['ZUKKO_API_KEY']}",
            "anthropic-version": "2023-06-01",
            "content-type": "application/json",
        },
        json={"model": model, "messages": [{"role": "user", "content": text}]},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()["input_tokens"]


sample = text[:20000]
chars_per_token = len(sample) / exact_tokens(sample)
print(f"characters per token: {chars_per_token:.2f}")

Say the sample came to 8,000 tokens. Then the measurement costs 8,000 × 0.4 / 1M = $0.0032 (the claude-sonnet-5 price as of October 18, 2026). The number includes a few service tokens of the message wrapper, which is acceptable for an estimate. We plug the resulting coefficient into the character-based splitter:

TARGET_TOKENS, OVERLAP_TOKENS = 300, 40

splitter = RecursiveCharacterTextSplitter(
    chunk_size=int(TARGET_TOKENS * chars_per_token),
    chunk_overlap=int(OVERLAP_TOKENS * chars_per_token),
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(text)
print(max(exact_tokens(c) for c in chunks[:10]))

The last line checks on ten chunks that the maximum is close to the target. The endpoint description: count_tokens↗. How to use it to estimate the cost of a request in advance is covered in the article calculate the request cost with count_tokens↗.

Which size and overlap to choose

There is no universal answer, but there are starting points:

• RAG. Start with a few hundred tokens per chunk. Small chunks are found more precisely, large ones keep more context. Test on 20 to 30 real questions whether the answer lies in the retrieved fragments.
• Summarization (map-reduce). Chunks can be larger, up to the size at which the model still performs well. Zukko model contexts range from 128K (GLM) to 1M, so for a single document chunking is sometimes not needed at all.
• Overlap. A guide is 10 to 15% of the chunk size; this is a starting point, not a rule. Overlap costs money: repeated text is paid for twice.

What overlap costs: the math

The number of chunks for a text of N tokens, chunk size c and overlap o is roughly N − o divided by c − o, rounded up. Say N = 100,000, c = 500, o = 50. Then (100,000 − 50) / (500 − 50) = 222.1, that is, 223 chunks. The total volume of chunks is about 223 × 500 = 111,500 tokens, an excess of about o / (c − o) = 11%. Suppose for every chunk the model produces a 150-token summary (33,450 output tokens in total). Prices as of October 18, 2026, math in millionths of a dollar:

Without overlap there would be 100,000 input tokens, so 11,500 tokens (about $0.003 on gpt-6.1-sol) go to duplicates. This is a fair price for preserving meaning at the boundaries, but do not inflate the overlap without need. For Claude in the OpenAI format the whole input is billed at the input price, and a cache discount is possible in the Anthropic format. Fresh prices are always in the GET https://api.zukko.pro/v1/models response.

FAQ

How do I split code and Markdown?

Use specialized splitters: MarkdownHeaderTextSplitter for headings and RecursiveCharacterTextSplitter.from_language for code. They preserve structure better than plain separators.

Do I need chunking if the model context is a million tokens?

For summarizing a single document, often not. For RAG chunks are needed anyway: you send the model only relevant fragments and pay for them, not for the whole document.

Why are my chunks larger than the set size?

For some splitters the documentation warns that pieces can exceed the set size. A hard limit comes from RecursiveCharacterTextSplitter.from_tiktoken_encoder.

Which tokenizer should I choose for non-OpenAI models?

The LangChain documentation advises using the tokenizer of the same model you call. For Claude models count_tokens gives an exact count, and for others tiktoken serves as an estimate with a margin.

Create a key in the dashboard↗ and check your own coefficient on a sample. For RAG on ready chunks see a RAG pipeline by hand↗.