RAG explained in plain words and when you really need it

RAG, what is it: the abbreviation stands for Retrieval-Augmented Generation. The idea is simple: before asking the model a question, the system finds several relevant fragments in your documents and puts them in the request together with the question. The model answers based on this text, not only on what it learned in training. Below we cover how RAG works, what it consists of, and when you do not need it at all.

What problem RAG solves

A model does not know your internal documents: regulations, contracts, the knowledge base, correspondence. It also does not know what appeared after its training. If you ask about such things directly, it either honestly says it does not know or plausibly makes up an answer.

There are three ways to give a model your data:

1. Put the documents straight into the request (long context).
2. Find the needed fragments and put in only those (RAG).
3. Fine-tune the model on your data.

The third is more expensive to prepare and deserves a separate discussion. Here we compare the first two.

How RAG works: the steps

The system has two parts: preparation, done in advance, and question handling, done on every request.

Preparation.

1. Loading. Take documents (PDFs, wiki pages, a ticket export) and turn them into text.
2. Splitting into chunks. The text is divided into pieces of a few hundred words, sometimes with a small overlap so a thought is not cut at the boundary.
3. Index. For each chunk a "fingerprint" for search is computed. Most often it is an embedding, a vector of numbers used to find texts close in meaning. It is stored in a vector database or even in a plain file.

For each question.

1. Retrieval. The question is turned into the same kind of fingerprint, and the closest few chunks are taken from the index.
2. Prompt assembly. The prompt gets an instruction ("answer only from the text below"), the found chunks and the question.
3. Generation. The model answers. A good practice is to ask it to say which chunk the answer came from and to say "no data" if the answer is not in the text.

More advanced systems also add re-ranking of the found chunks and automatic quality evaluation. You do not need them to start.

A RAG system: a minimal Python example

Below is a teaching example without an external vector database. Search uses TF-IDF from scikit-learn, which measures similarity by shared words. For Russian without lemmatization this is crude (words in different forms count as different), so in a real system search is usually replaced with embeddings. But the principle is fully visible.

pip install scikit-learn openai

import os
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from openai import OpenAI

client = OpenAI(base_url="https://api.zukko.pro/v1",
                api_key=os.environ["ZUKKO_API_KEY"])

def split_text(text, size=120, overlap=20):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]

doc = open("handbook.txt", encoding="utf-8").read()

Request parameters are described in the chat completions reference↗. A fuller walkthrough with embeddings is in the article RAG by hand↗.

Where RAG goes wrong

It is worth knowing this before adoption:

• Bad retrieval. If the needed chunk did not make the top, the model answers from something else or says "no data". RAG quality depends on search at least as much as on the model: test retrieval separately, on a set of questions with known answers.
• Poor splitting. If the answer is spread across two chunks and only one made it into the request, the answer will be incomplete.
• Contradictions. Two versions of one regulation in the index will give two different answers. Delete outdated ones.
• Access rights. If the index holds documents from different departments, retrieval must take into account who is asking.

RAG or long context: what each approach costs

Modern models have a 1M-token window (GLM 5.3 has 128K, Grok has 256K). If your knowledge base fits in the window, it is simpler to skip RAG and put everything in the request. Let us compare costs with an example.

Assumptions: a knowledge base of 250 thousand tokens (on the order of 500 pages), 1000 questions per month. Output is counted as equal in both approaches and left out of the comparison. For RAG one request has 3000 input tokens (4 to 5 chunks, the instruction and the question). Prices as of 18 October 2026, $ per 1M tokens (input), formula: input tokens × price.

With long context, cache helps. If the prefix with the documents is identical in all requests, repeated requests read it from cache at a lower price. The ideal case, where the cache hits on 999 of 1000 requests:

• gpt-6.1-sol (cache is visible in both formats): 0.27 × 0.25 + 0.013 × 999 × 0.25 = 0.0675 + 3.247 = about $3.31.
• claude-sonnet-5-5 in the Anthropic format (write 0.5, read 0.05): 0.5 × 0.25 + 0.05 × 999 × 0.25 = 0.125 + 12.49 = about $12.61.

This is the lower bound. Cache is kept for a limited time, so with infrequent requests it "cools down" and some requests go at full price. The cache accounting rules are in the limits and cache documentation↗. The conclusion from the table: RAG is cheaper on input, while long context is simpler to build and better suited to questions that need to see the whole document.

When RAG is really needed

• The knowledge base is larger than the model window or grows quickly.
• There are many questions, and paying for the whole document in every request is costly.
• Documents are updated, and source references are needed.
• Access to parts of the knowledge base must be separated.

When RAG is not needed

• There are few documents and they fit in the window with room to spare.
• There are few questions.
• You need answers about the document as a whole: a summary, a comparison of two versions, finding contradictions. Fragment search loses such questions.

The general principle: first test long context on 10 real questions. If quality and price are acceptable, postpone RAG. For ready-made knowledge base search there is the article knowledge base search on the cheap↗.

FAQ

Are RAG and internet search the same thing?

No. In RAG the search runs over your own documents, prepared in advance. Internet search is a different feature where the source is external.

Does RAG remove hallucinations?

It reduces them but does not remove them. The model can misread a chunk or answer against the instruction. Require a source quote in the answer and check it.

Do I need a vector database?

For small volumes an in-memory array or a file is enough, as in the example above. A dedicated database is needed with hundreds of thousands of chunks and frequent updates.

Which model should I start with?

A cheap one, for example deepseek-v4.1-flash: it receives ready context, and the task comes down to answering from the found text. If answers are weak, check retrieval first and only then change the model. A key is created in the dashboard↗.