RAG on a cheap model: knowledge base search without a vector DB

To have a bot answer questions from your documentation, you don't have to run a vector database or pay a service for embeddings. For many knowledge bases, RAG on a cheap model is enough: keyword search on your own server and answer generation by a model for a fraction of a cent. Below: the architecture, working Python code with rank_bm25, the per-query cost, and signs that it's time to move to a stronger model.

The architecture: search locally, answer with the model

1. Documents are split into chunks — paragraphs or pieces of a few hundred tokens.
2. A BM25 index is built over the chunks, right in process memory.
3. For a user's question, the index returns the 10–30 best-matching chunks.
4. The chunks and the question go to the model with an instruction to answer only from them and cite the source.

BM25 is a classic ranking algorithm based on word overlap weighted by how rare the words are. It's good at finding exact terms: product names, error codes, SKUs, policy clause numbers. It's weaker on paraphrase: if the question says "get my money back" and the document says "file a refund", the match will be weaker.

A cheap model with a long context partly compensates for this. Instead of five chunks you can pass twenty or thirty: the model ignores the extra ones, and the right one is more likely to be among them.

The index with rank_bm25

Install the package with pip install rank_bm25. The main class is BM25Okapi: it takes a list of documents, each a list of tokens, and returns scores via get_scores. The package does no text preprocessing — that's up to you.

import re
from pathlib import Path

from rank_bm25 import BM25Okapi


def tokenize(text: str) -> list[str]:
    return re.findall(r"\w+", text.lower())


chunks = []
for path in Path("kb").glob("**/*.md"):
    text = path.read_text(encoding="utf-8")
    for i, part in enumerate(re.split(r"\n\s*\n", text)):

The code targets Python 3.9+. A couple of notes:

• For heavily inflected languages such as Russian, plain word splitting misses some matches. If you see this on your questions, add stemming — reducing words to their stem — to tokenize with any library you trust. The key is to process documents and questions the same way.
• Short paragraphs are better merged with their neighbors: a one-line chunk without context doesn't help the model much.

Answering with a cheap model

The model receives the retrieved chunks tagged with their source, plus the question. The instruction forbids answering from memory.

import os

from openai import OpenAI

client = OpenAI(base_url="https://api.zukko.pro/v1", api_key=os.environ["ZUKKO_API_KEY"])

SYSTEM = (
    "You answer questions about the company knowledge base. "
    "Use only the fragments in the user message. "
    "After each statement, cite the source in square brackets. "
    "If the fragments don't contain the answer, say so."
)

Why the 1M context matters: deepseek-v4.1-flash and gemini-3.8-flash have a 1M-token context, so even 50 chunks of 500 tokens (25,000 tokens) take up a small part of the window. GLM 5.3 and GLM 5.3 Flash have 128K — enough for this setup too, but with less headroom. More on the client is in the Python docs↗.

If the base is small, say 30–50 thousand tokens, you can skip search entirely and send all of it. But as the base grows this gets expensive quickly — see the numbers below.

What RAG on a cheap model costs

Per-query assumptions: 20 chunks of 500 tokens = 10,000 tokens, plus the instruction and the question — about 350. That's 10,350 input tokens. The answer is 400 output tokens.

Formula: cost = 10,350 × input price / 1,000,000 + 400 × output price / 1,000,000.

Checking the first row: 10,350 × 0.013 / 1,000,000 = $0.00013455; 400 × 0.038 / 1,000,000 = $0.0000152; the sum is $0.00014975, or $1.4975 per 10,000 queries.

Now the "whole base in context" option for a 600,000-token base on the same deepseek-v4.1-flash: 600,000 × 0.013 / 1,000,000 = $0.0078 for the input of a single query. With answers, 10,000 queries come to about $78.15 versus $1.50 with search. Prompt caching won't help here: for DeepSeek a cache read costs the same as input. For Claude models in the Anthropic format a cache read is much cheaper than input, but that's a separate setup with its own math.

Current models and prices: GET https://api.zukko.pro/v1/models or the Models page in the dashboard.

When to use a stronger model

First build a test set: 30–50 real questions with expected answers. Run the cheap model and, for each error, find where it happened.

• The needed chunk isn't among the retrieved ones. That's a search problem, and a stronger model won't help. Increase k, add stemming, change how you chunk.
• The chunk was found, but the answer is wrong or "no answer". This is where it makes sense to try a model with a higher index: gpt-6.1-sol (52) or claude-opus-5-5 (58), on the same questions.

A stronger model is also worth considering when the answer requires combining many chunks, resolving contradictions between documents, or computing something from data in different places, and the cost of a mistake is high.

You don't have to move all traffic to it. A working pattern is a cascade with escalation (covered in detail in the article on model cascades↗): the cheap model answers by default, and the question is retried on the strong one if the user clicks "didn't help" or the answer fails a check (for example, it has no source citations at all). Let's do the math from the table for 10% escalations to claude-opus-5-5: the average query = 0.00015 + 0.1 × 0.00494 ≈ $0.00064. Sending everything to Opus 5.5 would be $0.0049 per query.

More on building the pipeline is in the article on a RAG pipeline over your own data↗. Create a key for the experiment under API keys↗.

FAQ

Why not a vector database?

You can add one later. BM25 is a way to get search running without separate infrastructure and see where it fails. If most misses are paraphrases, move to hybrid search: BM25 plus embeddings.

How many chunks should I pass to the model?

Start with 20 and check on your test set how often the needed chunk ends up in the results. Each additional 10 chunks of 500 tokens on deepseek-v4.1-flash add 5,000 × 0.013 / 1,000,000 = $0.000065 per query.

Does this work for Russian?

Yes, with a caveat about word forms: without stemming, "возврат" and "возврата" count as different words. Check on your own questions whether you need normalization.

How do I update the index?

BM25Okapi is rebuilt from the list of chunks. For a small base, the simplest approach is to rebuild the index when the service starts or on a schedule.