LangChain RAG: a working example with a model via a gateway

LangChain RAG in Python solves a typical task: you have your own documents and the model does not know them. Instead of pasting the entire knowledge base into the request, we find a few relevant fragments and send only those. Below we assemble a minimal working example: loading, splitting, index, search and a model answer through an OpenAI-compatible gateway. Along the way we count where tokens go. The code follows the LangChain documentation (the Retrieval and semantic search sections); check package versions against it.

LangChain RAG: what it consists of

The LangChain documentation splits indexing into four steps: load data into Document objects, split into fragments, turn fragments into vectors (embeddings) and store them in a vector store. At query time the question is also turned into a vector, and the nearest fragments are sent to the model as context. Splitting is needed for two reasons: big pieces are harder to search and waste tokens.

One constraint matters right away. The Zukko API reference has no embeddings endpoint (it has chat, Messages, Responses, a model list and others). So we compute vectors locally and use the cloud model through the gateway only for the answer. That way you pay the gateway only for questions, not for indexing.

Installation

pip install langchain langchain-openai langchain-text-splitters langchain-huggingface sentence-transformers
export ZUKKO_API_KEY="fetch_your_key"

Put a few text or Markdown files into a knowledge/ folder.

The index: load, split, vectors

from pathlib import Path

from langchain_core.documents import Document
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

docs = [
    Document(page_content=p.read_text(encoding="utf-8"), metadata={"source": p.name})
    for p in Path("knowledge").glob("*.md")
]

splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
chunks = splitter.split_documents(docs)

embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
)
store = InMemoryVectorStore(embedding=embeddings)
store.add_documents(chunks)
retriever = store.as_retriever(search_kwargs={"k": 4})

What matters here:

• According to the LangChain documentation, RecursiveCharacterTextSplitter is a sensible default: it tries to keep paragraphs, then sentences. chunk_size here is in characters, and chunk_overlap is the overlap between neighboring fragments. Pick the fragment size on your own questions: fragments that are too small lose context, too large ones waste tokens.
• For Russian text the embedding model should be multilingual. We took a multilingual model from the sentence-transformers family as an example; compare several on your own texts.
• InMemoryVectorStore keeps everything in memory and suits an example. For a persistent base pick a store from the LangChain integrations list.
• If you do not want to download an embedding model, there is a lighter option: BM25Retriever.from_documents(chunks, k=4) from langchain_community.retrievers (it needs the rank_bm25 package). It searches by words without morphology, so for Russian text it misses word forms; test quality on your own questions.

The model answer through the gateway

The cloud model plugs in as ChatOpenAI with the gateway address. The base for OpenAI clients ends with /v1:

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-6.1-sol",
    base_url="https://api.zukko.pro/v1",
    api_key=os.environ["ZUKKO_API_KEY"],
    max_tokens=600,
)

SYSTEM = (
    "Answer only from the given context. "
    "If the context has no answer, say 'I don't know'. "
    "At the end list the source numbers in square brackets."

The "answer only from the context" instruction and permission to say "I don't know" lower the risk of invented answers but do not remove it: test with questions that have no answer in the base. A detailed look at a scheme without extra components is in a RAG pipeline by hand↗. Connecting the gateway to LangChain is described here: LangChain and base_url↗.

Where tokens go

You pay for tokens only at the last step. One question consists of:

• the system instruction (tens of tokens);
• the retrieved context: k fragments of chunk_size characters, usually the heaviest part;
• the question itself;
• the model reply (capped by max_tokens).

Say a question uses 1,800 input tokens (about 1,500 of context, the rest instruction and question; measure the exact number on your own texts) and 300 output tokens. Formula: input × input price / 1M + output × output price / 1M. Prices as of October 18, 2026:

For comparison, recall the "put the whole document into the prompt" approach. With a 200,000-token document on gpt-6.1-sol, one question costs 200,000 × 0.27 / 1M = $0.054, and 10,000 questions cost $540. RAG pays off exactly when there are many documents or many questions; for a one-off job on a single file a long context may be simpler. For Claude in the OpenAI format the whole input is billed at the input price.

How to cut spending

1. Reduce k and chunk_size. Every extra fragment is paid for on every question. Start with k=3 and check quality on 20 to 30 real questions.
2. Remove duplicates. Because of overlap, neighboring fragments repeat text; do not insert both in a row if they cover the same thing.
3. Cap the answer. max_tokens and a request to answer briefly cut output tokens, which cost more than input ones.
4. Use a cheap model for simple questions. If the context already contains the answer, a small model is often enough. Test on your data, and the "cheap first, escalate the hard ones" scheme is described in the article on model cascades↗.
5. Do not drag the whole history along. In a chat pass the latest turns, not the entire dialog.
6. Cache. If you have a long repeating prefix (instructions, examples), then for GPT models the gateway sees cache reads in both formats, and that part is billed at the read price. For Claude the cache discount comes only in the Anthropic format. More: cache and key limits↗.

FAQ

Do I need a separate embeddings service for RAG?

The Zukko gateway does not provide an embeddings endpoint, so the example computes them locally. If you already have an embeddings service, connect it through the matching LangChain class.

Which model should I take for the answer?

Go by price, context and strength index: gpt-6.1-sol has 52, claude-sonnet-5-5 has 48, deepseek-v4.1-flash has 39. Whether a cheap model is enough will be shown by a test on your questions.

How do I know search works?

Print the retrieved fragments next to the answer and check whether the answer is in them. If not, the issue is the splitting or the embeddings, not the model.

Can I replace InMemoryVectorStore?

Yes, any LangChain vector store has as_retriever, and the rest of the code does not change.

Create a key in the dashboard↗, give it a daily limit and run the example.