Knowledge base in Open WebUI: RAG setup on your own documents

Open WebUI RAG lets you upload your own documents and ask a model about them without pasting texts into the chat by hand. It suits internal instructions, contracts and reference material. Below: how to create a knowledge base, which settings affect quality, how to pick a model through a gateway, and what one question costs.

The description is based on the Open WebUI documentation (the Knowledge section and the RAG troubleshooting page). The interface and settings change between versions, so verify menu names against your installation.

Open WebUI RAG: how it works

RAG (retrieval augmented generation) works in three steps. The document is split into pieces, a vector (embedding) is built for each piece, and when you ask a question Open WebUI finds the pieces closest in meaning and adds them to the request to the model. The model then answers with that text in front of it. Two different models take part along the way.

• The embedding model turns text into vectors. By default in Open WebUI this is a small local model (the docs mention all-MiniLM-L6-v2), and you can choose a model through Ollama or OpenAI instead.
• The chat model reads the retrieved pieces and answers. This is the one you connect through the gateway.

The gateway documentation does not list an embeddings endpoint, so keep embeddings on the built-in model or connect a separate source. The chat model can be anything: Claude, GPT, DeepSeek and others.

Step 1. Connect a model

In Open WebUI open Settings, Admin, Connections and add an OpenAI-compatible API connection. Enter https://api.zukko.pro/v1 as the address and your fetch_... key as the key. If the connection check shows an error, the Open WebUI docs suggest adding the model identifiers to the list manually: chat will still work. Detailed setup is in Open WebUI: connecting to Zukko↗.

Step 2. Create a knowledge base

According to the Open WebUI docs:

1. Open Workspace, Knowledge and click Create. Give it a name and description.
2. Upload a small text PDF or a text file and wait for processing to finish.
3. In a chat, call the base with # or attach it to a model in Workspace, Models, Edit.
4. Ask a question whose answer is in the file and ask the model to quote the passage.

Always run the first test on a document where you know the answer. That shows right away whether the model sees the content.

Files can be arranged in folders, uploaded as a whole directory and updated. For one-off questions it is simpler to upload a file straight into the chat, and to keep a knowledge base for a set of documents you will use many times.

Step 3. Choose the retrieval mode

Attached documents have two modes, switched by clicking the attached item.

• Focused Retrieval (default). Finds and inserts only the most relevant pieces. Suits large document sets.
• Full Context. Inserts the whole document with every message. Suits short references that are always needed.

One nuance to know in advance. In native function calling mode (enabled by default in current versions), knowledge attached to a model is not inserted automatically: the model must call the knowledge search tools itself. If it does not, the docs suggest adding an instruction to the system prompt to use knowledge search, choosing a model that calls tools reliably, or switching the attachment to Full Context. Each tool call is an extra request and extra tokens.

Step 4. Tune splitting and search

The parameters are in Settings, Admin, Documents. The Open WebUI docs give recommendations for different scenarios. For cloud models with a large context:

For local models with a small window (up to 8K tokens) the values are more modest: chunk 1000, overlap 100, Top K from 3 to 5. The logic is simple: the more pieces you insert, the more input tokens you spend and the bigger the model window you need.

Also check two things:

• Text extraction. If the model "does not see" the document, extraction is more often at fault than search. The docs advise a robust engine (Apache Tika, Docling) and checking the extracted text in the preview. Empty or truncated text must be fixed before anything else.
• Changing the embedding model. After a change you must reindex the base, otherwise old vectors are incompatible with new ones. Changing chunk size does not require reindexing, but it is recommended for consistency.

What a question over documents costs

Suppose a handbook takes 60,000 tokens and 30 questions are asked about it. Formula: input tokens × input price. Output (the answer) is billed separately and is not included here. Prices as of October 18, 2026.

Scenario A, Full Context: 60,000 × 30 = 1,800,000 input tokens. Scenario B, Focused Retrieval: 5 pieces of 1000 tokens plus the question and context, about 5,500 tokens per question, so 5,500 × 30 = 165,000 tokens.

Two notes. First: Open WebUI talks to the gateway in the OpenAI format. For GPT models the gateway sees cache reads, so with repeated Full Context part of the input may be billed at the cache read price (for gpt-6.1-sol, $0.013 instead of $0.27 per 1M), and the real bill will be lower than the table. For Claude in this format the whole input is billed at the input price, exactly as in the table. Second: the figures show an order of magnitude, not a forecast, because the real context size depends on your conversation. Which model answers your documents best is something to check on your own questions. More about caching is in the limits and cache docs↗.

Improving answer quality

• Phrase questions to include terms from the document: search works on meaning, but rare terms help.
• Do not dump everything into one base. Attach only the needed bases to a model so search does not get distracted.
• Ask for a quote and source. If the answer disagrees with the quote, look for the cause in the extracted text or in the splitting parameters.
• For small documents try Full Context: with a large model window it is often more reliable than piece-based search, but it costs more per question, as the table shows.
• If you run a model on Ollama, check the context window: according to the docs, on GPUs under 24 GiB it defaults to 4096 tokens, and the retrieved pieces simply do not fit.

About local servers such as LM Studio: Open WebUI connects to any server with an OpenAI-compatible API, so RAG works with them too, but check the tool's own documentation for connection details. The general scheme of searching a knowledge base with a cheap model is in RAG on a cheap model without a vector DB↗.

FAQ

Do I need a separate vector database?

No. Open WebUI stores vectors itself; the docs list supported stores, including ChromaDB and PGVector. You can attach your own vector database as an external knowledge source, but it is not required to start.

Which model should I choose for answers over documents?

Any model with a sufficient window and reliable tool calling will do. By input price the cheapest models, for example deepseek-v4.1-flash, are the most economical, but check quality on your own documents and questions.

Why does the model say "not found in the documents" when the answer is there?

Check that file processing finished, that the text was extracted fully, that the base is attached to the model, and that the model calls the search tools. If not, switch the attachment to Full Context as a test.

How much does uploading documents cost?

The upload itself and building embeddings with the built-in model do not use the gateway balance. Only requests to the chat model are billed.

To try it, create a key in your account↗ and connect it to Open WebUI. Request parameters for the chat model are described in the chat completions reference↗.