A minimal RAG pipeline on your own data

RAG usually gets explained as "embeddings → vector database → top-k chunks → generation," and half the write-ups on it start with picking a vector database. In practice, for a small or mid-sized document corpus, that whole chain is often overkill — and the zukko gateway doesn't provide embeddings at all, only text generation and web search. Here's what actually gets built with that, and when vector search is genuinely needed.

When you don't need a vector database at all

The Claude and GPT models in the catalog carry a 1,000,000-token context window (glm-5.3 and glm-5.3-flash have 128K). That's roughly 700-800 thousand words — a mid-sized company's whole knowledge base, a set of internal docs, or a hundred contracts fit into a single request with no retrieval step at all. If your corpus fits that budget, the minimal RAG setup isn't a pipeline — it's one system prompt:

curl https://api.zukko.pro/v1/messages \
  -H "Authorization: Bearer fetch_your_key" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-opus-5-5",
    "max_tokens": 1500,
    "system": "'"$(cat knowledge_base.md | jq -Rs .)"'",
    "messages": [{"role": "user", "content": "How do I process an equipment return under a lease contract?"}]
  }'

Before reaching for a vector database, it's worth measuring the actual data volume with the free token count — the corpus is often well below the context limit:

curl https://api.zukko.pro/v1/messages/count_tokens \
  -H "Authorization: Bearer fetch_your_key" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model": "claude-opus-5-5", "messages": [{"role": "user", "content": "'"$(cat knowledge_base.md | jq -Rs .)"'"}]}'

Keep the cost of this setup in mind: the whole knowledge base is billed as input tokens on every request. Vendors charge ~10% of the input price for prompt-cache reads, while Zukko bills cached tokens at the regular price — for scenarios with a large repeated context, the real price gap is smaller than a head-to-head price-list comparison suggests.

When you actually need retrieval

If the corpus grows faster than the context window — hundreds of thousands of documents, years of logs or tickets — stuffing everything into the prompt stops working: request cost climbs and the model loses focus on the relevant fragment among the noise. That's where classic retrieval comes in: embeddings are computed by a separate embeddings service of your choice (the zukko gateway doesn't provide them — it's a purely generative API), results live in whatever vector database you pick, and the gateway is only used at the answer-generation step over the retrieved chunks:

import requests

# retrieval - your own vector search, outside zukko
relevant_chunks = vector_search(query, top_k=8)
context = "\n\n---\n\n".join(chunk.text for chunk in relevant_chunks)

resp = requests.post(
    "https://api.zukko.pro/v1/messages",
    headers={
        "Authorization": "Bearer fetch_your_key",
        "anthropic-version": "2023-06-01",
    },
    json={
        "model": "claude-sonnet-5",
        "max_tokens": 1000,
        "system": f"Answer only from these fragments:\n\n{context}",
        "messages": [{"role": "user", "content": query}],
    },
)

The split is clean: retrieval is your infrastructure, generation is a call to /v1/messages or /v1/chat/completions.

A hybrid setup: tool calling instead of a rigid pipeline

Instead of always searching first and generating second, you can let the model decide whether retrieval is needed at all, via /v1/responses with a search tool description:

curl https://api.zukko.pro/v1/responses \
  -H "Authorization: Bearer fetch_your_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.6-sol",
    "input": "How many open auto-renewal contracts do we have through 2027?",
    "tools": [{
      "type": "function",
      "name": "search_documents",
      "description": "Searches the internal document base for relevant fragments",
      "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}
    }]
  }'

The model decides on its own whether to call search_documents — your application-side retrieval function — or answer directly if the question doesn't need the document base at all. /v1/responses accepts a plain string, a message array, and a tool list, which suits this kind of agentic setup well.

Web search as a source for external context

If part of the RAG system needs to answer questions about the outside world, not just internal documents — say, "compare our terms to market rates" — models without built-in search have a dedicated call for that:

curl https://api.zukko.pro/v1/web-search \
  -H "Authorization: Bearer fetch_your_key" \
  -H "Content-Type: application/json" \
  -d '{"query": "average commercial lease rates 2026"}'

The result gets folded into the next model call the same way internal chunks do — from the generation step's point of view, there's no difference between "internal retrieval" and "web search"; both just supply text for a system or user message.

Caching for repeated questions

In a RAG setup for support or a knowledge base, the same questions get asked over and over. If the answer should be deterministic for the same set of retrieved chunks, it's worth setting the X-Zukko-Cache: true header (temperature: 0 also enables the cache, but only on models that accept that parameter; Claude models — claude-sonnet-5, claude-opus-5-5 — may reject a temperature other than 1 with a 400, so don't pass it to them) — a repeated identical request from the same user comes back from cache instantly and for free (X-Zukko-Cache: hit in the response), with chat cache entries living up to 7 days. Note that the cache only works on a fully identical request body — the question wording, the chunk set and the parameters all have to match; "almost identical" questions don't hit it, and cache hits aren't written to /v1/usage. The saving shows up where a popular question arrives with the same wording and retrieval returns the same chunk set day after day.

Cost estimation and model choice per stage

For the answer-generation step over retrieved chunks, people usually try claude-sonnet-5 or gpt-5.6-sol — the task doesn't need extended reasoning once the chunks are already relevant; for simple quote lookups, the cheaper gpt-5.6-luna or glm-5.3 ($0.10/$0.10) are worth testing too. claude-opus-5-5 is worth trying where the answer requires cross-referencing several documents and an explicit conclusion — say, "is there a contradiction between clause 4.2 of contract A and clause 7.1 of contract B" — and on Zukko it costs the same as Sonnet 5 and Sol ($0.30/$0.30), so test on your own tasks which model does better.

Errors specific to a RAG pipeline

404 model_not_found in tests often shows up when an agent's config still has an old model id left over from a migration (a model switched off in the catalog answers 503 model_disabled instead) — check against the current list via GET /v1/models, which is public and needs no key. 422 idempotency_conflict shows up when a pipeline's retry logic reuses one Idempotency-Key for requests carrying a different set of retrieved chunks — the key needs to depend not just on the user's question but on the version of the context that got substituted into it. A response cutting off mid-generation almost always means insufficient max_tokens, not a retrieval problem — if the model needs to quote several long fragments, budget more output tokens than for a short answer to a direct question.

Bottom line: for a small corpus, RAG is a system prompt and a large context window, no extra service required. For a growing corpus, it's your own retrieval plus generation through the API, where model choice depends on whether the task is restating what was found or cross-referencing multiple sources.