LlamaIndex: indexing data with Claude and GPT

LlamaIndex relies on two different models: one turns text into embeddings while building the index, the other synthesizes the answer at query time. The gateway covers the second part completely — Claude, GPT, and GLM are all available as Settings.llm. There is no embeddings endpoint in the API, though, and that is worth knowing upfront rather than after an hour spent chasing an IndexError.

The split between Settings.llm and Settings.embed_model is a LlamaIndex design choice, not something specific to any one provider, but it's exactly where people trip up the first time they wire up a third-party gateway: the library defaults to pulling both services from the same OpenAI-like client, and if embed_model isn't set explicitly, it quietly reaches for embeddings straight from OpenAI using its own environment variable for a key.

Query LLM through the OpenAI-compatible class

from llama_index.llms.openai_like import OpenAILike
from llama_index.core import Settings

Settings.llm = OpenAILike(
    model="gpt-5.6-sol",
    api_base="https://api.zukko.pro/v1",
    api_key="fetch_your_key",
    is_chat_model=True,
    context_window=1000000,
)

The OpenAI class from llama_index.llms.openai checks the model name against its own table of known OpenAI models and trips over unfamiliar names, so for gateway models use OpenAILike (the llama-index-llms-openai-like package). is_chat_model=True sends requests to /chat/completions, and context_window sets the context limit explicitly (for glm-5.3 and glm-5.3-flash, use 128000). api_base here is the full path including /v1, because the class passes it straight into the openai.OpenAI(base_url=...) constructor, which then appends /chat/completions on its own. Without /v1 in api_base, the request lands on a path that doesn't exist and you get a 404.

Query LLM through the Anthropic class

from llama_index.llms.anthropic import Anthropic
from llama_index.core import Settings

Settings.llm = Anthropic(
    model="claude-opus-5-5",
    base_url="https://api.zukko.pro",
    api_key="fetch_your_key",
    context_window=1_000_000,
)

The Anthropic class names the parameter base_url, not api_base, and the address has no /v1 — the Anthropic client appends /v1/messages itself. Set context_window explicitly: LlamaIndex keeps an internal table of known models and their context limits, and for a model not yet in that table it falls back to a conservative default. That can silently truncate the conversation history as if the model only held a few thousand tokens, when the gateway's actual limit is 1,000,000.

Why embeddings stay separate

The gateway's endpoints — /v1/messages, /v1/chat/completions, /v1/responses, /v1/models, /v1/models/info, /v1/balance, /v1/usage, /v1/web-search — don't include /v1/embeddings. That means Settings.embed_model needs to stay on a separate embeddings provider, local or otherwise:

from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

In this setup the gateway only handles the LLM that synthesizes the final answer (Claude and GPT models read images and take up to a 1,000,000-token context; GLM has 128K and no vision) — building the vector index itself happens outside it.

Building the index and querying it

Order matters here: Settings.llm and Settings.embed_model are module-level globals, and setting them after an index has already started building on another thread can leave some documents indexed with default settings. For batch indexing scripts, it's safer to set both values at the very top of the file, before the first SimpleDirectoryReader import. This shows up most often in Jupyter notebooks, where cells get rerun in whatever order during experimentation.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)  # embeddings via Settings.embed_model

query_engine = index.as_query_engine(llm=Settings.llm)
response = query_engine.query("What does the documentation say about API limits?")
print(response)

Web search on top of your own index

Catalogue models don't browse the web on their own — when a question falls outside the indexed documents, that's a separate POST /v1/web-search, wrapped in a FunctionTool:

from llama_index.core.tools import FunctionTool
from llama_index.core.agent import ReActAgent
import requests

def web_search(query: str) -> str:
    """Looks up current information online when the index doesn't have it."""
    resp = requests.post(
        "https://api.zukko.pro/v1/web-search",
        headers={"Authorization": "Bearer fetch_your_key"},
        json={"query": query},
    )
    return resp.json()

search_tool = FunctionTool.from_defaults(fn=web_search)
query_engine_tool = FunctionTool.from_defaults(
    fn=lambda q: str(query_engine.query(q)),
    name="local_docs",
    description="Answers from the local document base",
)

agent = ReActAgent.from_tools([query_engine_tool, search_tool], llm=Settings.llm)
response = agent.chat("Compare our internal policy against the vendor's latest published changes")

Web search and answer synthesis are billed separately: the search itself is charged as its own call, and the synthesis follows the usual chat-model rules.

Parallel requests and the allowed-model list

VectorStoreIndex.from_documents can synthesize answers in parallel when num_workers is set on an IngestionPipeline, and the same applies to queries running through a multi-worker query_engine. If a key is restricted to an allowlist of models (set on the key's page in the dashboard) and somewhere in the pipeline a different model slips in by mistake — say, the default gpt-3.5-turbo from a README example instead of the one you actually configured — some parallel requests fail with 403 model_not_allowed_for_key while others go through, because Settings.llm gets applied correctly for some calls but not for wrapper classes like SubQuestionQueryEngine, unless the model is passed explicitly at every level.

Model not found

Older LlamaIndex examples often reference text-davinci-003 or gpt-4:

{"error": {"message": "The model 'text-davinci-003' does not exist", "code": "model_not_found"}}

Check the current list without a key:

curl https://api.zukko.pro/v1/models

429 during bulk indexing

as_query_engine with response_mode="tree_summarize" can fire off dozens of requests in a row against one large document corpus — closer to a load test than a normal chat session. If a key has a daily or monthly spend limit configured in the dashboard, a run like that can hit key_daily_limit_reached mid-index. The daily limit resets at 00:00 UTC. Pauses between batches won't help here: the key limit is counted in dollars, and the gateway has no rate limit of its own (a 429 without the key-limit codes comes from the provider). For batch jobs, either raise or lift the limit on the working key, or split the indexing into chunks that fit the daily budget.

Check before running the pipeline

curl https://api.zukko.pro/v1/chat/completions \
  -H "Authorization: Bearer fetch_your_key" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.6-sol","messages":[{"role":"user","content":"ping"}]}'

If that answers but query_engine.query() still fails, check whether Settings.embed_model is set at all: without it, LlamaIndex defaults to calling OpenAI directly for embeddings and fails with an authorization error that looks like a gateway key problem at first glance, even though the gateway key has nothing to do with it. Balance and spend are visible via GET /v1/balance and GET /v1/usage, and keys are issued in the dashboard at https://zukko.pro/app/api-keys↗. If indexing runs in CI and fails there for no obvious reason while working fine locally, check that the key variable is actually passed into the runner's environment, not just sitting in a .env file CI doesn't read by default.