AnythingLLM with Zukko: document chat on your own OpenAI-compatible API

AnythingLLM is an app for chatting with your documents: you upload files to a workspace, and the model answers based on the passages it finds. You connect your own OpenAI-compatible API in AnythingLLM through the Generic OpenAI provider, and that's how Zukko is added: one fetch_... key gives you Claude, GPT and GLM models. Below is the setup based on AnythingLLM's source code and example config, the settings that determine answer cost, and troubleshooting.

Where the model is configured

The system model is set in Settings → AI Providers → LLM (the page is called LLM Preference). All workspaces use it by default. In a workspace's own settings you can pick a different provider and model (Workspace LLM Provider and Workspace Chat model); if the field is empty, the system model is used.

Connecting your own OpenAI-compatible API in AnythingLLM

1. Open Settings → AI Providers → LLM.
2. In the LLM Provider list, choose Generic OpenAI.
3. Fill in the fields (table below) and save.
4. Open a workspace and ask a short question to test the connection.

The Base URL hint shows an address without a version, but the service needs the full base address: for Zukko it's https://api.zukko.pro/v1. If AnythingLLM manages to load the model list, Selected Model becomes a dropdown; otherwise it's a text field where you type the ID.

Model context window

AnythingLLM uses this value to decide how much history and how many document passages fit into a request. The real context of Zukko models is 1,000,000 tokens for claude-opus-5-5, claude-sonnet-5, gpt-5.6-sol and gpt-5.6-luna, and 128,000 for glm-5.3 and glm-5.3-flash. There's no point setting more than the real value, since the model won't accept such a request. Setting less is fine and even useful, because it caps the maximum size, and therefore the cost, of a single request.

Configuring with environment variables

In Docker the same settings are set with variables (names taken from AnythingLLM's example .env):

LLM_PROVIDER='generic-openai'
GENERIC_OPEN_AI_BASE_PATH='https://api.zukko.pro/v1'
GENERIC_OPEN_AI_API_KEY='fetch_your_key'
GENERIC_OPEN_AI_MODEL_PREF='claude-sonnet-5'
GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=128000
GENERIC_OPEN_AI_MAX_TOKENS=2048

For the current list of model IDs, call GET https://api.zukko.pro/v1/models (see the models documentation↗) or open the "Models" page in the dashboard.

Embeddings are a separate setting

The chat model and the embedding model are configured separately in AnythingLLM. Embeddings are used to index documents and find relevant passages. Zukko currently offers only chat models, so choose the embedder separately: AnythingLLM's built-in one or another provider. Zukko handles answer generation.

What drives the cost of an answer

Workspace settings have three parameters that directly affect request size:

• Chat History — how many previous messages go into short-term memory (AnythingLLM recommends 20);
• Max Context Snippets — how many document passages are sent to the model (recommended: 4);
• Chat mode — Chat answers using both documents and the model's general knowledge; Query answers only if something relevant is found in the documents; Agent uses tools on its own if the model and provider support tool calling.

A sample calculation for one question: a 300-token system prompt, 4 passages of about 500 tokens each, roughly 3,000 tokens of history, and a 100-token question — 5,400 input tokens plus 500 for the answer, 5,900 in total:

claude-opus-5-5 and gpt-5.6-sol cost the same as claude-sonnet-5, and glm-5.3 costs the same as gpt-5.6-luna.

If you lower Chat History so that history takes about 750 tokens instead of 3,000, input drops to 3,150 tokens, and a question on claude-sonnet-5 costs (3,150 + 500) × 0.3 / 1,000,000 ≈ $0.0011.

Check for yourself which model answers best over your documents: create two workspaces with the same documents and different models, ask 15–20 real questions in each, and compare the answers and spending in the dashboard's "Usage" section.

Common errors

401

The key was copied partially, with a space, or was deleted in the dashboard. The key starts with fetch_.

404 or an empty model list

Base URL is missing /v1, or has an extra /chat/completions. It must be exactly https://api.zukko.pro/v1.

413 or a "request too long" error

Model context window is set higher than the model's real context, or too many passages go into the request. Check the window value and lower Max Context Snippets or Chat History.

The agent doesn't call tools

Agent mode uses native tool calling only if the model and provider support it. AnythingLLM's example .env includes PROVIDER_DISABLE_NATIVE_TOOL_CALLING to switch to a fallback mechanism; see the AnythingLLM docs for details.

400 on Claude models

claude-sonnet-5 and claude-opus-5-5 may reject a request with a 400 error if temperature is anything other than 1 or top_p is below 0.99. Workspace settings have an LLM Temperature field: for Claude, leave it empty — then AnythingLLM doesn't send temperature and the provider default applies.

402, 403, 429

402 means the balance ran out. 403 model_not_allowed_for_key means the model isn't on the key's allowed list. 429 means too many requests; wait and retry.

FAQ

Can different workspaces use different models?

Yes. The system model is set in LLM Preference, and a workspace can override it. For example, glm-5.3-flash for an internal policy handbook and claude-sonnet-5 for contract analysis.

What happens when the balance runs out?

The API returns a 402 error and AnythingLLM can't get an answer. You can top up in the dashboard: SBP, Russian bank cards or crypto, from $1.

How do I cap spending?

Set daily and monthly limits on the key in the dashboard, and lower Chat History and Max Context Snippets. How limits work is described in the key limits documentation↗.

Where do I start?

Create a key in the Zukko dashboard↗, choose the Generic OpenAI provider in AnythingLLM, and upload a couple of documents to a new workspace.