LangChain: pointing base_url at the gateway instead of OpenAI

LangChain defaults to hitting api.openai.com or api.anthropic.com, and both clients — ChatOpenAI and ChatAnthropic — accept a base_url parameter that fully overrides that. The key is the same fetch_... in both cases.

ChatOpenAI: GPT, GLM

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-5.6-sol",
    api_key="fetch_your_key",
    base_url="https://api.zukko.pro/v1",
)

response = llm.invoke("Explain the difference between base_url and a client's base URL")
print(response.content)

The same class, unchanged, covers the rest of the OpenAI-compatible catalogue — only model changes: gpt-5.6-luna, glm-5.3, glm-5.3-flash (current list: GET https://api.zukko.pro/v1/models).

ChatAnthropic: Claude, no /v1 in base_url

from langchain_anthropic import ChatAnthropic

llm = ChatAnthropic(
    model="claude-sonnet-5",
    api_key="fetch_your_key",
    base_url="https://api.zukko.pro",
)

response = llm.invoke("Compare claude-opus-5-5 and claude-sonnet-5 for different tasks")
print(response.content)

The difference from the previous example isn't just the class — base_url here has no /v1, because ChatAnthropic appends the path to /v1/messages itself and adds the anthropic-version header. Carry over the habit from OpenAI code and add /v1 here too, and every call flies off to a path that doesn't exist and returns 404, while the key itself keeps getting accepted fine — this is the most common mistake when porting code between the two classes.

Available Claude models: claude-opus-5-5 and claude-sonnet-5; current list: GET https://api.zukko.pro/v1/models.

Streaming works as usual

Streaming needs nothing gateway-specific — it works through the standard .stream() on both classes:

for chunk in llm.stream("Compute the factorial of 10 and show the steps"):
    print(chunk.content, end="", flush=True)

Counting tokens without an actual call

To estimate a Claude prompt's length ahead of time without paying for the call itself, it's simpler to hit /v1/messages/count_tokens directly via requests, bypassing LangChain — this endpoint is free and doesn't trigger generation:

import requests

resp = requests.post(
    "https://api.zukko.pro/v1/messages/count_tokens",
    headers={
        "Authorization": "Bearer fetch_your_key",
        "anthropic-version": "2023-06-01",
        "Content-Type": "application/json",
    },
    json={
        "model": "claude-sonnet-5",
        "messages": [{"role": "user", "content": "text to measure"}],
    },
)
print(resp.json())

Handling errors in chains

LangChain surfaces API errors as exceptions from the openai or anthropic SDK — wrap the call in try/except and inspect the code inside the response body, not just the HTTP status:

from openai import APIStatusError

try:
    llm.invoke("...")
except APIStatusError as e:
    body = e.response.json()
    code = body.get("error", {}).get("code")
    if code == "insufficient_balance":
        # balance is at zero, the request never reached a model, nothing was spent
        ...
    elif code == "model_disabled":
        # 503: the model is switched off — move to a model from GET /v1/models
        ...

Codes worth knowing: 401 — wrong or truncated key; 402/insufficient_balance — empty balance, checked before the upstream call; 404/model_not_found — a typo in the id, check against GET /v1/models; 429/key_daily_limit_reached or key_monthly_limit_reached — hit that key's specific cap from the dashboard; 403/model_not_allowed_for_key — the model is blocked by that key's policy; 503/model_disabled — the model is switched off, replace it with one from the current list; 503/provider_unavailable — the model is temporarily down, worth a delayed retry.

Idempotency for retries in chains

If you wrap the call in your own retry logic (via tenacity, say), pass an Idempotency-Key header on the specific call rather than in the constructor — for example llm.invoke(prompt, extra_headers={"Idempotency-Key": key}) on ChatOpenAI. Generate the key once per logical request, before the retry loop, and keep the same value across all its attempts — repeating it within 24 hours returns the same response and doesn't charge twice. Don't put Idempotency-Key in default_headers: one key would then go out with every request from that client, and the very next, different request gets a 422 idempotency_conflict.

Keys and limits

The key is created once in the dashboard at https://zukko.pro/app/api-keys and shown there exactly once — store it in environment variables or a secrets manager, not in repository code. The key's page lets you set a daily or monthly dollar spending cap and a list of allowed models — useful for a production service key kept separate from the one used for local development.

Caching for repeated calls

If a chain has a step sending the same request over the same kind of data (classification, entity extraction against a fixed schema), turn on caching with an explicit X-Zukko-Cache header (temperature=0 in the constructor also enables the cache, but only on models that accept that parameter; ChatAnthropic with claude-sonnet-5 or claude-opus-5-5 may get a 400 on a temperature other than 1, so don't pass it to Claude models):

llm = ChatOpenAI(
    model="gpt-5.6-sol",
    api_key="fetch_your_key",
    base_url="https://api.zukko.pro/v1",
    default_headers={"X-Zukko-Cache": "true"},
)

A repeat request with a fully identical body from the same user comes back instantly and free, with an X-Zukko-Cache: hit header in the response; "almost identical" requests or a shared prefix don't hit the cache, and cache hits aren't written to /v1/usage. For chat, the cache lasts up to 7 days — a cost cut for chains that re-send exactly the same request over the same data.

Your own tools through LangChain

Tool calling on ChatOpenAI and ChatAnthropic works without modification — bind_tools with a list of Pydantic schemas or plain functions goes into the request body the same way it does in the original SDKs, because /v1/chat/completions and /v1/messages accept the tools field in the same format:

from langchain_core.tools import tool

@tool
def get_weather(city: str) -> str:
    """Get current weather for a city"""
    return f"It's 18C in {city} right now"

llm_with_tools = llm.bind_tools([get_weather])
response = llm_with_tools.invoke("What's the weather in Moscow?")

Monitoring spend from code

If you're building a service on LangChain and want to log the cost of every call rather than just check the balance once a day, read the X-Zukko-Cost-Micros and X-Fetch-Balance-Cents (the balance there is in microdollars, 1,000,000 = $1) headers from the raw HTTP response — they're only present on regular calls, not on .stream(); with ChatOpenAI this is reachable through .invoke(...).response_metadata in some setups, but it's more reliable to get at the headers directly via with_raw_response on the openai client, or to query /v1/usage separately for history (it covers the whole account, all keys together):

usage = requests.get(
    "https://api.zukko.pro/v1/usage",
    headers={"Authorization": "Bearer fetch_your_key"},
    params={"limit": 50},
).json()