LLM model cascade: cheap model first, strong model on escalation

An LLM model cascade is a technique for streams of similar requests: ticket classification, field extraction from documents, moderation. A cheap model handles most requests, and only those where the answer fails validation or the model is unsure go to a strong model. Below: how to build it in Python with a single Zukko key, how to pick the threshold, and how to compute cost with a formula rather than by eye.

How an LLM model cascade works

Three parts:

1. First attempt: a cheap model. For example deepseek-v4.1-flash ($0.013 per 1M input, $0.038 per 1M output).
2. Answer check. Code decides whether to accept or escalate. Escalation signals:
   • the answer doesn't parse as JSON or doesn't match the schema;
   • a value is outside the allowed set (a label not on the list);
   • a business rule is broken (line items don't add up to the total, a date in the future);
   • self-reported confidence is below the threshold.
3. Second attempt: a strong model. For example claude-opus-5-5 ($0.4 / $2 per 1M). If its answer also fails the check, the request goes to manual review.

To be honest about self-reported confidence: the number a model writes about itself is a weak signal and may be poorly calibrated. So don't pick the threshold at random; tune it on a labelled sample (shown below). Formal checks — schema, allowed values, business rules — are more reliable and should come first.

Both models are available with one key and one address: Zukko routes the request to the right provider, no client changes needed.

Python code

Example: classifying support tickets into four topics.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.zukko.pro/v1", api_key="fetch_...")

CHEAP = "deepseek-v4.1-flash"
STRONG = "claude-opus-5-5"
LABELS = {"billing", "bug", "feature_request", "other"}

SYSTEM = (
    "Classify the customer's message by topic. Reply with JSON only, no comments: "
    '{"label": "billing" | "bug" | "feature_request" | "other", '
    '"confidence": a number from 0 to 1}'
)

What matters here:

• Both models are called through the OpenAI SDK with base URL https://api.zukko.pro/v1; the gateway converts the request for Claude.
• The strong model gets the same input as the cheap one. You can also pass it the first model's answer, but then input grows; account for that in the estimate.
• For Claude in the OpenAI format all input is billed at the input price; cache isn't counted. For short requests like these it doesn't matter. If the strong model has a long shared system prompt, call it through the Anthropic SDK (Anthropic(base_url="https://api.zukko.pro", ...)): in the Anthropic format cache reads are billed at their own, lower price.

More on the Python client in the docs↗.

Picking the threshold

The confidence threshold determines the escalation share and the error rate among the cheap model's accepted answers. Tune it on 200–500 tickets with known correct labels:

def calibrate(samples: list[tuple[str, str]], thresholds=(0.5, 0.6, 0.7, 0.8, 0.9)):
    answers = [(ask(CHEAP, text), gold) for text, gold in samples]
    for t in thresholds:
        accepted = [(a, gold) for a, gold in answers if is_valid(a) and a["confidence"] >= t]
        r = 1 - len(accepted) / len(answers)
        errors = sum(a["label"] != gold for a, gold in accepted)
        err_rate = errors / len(accepted) if accepted else 0.0
        print(f"threshold {t}: escalations {r:.0%}, errors among accepted {err_rate:.1%}")

Pick the lowest threshold at which the error rate among accepted answers is acceptable to you. The resulting escalation share r is exactly the number you need for the cost estimate. Re-run calibration on fresh data after a couple of weeks: the request stream changes.

What a cascade costs: the formula

Notation:

• C₁ — average cost of a request to the cheap model;
• C₂ — average cost of a request to the strong model;
• r — share of requests escalated (0 to 1).

An escalated request is paid for twice: the cheap attempt, then the strong one. So:

Example on a hypothetical stream

100,000 tickets, each averaging 800 input tokens (system prompt plus text) and 150 output tokens.

• C₁ (deepseek-v4.1-flash) = 800 × $0.013 / 1,000,000 + 150 × $0.038 / 1,000,000 = $0.0000104 + $0.0000057 = $0.0000161
• C₂ (claude-opus-5-5) = 800 × $0.4 / 1,000,000 + 150 × $2 / 1,000,000 = $0.00032 + $0.0003 = $0.00062

For 100,000 requests: the cheap model costs $1.61, the strong one $62.00. Cascade cost = $1.61 + r × $62.00:

Break-even: r = 1 − 0.0000161 / 0.00062 ≈ 0.974. As long as fewer than 97.4% of requests escalate, the cascade costs less than sending everything to Opus 5.5. The r values in the table are examples to plug in, not a promise. You get your own share from calibration.

The flip side: sending everything to the cheap model costs $1.61, but then nobody catches the bad answers. A cascade is what you pay so that hard cases are handled by the strong model.

Three tiers

You can put a mid-tier model between the cheap and the strong one, for example claude-sonnet-5. The formula extends to:

where r₁ is the share reaching the second tier and r₂ the share reaching the third. A third tier only makes sense if calibration shows the middle model resolves a noticeable part of the escalations.

What to check before launch

• Quality of accepted answers. Nobody re-checks cheap-model answers that pass the threshold. Once a week take a random sample of 50–100 accepted answers and verify them by hand or against the strong model.
• Actual cost. Check C₁ and C₂ against real charges in the dashboard under "Usage" or via the spending history API: real answer lengths may differ from the estimate.
• Key restrictions. For a pipeline, a separate key with allowed models (just the cascade's two) and a daily dollar limit is convenient. You can create one in the dashboard↗.
• Model availability. Models and prices change: the current list is at GET https://api.zukko.pro/v1/models or on the "Models" page in the dashboard.

A similar but different pattern is routing, where the model is chosen up front by request type rather than after checking the answer. See the article on multi-model routing strategy↗.

FAQ

How is a cascade different from routing?

A router picks the model before the request, based on its features. A cascade tries the cheap model first and decides from its answer. You can combine them.

Can I trust the confidence field?

Only after calibrating on labelled data. Formal checks — answer schema, allowed values, business rules — are more reliable and should come first.

Which cheap model should I use?

Among the least expensive in the catalog are deepseek-v4.1-flash and gemini-3.8-flash. Which suits your task better will show in the same calibration: compare escalation and error rates for both. Prices and calculations for all the low-cost models are in the article on the cheapest models for bulk tasks↗.

How much will I save?

That depends on the escalation share r. Compute C₁ + r × C₂ with your own C₁, C₂ and the r measured during calibration.