How do I detect when to escalate?

A confidence score if the task produces one, an explicit instruction telling the cheap model to say when it is unsure, or a validation check on its output. All three cost less than the flagship call they avoid.

Is routing worth the complexity?

At low volume, no. Above a few thousand dollars a month it is usually the single largest saving available, and a rule on an obvious signal is an afternoon of work rather than a project.

Comparison

Large models vs small models

The price gap between tiers is often twenty to one while the capability gap on ordinary tasks is small. Why routing beats picking, and where a flagship is genuinely required.

The short answer

Do not choose — route. Most requests in most products are not hard, and a model costing a twentieth of the flagship handles classification, extraction, routing and formatting well. Send the easy majority to the cheap model, escalate on low confidence, and reserve the flagship for what genuinely needs judgement. That commonly cuts a bill by half or more without a quality loss anyone notices.

At a glance

Large model compared with Small model
	Large model	Small model
Price	Often 20× the small tier	A fraction
Latency	Higher per token	Lower
Simple classification	Overkill	Fine
Hard reasoning	Substantially better	Struggles
Long-context recall	Better	Weaker
Instruction following	More reliable	Needs a tighter prompt

When to choose which

Choose Large model when

Multi-step reasoningProblems where the answer requires holding several constraints at once, or where a wrong intermediate step invalidates the result.
The output is expensive to get wrongAnything customer-facing without review, anything touching money, anything where a plausible-but-wrong answer causes real harm.
Long context that must be used wellRecall over a large context is where the capability gap is widest.

Choose Small model when

Classification, extraction, routingStructured tasks with a small answer space. A small model with a clear prompt and two examples handles these reliably.
High volume, low stakesWhere the price difference multiplied by the request count dwarfs the occasional error, and errors are recoverable.
Latency mattersSmaller models generate faster. On a user-facing path that difference is felt.
A first pass before escalationThe cheap model answers, and low confidence escalates. Escalated requests are paid twice and it is still a large net win.

What it costs either way

The price gap between the cheapest and dearest text model tracked here spans more than two orders of magnitude. No capability difference is that large on ordinary tasks.

Routing pays even with a meaningful escalation rate. At a twenty-to-one price gap and an escalation rate in single digits, sending seventy percent of traffic to the small tier cuts the bill substantially.

The break-even escalation rate is computable: above it, routing stops paying and you should send everything to the flagship. It is usually far higher than real escalation rates.

Latency improves at the same time, which is unusual — most cost optimisations trade against something.

The mistake people make

Picking one model for the whole product

It is the simplest architecture and it means either paying flagship prices for classification or accepting flagship-tier failures on the hard requests. The tasks inside one product vary enormously in difficulty, and pricing them all at the hardest one is the most common source of an inexplicable bill. Start with a rule on an obvious signal — input length, endpoint, customer tier — before reaching for a classifier.

How to decide

1Sample fifty real requests and label them easy or hard. The ratio is usually more lopsided than expected.
2Run the easy ones against the cheap model and measure quality honestly against your evaluation set.
3Add an escalation signal: a confidence score, an explicit uncertainty instruction, or a validation check on the output.
4Price the routed version including escalations paid twice, then compare against everything on the flagship.

Price it yourself

Frequently asked questions

How do I detect when to escalate?: A confidence score if the task produces one, an explicit instruction telling the cheap model to say when it is unsure, or a validation check on its output. All three cost less than the flagship call they avoid.
Is routing worth the complexity?: At low volume, no. Above a few thousand dollars a month it is usually the single largest saving available, and a rule on an obvious signal is an afternoon of work rather than a project.