Cache-aware model routing
Routing a prompt to a cheaper model is usually described as free money: classify the prompt, send it to Haiku instead of Opus, pocket the price difference. The prompt cache breaks that assumption. A provider's prompt cache is scoped to one model, so switching models mid-conversation does not resume the old cache — it pays to write the whole context into the new model's cache from scratch. Named and measured by claude-code-model-router, a proxy built for claude-code.
Why the switch has a price
A cache read costs a fraction of input price (0.1× on Anthropic's API); a cache write costs a
multiple of it (1.25× on a 5-minute TTL, 2× on a 1-hour TTL — the one claude-code uses on a
subscription). Staying on the current model for a context of C tokens costs 0.1·p·C to keep
reading it. Switching to a cheaper model costs w·p_new·C to write it fresh. Working through
current Claude prices, the context-only breakeven needs the two models' price ratio above roughly
12.5–20×, which no pair on Anthropic's current line-up reaches. A downgrade therefore only pays
through the output tokens of the coming turn and the rest of a long tool loop — never through the
switch itself, and an upgrade is comparatively free to justify because it buys correctness rather
than saving money.
The resulting policy shape
- A fresh session has no cache to lose, so it routes purely on the classifier's read of the prompt.
- Upgrading is cheap to justify at any point — the risk of staying on a model that gets the answer wrong usually outweighs a cache-write cost.
- Downgrading needs its own threshold: only worth it when the coming turn, cache write included, is estimated to cost meaningfully less (the source uses 15%) on the smaller model than staying put. The estimate has to know the session's usage pattern (requests per turn, output tokens per request), the target model's cache TTL, and what that model's cache already holds — including the large fixed prefix a harness like claude-code resends every session (system prompt and tool definitions, on the order of 40k+ tokens).
The practical result is conservative: mid-session downgrades are rare by construction, because the context accumulated so far is exactly what a downgrade would forfeit.
Where it sits against the spoke's other routing coverage
tokenmining names routing (an LLM gateway classifying prompt complexity into a cheaper model) as one of its levers and reports roughly 60% cost reduction from routing alone, without discussing what switching itself costs. That figure describes routing before a conversation accumulates a large cached context — a first classification, not a mid-session downgrade. Cache-aware routing is the qualification the tokenmining framing is missing for exactly the harness it names most often (claude-code): once a session has built up tens of thousands of tokens of cached context, the same routing move that saved money on turn one can cost money on turn twenty. Both claims can be true at once because they describe different points in a session; see synthesis for where this is flagged as a live tension rather than resolved.
agent-orchestration's fan-out pattern sidesteps the problem structurally rather than solving it: each subagent runs its own context and its own cache, so there is no mid-conversation model switch to price — the cache-switch cost only exists for a single agent changing models under itself, never for a new agent spawned at whatever model tier the task needs from the start.
effort-level is the orthogonal control this mechanism is easy to conflate with: effort changes how much work one model does at a fixed capability ceiling; routing changes which model does the work. A harness can tune both independently, and neither one substitutes for the other.
Related
claude-code-model-router · claude-code · tokenmining · agent-orchestration · effort-level · claude-apps-gateway · agentic-coding-harness