Model routing
Model routing is choosing, per request, which model (or which mode of one model) answers it, before inference starts. A small classifier reads the prompt and emits a decision; a gateway acts on it. The models behind the gateway need not share a vendor, a size, or a location.
It is a serving-layer lever of a different kind from the ones on llm-inference. kv-cache, continuous-batching, speculative-decoding and quantization each make a given forward pass cheaper. Routing changes which forward pass runs — a 7B model instead of a 400B one, or reasoning off instead of on — so its savings are in tokens not generated, and its cost is the classifier's own latency plus every misroute.
What is sourced
One source so far, vllm-semantic-router:
- Reasoning on/off is the best-measured case. The founding paper (arXiv:2510.08731) routes on whether a query benefits from chain-of-thought and reports +10.2 MMLU-Pro points, −47.1% latency, −48.5% tokens against direct vLLM inference. The gain in accuracy comes from the queries that got reasoning; the savings come from the ones that did not.
- Model selection across a pool ("mixture of models") is the product claim — a pool scoring 92.6 vs 92.0 on LiveCodeBench against one frontier model — with no method published.
- Cascades and loops. A route can be a sequence: try cheap, escalate on a signal, or send the request through several models. The source calls these "loopers" and names five; none is measured.
- Placement. Envoy ExtProc is the deployment the project leads with — the router is a filter the proxy consults, and the proxy carries the traffic. The router also gates on PII and jailbreak classifiers in the same pass, so routing and policy share one hop.
A second pattern: gating after the fact, not routing before it
lev (introducing-lev, 2026-09-24) is a different shape of the same idea. vLLM Semantic
Router decides before any inference runs, from the prompt alone; Lev runs a cheap classifier
first and only escalates the specific questions that classifier itself scored below a confidence
threshold. That answers part of this page's own open question about classifier cost: Lev's fast
tier (laya checkpoints) runs in ~95 ms on a laptop CPU, and on its own 144-case adversarial set,
gating at 0.5 escalated 126 of 144 cases and raised accuracy from 61.1% to 92.4% at roughly double
the per-case latency. It also surfaces a hazard neither source names elsewhere in this spoke: a
binary (yes/no) question's confidence is max(p, 1-p), which structurally can never fall below
0.5, so a single global threshold silently never escalates a whole class of questions. Lev's fix —
a separate threshold per question type — is itself a routing decision, just one made after a cheap
forward pass rather than instead of one.
Open questions
- What the classifier costs. A BERT-class model on every request adds a fixed prefill-like latency, of the same order as the ~19 ms tokenization step. No source here measures it.
- What a misroute costs. A hard query sent to the cheap path fails; an easy one sent to the expensive path wastes exactly the tokens routing was meant to save. The paper reports averages, not the error distribution.
- Whether the kv-cache survives the route. Prefix caching and prefill-decode-disaggregation assume the request stays on one engine; a cascade that re-sends the prompt pays prefill again at every hop.
Related
vllm-semantic-router · lev · vllm · llm-inference · token-sampling · local-llm-stack · tokens-per-second