vLLM Semantic Router

Project site for vLLM Semantic Router, a routing layer that sits in front of a pool of models and picks one (or a chain of several) for each request. The pitch on the page is "Build Your Mixture-of-Models": put closed models (Claude, ChatGPT, Gemini) and open-weight ones (Mistral, DeepSeek, Qwen) behind one OpenAI-compatible endpoint and let a classifier decide which handles what. It is part of the vllm-project GitHub organisation, Apache-2.0, 5.8k★ on the day of reading; launched September 2025, latest release v0.3 "Themis" (2026-06-05). Written in Go with Python and Rust-side bindings (ONNX, OpenVINO, Candle).

What it does, in the site's own terms

• Signals. The request is inspected for "20 signal families" — context, safety, intent, user preference, system state. This is where the classifiers live.
• Decisions. "11 selection algorithms and 5 loopers" turn signals into a model path: direct, specialist, local, cascade or collaborative. A "looper" is a multi-step path — the request goes to more than one model in sequence.
• Policy. PII and jailbreak detection, authorisation, rate limits, applied before the model is reached.
• Deployment. Runs as an Envoy ExtProc filter, a Kubernetes operator, a Gateway API backend, or a local binary (install.sh). Envoy carries the bytes; the router only decides.
• Observability. Per-request "replay records" with the routing evidence, OpenTelemetry metrics, a dashboard.

The page names maintainers from AMD, Microsoft, Red Hat, Delta Electronics and Google, and a second "vision" paper for a Workload → Router → Pool architecture.

The numbers

The site gives one headline: a composed pool scored 92.6 on LiveCodeBench against 92.0 for a single frontier model it calls "Fugu Ultra". No method, no cost column, no date for the run.

The founding paper — When to Reason: Semantic Router for vLLM (Wang, Liu, Liu, Zhu, Mo, Jiang, Chen; arXiv:2510.08731, 2025-10-09, a 5-page NeurIPS 2025 workshop submission) — is narrower and more checkable. It routes on one question only, does this query need chain-of-thought reasoning?, and reports, against direct vLLM inference on MMLU-Pro: +10.2 points accuracy, −47.1% latency, −48.5% tokens. The classifier is not named in the abstract. A second paper on semantic caching for heterogeneous workloads is listed (arXiv:2510.26835), unread here.

Why it sits in this spoke

Every mechanism page here — kv-cache, continuous-batching, speculative-decoding, quantization — makes one model cheaper per token. This is the first source that makes the system cheaper by not sending the tokens to that model at all. The paper's −48.5% tokens is a sampling-budget lever pulled from outside the engine: reasoning mode on or off, decided by a small classifier before prefill starts. That makes routing a fifth production lever, above the four the spoke had, and it is why model-routing gets a page.

What the source does not say: nothing on the classifier's own latency (a BERT-class model in the request path is not free), nothing on how often it is wrong and what a wrong route costs, and the LiveCodeBench claim is unsupported on the page. Treat the marketing numbers as T3 inside a T1 page.

Cross-spoke context

• ../agentic-tooling-wiki already holds inferoa, a coding harness built on this router, and its author xunzhuo — Xunzhuo Liu, second author of the paper and "Intelligent Routing @vllm-project". The harness is theirs; the router mechanism is here.
• ../llm-providers-wiki owns the models the pool is made of and their prices. The mixture-of-models pitch is an argument about that market — that a routed pool of cheaper models matches a frontier one — but this source only asserts it.
• ../platform-ops-wiki owns Envoy, Kubernetes operators and the Gateway API as operating practice. Here they are the delivery mechanism, not the subject.

Related

model-routing · vllm · token-sampling · tokens-per-second · llm-inference · local-llm-stack