AI gateway

The layer between an application and the providers: one endpoint, usually OpenAI-compatible, that fans out to many model backends and handles key storage, fallback, quota accounting and cost tracking. Named examples in the wiki’s sources: OmniRoute, OpenRouter, LiteLLM, CLIProxyAPI, 9router omniroute — and, in managed form, amazon-bedrock.

Two shapes of the same idea

• Managed reseller — amazon-bedrock, Azure AI Foundry. The vendor holds the contracts and bills you; the catalog is what they chose to carry.
• Self-hosted router — omniroute and the OpenRouter/LiteLLM class. You hold the keys, run the process locally, and the catalog is whatever you can sign up for. Free tiers become usable in bulk here, because the router is the thing tracking which pool still has headroom.

Both commoditize model choice into a config parameter, but they push the leverage to different places: the reseller sells convenience and enterprise trust, the local router sells cost control and the ability to burn other people’s free quota systematically.

What the router layer actually controls

• Failure policy — cascade fallback, circuit breakers, model lockouts. This turns a provider’s rate limit from an outage into a routing event, which is why rate limits and refusal/fallback semantics are gateway concerns rather than application ones.
• Cost policy — cost-optimized and headroom strategies pick the cheapest backend that can serve the request, the practical form of the “model routing” lever in llm-api-pricing.
• Token volume — prompt compression before the call. A gateway that strips 15–95% of the prompt is manipulating the one multiplier the rate card doesn’t show (cf. the tokens-per-task argument in grok-4-5-price-vs-benchmarks).

A third shape — the gateway that calls itself a model (sakana-fugu)

Both shapes above admit what they are: you know you’re using a router, and you pick the backend. Fugu routes across all supported providers by default, narrows the pool at the API key rather than the request, and presents the whole thing as model="fugu" with a reasoning-effort parameter fugu-get-started. The routing is not a feature of the product; it is the product, wearing a model’s name.

That inverts the leverage described above. A gateway exists so the buyer can control routing; here the vendor controls it and sells the outcome. The buyer gains one dial (effort) and loses the two that matter to this spoke — which model answered, and what it cost per token. It’s the logical end of “provider loyalty is an implementation detail”: if model choice is already a config parameter, someone will sell you the configuration.

Why it matters to the market

It is the demand-side answer to a supply side that stopped differentiating. If a closed Claude and an open Llama arrive through the same API call, the buyer’s real decisions move up a level — to routing, caching and quota arbitrage. The gateway is where those decisions get made, and it makes provider loyalty an implementation detail.

Related

llm-provider · amazon-bedrock · omniroute · sakana-fugu · fugu-get-started · llm-api-pricing · claude-refusals-and-fallback