Cerebras Inference

A hosted LLM inference service from cerebras-systems that serves open models (Llama, Qwen and the like) at token rates far above a typical GPU endpoint — the vendor frames the jump as seconds-to-milliseconds. The speed comes from the hardware underneath: Cerebras runs on a wafer-scale processor rather than a rack of discrete GPUs, so the whole model sits in on-chip memory and the decode loop is not paying to shuttle weights across a PCIe/NVLink boundary every step.

It is an inference provider, not an engine you install — the counterpart to vllm/llama-cpp one layer up: you send it a prompt over an API, you don’t pick the serving stack. That makes it an instance of the datacenter regime pushed to an extreme on the latency axis, where the interesting consequence is not throughput but what sub-second responses do to the application built on top — see designing-for-cerebras and the generalized fast-inference-architecture.

Why it belongs in this spoke

The spoke reads inference as a set of serving regimes defined by how the runtime reaches the hardware and what constraint binds. Cerebras is a data point at the fast end of the datacenter regime: same prefill/decode pipeline, same kv-cache, but a hardware substrate (wafer-scale) chosen to minimize per-token latency rather than to maximize GPU utilization across many tenants. It is the first source here where the binding story is raw speed and its knock-on effects, not memory or batching.

Open / unverified

• The token-rate advantage is a vendor claim (designing-for-cerebras, T2) — no independent, hardware-specified benchmark is in the spoke yet. This is the same which-lever-bought-what gap synthesis tracks, now pointed at a hardware lever.
• How the wafer-scale substrate changes the standard optimizations (does it need continuous-batching the same way? where does quantization land?) is not covered by this source.