Tokens per second (TPS)

The rate at which a serving stack moves tokens, and the unit almost every claim in this wiki is denominated in. It is also the wiki's most abused number, because "tokens per second" names at least three different quantities and sources rarely say which one they mean.

Three numbers, one name

Prompt (input) TPS measures prefill: the whole prompt goes through the model in parallel, so this is the fast, compute-bound direction. Eval (output) TPS measures decode: one token at a time, each pass reading the kv-cache, bound by memory bandwidth rather than arithmetic. Total TPS divides both together and is the one to distrust — openmetal-tokens-per-second measured prompt TPS running up to 10× eval TPS, so a total figure is mostly a statement about the prompt-to-output ratio the benchmarker chose. A long prompt and a short answer make any stack look fast.

The split is not a benchmarking convention layered on top of the mechanism; it is prefill-decode-kv-cache's two-phase asymmetry, seen from the outside.

Per user or per server

A second ambiguity the sources here do not always resolve: TPS per request (what one user perceives as speed) and TPS aggregated across concurrent requests (what the operator is buying) move in opposite directions as batch size rises. continuous-batching exists to raise the second, and it raises the first not at all. Whenever a number here has no concurrency attached, it is a per-request figure at unstated load.

What moves it

• Parameters. ~52 → ~33 eval TPS going from a 3B to an 8B Llama on one CPU, ~171 → ~122 on an A100 openmetal-tokens-per-second.
• Hardware. The same source puts an A100 at 3.3–3.7× an Intel Xeon Gold 6530 on those two models.
• quantization, continuous-batching, speculative-decoding, attention variants (grouped-query-attention, latent-attention) — every lever this wiki holds is ultimately reported in this unit, which is why growth edge 1 (which lever bought what) is stated the way it is.
• The denominator itself. Tokens are not a fixed amount of text: tokenization differs by model family, so two stacks quoting the same TPS on the same prose are not necessarily doing the same amount of work.

Requirement side

The only stated target the spoke holds: 50–100 TPS per user for code completion, with batch summarization tolerating less and RAG stressing prefill and decode together openmetal-tokens-per-second. A vendor's figure, unreplicated.

Related

openmetal-tokens-per-second · prefill-decode-kv-cache · continuous-batching · tokenization · quantization · llm-inference · synthesis