Measuring AI Model Performance: Tokens per Second, Model Sizes, and Inferencing Tools (OpenMetal)

An explainer by Rafael Ramos for openmetal, a bare-metal and private-cloud IaaS vendor, on what tokens per second means and how a benchmark using it can mislead. Published 2025-03-28, last modified 2025-05-20. It carries the spoke's first CPU-versus-GPU throughput table and its first statement of the prompt-versus-eval distinction as a reporting rule rather than a mechanism.

The claim that earns the page

Report output tokens per second, not the total. Prompt (input) processing and eval (output) generation are different speeds, and OpenMetal says its own measurements put prompt TPS as much as 10× higher than eval TPS, varying by model and hardware. So a benchmark that quotes one combined number and feeds it a long prompt with a short answer reports mostly prefill and calls it generation:

"An easy way for a benchmark to skew results is to provide total tokens per second with a large prompt and small output."

This is prefill-decode-kv-cache's asymmetry restated as a measurement hazard, and it is the first source here to name the hazard directly. What it does not do is separate throughput from latency, or say anything about batching — the numbers below are per-configuration, with concurrency unstated.

The table

Read across, the GPU is 3.3× and 3.7× the CPU. Read down, going from 3B to 8B costs about a third of the throughput on either (52→33, 171→122) — 2.7× the parameters for ~1.5× the slowdown, which is not the ratio a memory-bandwidth-bound decode would predict on its own. No methodology is given: no quantization level, batch size, concurrency, context length, or serving engine. Treat the four numbers as an order of magnitude, not as measurements.

What else it covers

Tokenization — a token is a word, word-piece or punctuation mark; English averages ~1.3–2 tokens per word; punctuation and whitespace are usually separate tokens; code tokenizes differently. Consistent with tokenization and adds nothing measured to it.

quantization — FP16 down to INT8 (called the common production format) or 4-bit (FP4/INT4, "emerging", with accuracy risk), for speed, memory and power. It also distinguishes dynamic quantization (converted during inference) from static (converted ahead of time), and says static performs better. That dynamic/static split is not on the mechanism page here, and this is the source that names it.

Tools, briefly and with an Intel accent: OpenVINO ("up to 2× " on Intel CPUs via AMX), Intel PyTorch Extension / IPEX ("up to 25%" in their tests), NVIDIA TensorRT, Hugging Face Optimum. No vllm, no sglang, no llama-cpp — the omission tells you the audience is CPU and private-cloud buyers, not GPU serving operators.

Concurrency and use-case targets — CPU concurrency contends for cores and memory bandwidth, GPU concurrency for CUDA cores and VRAM; code completion is given a 50–100 TPS per user target, batch summarization is throughput-tolerant, RAG stresses both phases. The per-user target is the only requirement-side number in the spoke.

Why T3

A vendor blog post that sells the hardware its numbers were run on: the page ends in GPU-server pricing, a consultation booking and a $50k credits offer. The prompt-versus-eval argument stands on its own reasoning and is worth holding; the throughput table is first-party, unreplicated and methodology-free. Authored (Rafael Ramos) and dated, which keeps it above T4.

Connections

• tokens-per-second — the metric page this source founds
• prefill-decode-kv-cache — the mechanism behind the prompt/eval gap
• quantization — dynamic vs static; INT8 as the production default
• tokenization — what the denominator counts
• openmetal — publisher; llm-inference — the umbrella
• cross-spoke: the buyer-side of this argument (bare metal vs rented GPU, pricing) is ../cloud-wiki's