Open ASR Leaderboard

The instrument behind most of the WER numbers in this spoke, finally read for itself. Eleven pages here cite it — whisper, canary-qwen, moonshine, vosk, open-source-stt-models, open-asr-models-2026-comparison and the speech-to-text spine among them — and until today none of them recorded how it measures. Apache-2.0, first commit 2023-07-12, 247 stars, still pushed 2026-09-04; the board itself is a Gradio Space at hf-audio/open_asr_leaderboard.

What it actually publishes

WER and RTFx, together. Every run_eval.py prints both after a run, and the repository's own plots pair them on two axes. RTFx is a throughput ratio — audio seconds transcribed per wall-clock second — so a model that reads an hour of audio in a minute scores 60.

On stated, fixed hardware, since 2026-07-24. English and multilingual short-form evaluations run through Hugging Face Jobs, "every run executes a Docker image on the same hardware, to minimize environment and driver differences." The flavor is named in the README:

Long-form evaluation had not migrated to Jobs when this page was written, so its numbers do not carry the same guarantee.

With the cost of a run published. A full English short-form pass: $2.92 for nvidia/parakeet-tdt-0.6b-v3, $4.75 for openai/whisper-large-v3-turbo, $5.58 for Qwen/Qwen3-ASR-1.7B. A benchmark that prints its own bill is rare enough to note, and it is what makes "re-run it yourself" a real invitation rather than a gesture.

Datasets, named. Short-form English from hf-audio/open-asr-leaderboard; long-form from hf-audio/asr-leaderboard-longform (earnings21, earnings22) plus CORAAL held separately for its splits; multilingual across FLEURS, Common Voice and MLS, with Hindi from Voice Arena. Artificial Analysis's chunked Earnings22 set is scored session-level: models transcribe chunks, the scorer concatenates a call's predictions before computing WER against the call-level reference.

And a private half. After a model is submitted, "the maintainers will evaluate on private sets" — the standard defence against tuning to a public test set, run by the party that holds the answers.

Why the research pass went looking

Growth edge 10 wants a neutral latency benchmark with stated hardware and concurrency. This is neutral (Hugging Face runs models it did not build), hardware-stated to the accelerator, reproducible to a Docker image, and priced. Two of the four conditions the edge names are met properly for the first time in three hunts.

It is a half close, and the residue is specific. RTFx is throughput, not latency: it says how fast a file is consumed, not how long a listener waits for the first word, and the short-form path is batch transcription rather than streaming. Concurrency never appears at all — every number is one model, one job, one GPU. So the corpus still has no independent measurement of what a hosted streaming API does under load, which is the claim vendors actually make. Corrected 2026-09-17 (claim drain): the streaming half is now measured independently. Artificial Analysis's AA-WER Streaming (1 June 2026, ~8 h of audio over three sets, latency from Silero-VAD end of speech to first partial and to final transcript) puts ElevenLabs Scribe v2 at 3.64% WER / 0.14 s final, Cartesia Ink-2 at 3.59% / 0.21 s, AssemblyAI U3 Pro at 4.46% / 0.47 s, and Deepgram Flux fastest at 0.020 s with 7.36% WER; Coval reports a five-API TTFT/WER run at 2,400 trials per model. What is still unmeasured by anyone independent is concurrency — every one of those runs is one stream at a time — so the load half of the sentence stands and the residue is narrower.

Checked against AA's method 2026-10-01. The AA-WER Streaming methodology page (artificialanalysis.ai/speech-to-text/methodology) says audio is "streamed in real time in 20ms chunks", that the latency timer starts when the audio up to the end of speech has actually been sent, and that "network delay is included in all latency measurements". It names no concurrency, no region and no client hardware, and gives no repeat count for the streaming runs. So the one-stream-at-a-time reading above is the method, not an omission in a summary of it.

The relationship to on-device-streaming-asr-study is worth stating plainly: that study has the streaming and the latency and no concurrency; this board has the reproducibility, the fixed hardware and the price, and no streaming. Between them the edge's four conditions are covered — but by two sources measuring different things, which is not the same as a benchmark that covers them.

Related

speech-to-text · on-device-streaming-asr-study · open-source-stt-models · open-asr-models-2026-comparison · whisper · canary-qwen · moonshine · vosk