Speech-to-text (STT / ASR)

Automatic speech recognition — transcribing speech audio into text. The recognition branch of speech-audio-ai (the mirror of text-to-speech's synthesis).

How it's measured

WER (word error rate) — the headline accuracy metric; lower is better. The neutral anchor is Hugging Face's Open ASR Leaderboard (since 2023; 700K+ visits; recently added a "benchmaxxer repellant" to resist overfitting). 2026 English leaders: canary-qwen 5.63%, IBM Granite Speech 5.85%, whisper V3 7.4% open-source-stt-models.
RTFx (real-time factor) — throughput; how many ×-real-time it transcribes. NVIDIA Parakeet TDT >2,000× (streaming, RNN-Transducer); Whisper V3 Turbo 216×.
Latency — for streaming/agents: Deepgram <300ms, ElevenLabs Scribe ~150ms stt-apis-comparison.
Language coverage — Whisper 99+, Google Chirp 125+, Qwen3-ASR 52.

The leaderboard stopped deciding things (2026-07)

The WER-ranking frame above has been overtaken twice in one month, from opposite directions.

From the top: MarkTechPost's July roundup finds the top ten open models inside one WER point, with the board reordering when Appen's private-track data is toggled on — and canary-qwen, recorded here in June as the leader at 5.63%, now sits behind ARK-ASR-3B (5.04), Granite Speech 4.1 (5.33), Cohere Transcribe (5.42) and MOSS-Transcribe-preview. The reported WERs aren't even measured on the same set count (8 sets, 7 with TED-LIUM dropped, LibriSpeech-only). What decides a pick instead: licence — Apache-2.0 versus the CC-BY-4.0 attached to Canary-Qwen, Parakeet and Kyutai — plus language coverage, whether the architecture streams at all, and cost per audio-hour. Two more axes arrive as products rather than research: diarization (MOSS-Transcribe-Diarize 0.9B; and, as a post-processing LLM, diarizationlm-gemma-4-e4b — phone-call WDER roughly halved, meetings barely moved) and non-autoregressive decoding (Granite 2B-NAR). Language coverage gained an extreme: Meta's Omnilingual ASR at 1600+ languages (5400+ zero-shot, ~10% CER on 78% of them) against Whisper's 99.

From below: a single average WER hides which kind of audio a model handles. VibeVoice-ASR-BitNet publishes a 15-benchmark table where it beats Whisper on meeting and far-field speech by 5–11 points (AMI-sdm 25.87 vs 36.92) and loses to it on clean read speech (Libri-clean 2.41 vs 1.98) — with Parakeet winning the clean sets outright and losing every meeting set. Which model is "most accurate" depends entirely on which half of that table matches your audio, which is the same conclusion the roundup reaches by a different route.

The market shape

Mirrors the rest of speech-audio-ai: commercial APIs hold a thin accuracy edge (ElevenLabs Scribe ~3.3% EN, Deepgram 5.26% batch — stt-apis-comparison) over the open-source field (whisper, canary-qwen, Parakeet, Granite, Qwen3-ASR — open-source-stt-models), with the usual build-vs-buy crossover at high volume.

The defining 2026 trend — STT meets the LLM

Top accuracy now comes from SALM-style models that bolt an LLM decoder onto a speech encoder (canary-qwen = FastConformer + Qwen3; Granite-Speech; Qwen3-ASR) — recognition reframed as language modeling, the bridge to llm-providers-wiki (gemini, Qwen/Llama). whisper no longer leads WER but wins on ecosystem (MIT, languages, tooling).

The application layer — on-device dictation (2026-07)

The WER leaderboard is only half the STT story. The other half is what these engines get used for on a personal machine, where the axis flips from accuracy to privacy, offline operation, and desktop integration. vocalinux (Linux voice dictation) is the wiki's first source on this layer: it wraps a chosen local backend — whisper-cpp, whisper, or vosk — and does the hard OS work of injecting text under both X11 and Wayland. It also surfaces the small/local pole the accuracy pages skip: vosk (Kaldi-based, ~50 MB, runs on a Pi) is picked for constraint, not WER, and whisper-cpp is what makes Whisper responsive on a laptop. So the open-weight STT wedge has a consumer end-point — assemble a private dictation tool from off-the-shelf engines, no cloud involved.

A third pole sits beside accuracy and constraint: how much a model is asked to hear. sensevoice does ASR, spoken language identification, emotion recognition over seven categories and audio-event detection (applause, laughter, coughing, BGM) in one pass, which treats transcription as one output of a speech-understanding model rather than as the task. Its licensing splits the way this wiki keeps seeing — MIT code, weights under a separate agreement — so "open" describes the repository and not the model.

What the accuracy numbers are measured on (added 2026-08-22)

Every WER quoted on this wiki until now came from read speech — librispeech, common-voice, fleurs. chime-6 measures the other case: four people talking over each other at a dinner party, captured on distant arrays. Baseline 51.3%, best system of the challenge era 45.6%, and 77.9% once the system has to find the utterance boundaries itself. Its metric is cpWER, which charges a system for attributing correct words to the wrong speaker, so it is not directly comparable with a single-speaker WER — that non-comparability is itself the finding.

And they are measured in the wrong mode as well (added the same day). on-device-streaming-asr-study runs six model families in batch and streaming on one pinned CPU, and the ranking inverts: Qwen3-ASR is the most accurate in batch (5.90%) and streams at 10.45% and 0.49× real time; Parakeet doubles its error rate to 12.83%; Nemotron goes from third to first at 7.28% and 0.56 s latency. Batch WER does not predict streaming WER. Nor does RTFx predict responsiveness — Canary has the best throughput in that table (2.93×) and the worst latency (4.8 s). The two failures are independent: fixing the audio would still leave the mode wrong, and fixing the mode would still leave the audio wrong.

on-device-streaming-asr-study · speech-audio-ai · text-to-speech · whisper · whisper-cpp · vosk · vocalinux · canary-qwen · open-source-stt-models · stt-apis-comparison · tts-benchmarks · sensevoice