On-device streaming ASR, measured across six model families

Banfic, Fan, Vaishnavi, Kemp, Choi, Ren, Shaw & Tang — CoreAI, Microsoft — Pushing the Limits of On-Device Streaming ASR, arXiv:2604.14493. Over 50 configurations across six model families, batch and streaming, on named hardware. Found by the research pass hunting growth edge 1.

T2. Microsoft built none of the six models compared — Whisper is OpenAI's, Nemotron, Parakeet, Canary and Conformer-Transducer are NVIDIA's, Qwen3-ASR is Alibaba's. A party measuring six other parties' models and publishing the losing configurations is the neutral position this spoke has been looking for on the model axis. What it does not have is a product in the race, which is exactly what disqualified every vendor comparison the 2026-08-08 hunt returned.

The hardware, named

AMD EPYC 7V12 64-core (AVX2, 2.45 GHz), inference pinned to 32 cores, ONNX Runtime, CPU-only for the streaming results; an NVIDIA H100 for the PyTorch baseline. Eight ESB benchmarks: AMI, Earnings22, GigaSpeech, LibriSpeech clean and other, SPGISpeech, TED-LIUM, VoxPopuli — so the conversational and far-field sets chime-6 argued for are partly here, beside the read speech of librispeech.

That is the first half of what edge 1 demanded. The second half — concurrency — is not varied anywhere in the paper. See below.

Batch WER, and then streaming WER, and they disagree

Batch mode:

Model	Avg WER	AMI	LS-Clean	VoxPopuli
Qwen3-ASR-1.7B	5.90%	11.76	1.60	6.34
Parakeet TDT-0.6B-v3	6.32%	11.39	1.92	6.09
Nemotron-0.6B	7.07%	11.16	2.28	7.48
Canary-1B-v2	7.15%	16.01	2.18	6.25
Whisper-v3-Turbo	7.83%	16.13	2.10	11.87
Whisper Small.en	8.59%	17.93	3.05	8.50

Streaming, same models, on the CPU:

Model	Size	Algorithmic latency	Avg WER	RTFx
Nemotron-0.6B	2.47 GB	0.56 s	7.28%	2.46×
Conformer Transducer XL	2.58 GB	2.4 s	11.06%	1.27×
Parakeet TDT-0.6B-v3	2.51 GB	2.4 s	12.83%	1.38×
Canary-1B-v2	6.36 GB	4.8 s	12.45%	2.93×
Qwen3-ASR-1.7B	4.70 GB	2.4 s	10.45%	0.49×

The ordering inverts. Qwen3-ASR is the most accurate model in batch and the second-worst in streaming, at 0.49× real time — it cannot keep up with the audio on this CPU. Parakeet goes from second place at 6.32% to 12.83%, doubling its error rate. Nemotron goes from third to first, and holds 7.28% at a latency an order of magnitude below the rest.

This wiki's accuracy evidence is almost entirely batch: open-asr-models-2026-comparison, tts-benchmarks's STT companion boards, the Open ASR Leaderboard numbers that circulate as the ranking. None of it predicts how the same model behaves streaming, which is the mode every real-time product runs in. That is a bigger correction than the latency number the edge went hunting for.

And the two speed axes are not the same axis. Canary has the best throughput here (2.93× RTFx) and the worst latency (4.8 s). Nemotron has lower throughput and eight times better latency. RTFx is a batch-economics number — how many audio-hours per wall-clock hour — and it says almost nothing about when the first word appears. The spoke has been carrying RTFx figures like 3333× from open-asr-models-2026-comparison with no way to say what they mean for a conversation; they mean cost, not responsiveness.

Quantization is close to free here

Same model, four builds, all measured:

Variant	Format	Size	Avg WER	RTFx	Device
Baseline	PyTorch	2.47 GB	7.28%	—	CUDA
FP32	ONNX	2.47 GB	8.03%	6.73×	CPU
int8 k-quant	ONNX	1.28 GB	8.01%	7.25×	CPU
int4 k-quant	ONNX	0.67 GB	8.20%	7.20×	CPU

2.47 GB to 0.67 GB for 0.17 points of WER against the FP32 ONNX build, and int8 is very slightly better than FP32 on accuracy — noise, but not a penalty. The cost that shows up is the PyTorch→ONNX move (7.28% → 8.03%), not the precision. A four-times smaller model that still runs at 7× real time on a pinned 32-core CPU is the on-device claim, evidenced.

Note the RTFx column disagrees with Table 5's 2.46× for the same model. The 7× figures are the quantization sweep's own measurement configuration; the paper does not reconcile the two, and this page reports both rather than picking one.

What it does and does not do for growth edge 1

Half-close. The edge wants "a T1/T2 independent latency measurement across providers with stated hardware and concurrency." Two of those three arrive: independent, and hardware stated down to the core count. Two do not.

No concurrency. Nothing in the paper varies simultaneous streams, which is the condition that separates a benchmark from a serving measurement — a model that meets its latency budget on one stream may not on forty, and that is the number an operator needs.

Not the providers the edge was aimed at. The vendor claims this spoke distrusts — elevenlabs's "under 150 ms", the Inworld and Deepgram figures, supertonic-2's table — are hosted APIs. This measures self-hosted open models on a CPU. It disciplines the open-model half of the field and leaves the API half exactly where it was: still unmeasured by anyone without a product in the comparison.

The authors state their own limits plainly, which is a point in their favour: the evaluation "does not cover several important system-level capabilities often required in production settings, such as robust inverse text normalization (numbers, dates, currencies), speaker diarization, code-switching, custom vocabulary adaptation," and it is English-only.

speech-to-text · open-asr-models-2026-comparison · open-source-stt-models · whisper · canary-qwen · chime-6 · moonshine · librispeech · synthesis

The other half of the instrument (2026-09-06, research pass)

open-asr-leaderboard was read for its methodology on the same edge this page half-closed, and the two are complements rather than rivals. This study has streaming, latency and a named CPU; the board has a fixed H200, a Docker image per model family, a published per-run cost and a private test half — and reports RTFx, which is throughput, on batch short-form audio. Neither varies concurrency. The edge's four conditions are now covered across two sources measuring different things.