TTS benchmarks

How text-to-speech quality is measured — and why "best" is multi-axis and volatile. The leading public ranking is the Speech Arena Elo board (tts-arena-leaderboard).

The metrics

• Elo (Speech Arena) — blind A/B human preference votes → an Elo rating; captures perceived naturalness but nothing about latency or accuracy. The headline number.
• MOS (Mean Opinion Score) — rated naturalness, typically on ~10-second clips (UTMOS training bounds it); kokoro scores ~4.5 MOS. The metric is older than TTS: it's a five-point absolute-category scale (5 Excellent → 1 Bad), averaged over listeners, standardized for telephony in ITU-T P.800 (mean-opinion-score). The number to distrust isn't the rating — it's comparing ratings across tests: ITU itself warns MOS values "should not be compared across experiments" unless designed to be, because listener pool, clips, and instructions shift the absolute scale. So "4.5 MOS" only ranks a model against the others in its own test — the standards basis for this page's snapshot discipline.
• WER / CER — word/character error rate via round-trip ASR transcription; depends on the ASR model, so it's noisy. fish-audio-s2-pro ~3.5% WER / 1.2% CER (English, vendor figure).
• TTFA (time-to-first-audio) — the latency metric that matters for real-time UX; Sonic 3.5 ~82ms, Deepgram Aura-2 <90ms, misotts claimed 110ms.

Caveats (why these are snapshots)

• Rankings shift weekly — tts-models-2026-benchmark stresses leaderboard positions are dated snapshots, not fixed truth (this wiki dates every figure).
• Sources "disagree" mostly because they are dated differently. This wiki recorded Gemini at 1216 vs 1217 and Fish S2 Pro at 1123 vs 1128 and filed it as methodological disagreement between vote pools. The 2026-08-03 refresh argues it was mostly drift: over two months Gemini moved 1217 → 1212, Fish S2 Pro 1128 → 1123, Inworld's Realtime TTS-2 fell 1206 → 1189. Single-digit gaps between sources published days apart are the board moving, not the boards differing. Real methodological difference would show as ordering changes, and those happened too — so date every figure and compare orderings, not points.
• Vendor-reported numbers (latency, WER) have incentives; treat as indicative.
• Metrics are partial: Elo ≠ accuracy ≠ latency. A model wins or loses per axis.

The missing metric: speaker similarity (added 2026-08-25)

The metrics above measure whether the speech is correct (WER/CER), pleasant (mean-opinion-score, Elo) and prompt (TTFA). None of them measures whether a cloned voice sounds like the person it cloned, which is the thing voice-cloning is bought for.

Re-checked 2026-09-02 (absence-claim drain): true of the metrics listed above, no longer true of the field. The sentence is correct as scoped — WER/CER, MOS/Elo and TTFA do not measure speaker similarity — but "missing metric" overstates it. A dedicated public benchmark exists (ClonEval, arXiv 2504.20581, "An Open Voice Cloning Benchmark") alongside an Open Voice Cloning Leaderboard that ranks models on mean cosine similarity between WavLM-Large speaker embeddings of reference and generated audio, across datasets including emotional speech. SIM is the standard name for the metric, and published figures put a human baseline around 0.73. So the axis is measured, publicly and comparably; what this spoke lacks is a page for it. Recorded as a lead for the growth edges rather than folded in here from a search result — neither benchmark has been read.

audio8-tts publishes both columns and they split cleanly: its ~0.17B checkpoint is within 0.06 WER of the 4.6B fish-audio-s2-pro on Seed-TTS English, and 8 similarity points behind it (56.7 vs 64.6), with IndexTTS2.5 26 points ahead of it at 82.3 while being the worst model in the same table on WER (3.253). Rank the table on error rate and you get one ordering; rank it on similarity and you get close to the reverse.

Two consequences. Any single-number cloning comparison is choosing an answer, and the roundups this wiki draws on (tts-models-2026-benchmark, open-source-tts-models) print error rates. And the Elo boards do not rescue it — tts-arena-leaderboard pools preference votes across whatever listeners weigh, so a similarity failure and a prosody failure land in the same number. No board tracked here reports similarity at all; the only figures the spoke holds come from one publisher's own table.

Related

tts-arena-leaderboard · text-to-speech · open-weight-tts · tts-models-2026-benchmark · mean-opinion-score · speech-audio-ai · voice-cloning · audio8-tts