Evaluation of Russian TTS models (Alpha Cephei)
nickolay-shmyrev of alpha-cephei ran thirteen Russian TTS systems — open and proprietary — through five metrics and published the table. Dated 2024-07-12. As far as this spoke can find, it is still the only independent side-by-side measurement of Russian TTS that exists, which is both why it is here and the main thing wrong with the field.
The test set is stated and it is the reason this page matters for narration work: audiobooks, about
100 speakers, about 1k utterances. The evaluation code is published
(alphacep/vosk-tts, extra/tts-test/ru).
The metrics, in the author's words
- CER, obtained with an ASR recognizer; "responsible for clarity of speech (phonemes are not swallowed, they are pronounced correctly)." Lower is better.
- xRT, the synthesis rate, GPU and CPU columns separately.
- UTMOS, "responsible for sound purity (studio sound quality)." Higher is better.
- Similarity, for multi-voice systems only, voice against the reference sample.
- Encodec FAD, "intonation quality." Lower is better.
Four of the five are automatic proxies rather than listeners, which is what makes a thirteen-system comparison affordable at all. mean-opinion-score is not in the table; no human panel was run.
The table
| Engine | Voice | CER | xRT GPU | xRT CPU | UTMOS | Sim avg/min | Encodec FAD |
|---|---|---|---|---|---|---|---|
| Silero v3_1 | Aidar | 0.7 | 0.0177 | 0.1256 | 2.544 | — | 97.36 |
| Silero v3_1 | Baya | 0.7 | 0.0177 | 0.1256 | 2.978 | — | 170.53 |
| Silero 4 | Aidar | 1.0 | 0.0149 | 0.0544 | 1.755 | — | 79.33 |
| Silero 4 | Baya | 0.9 | 0.0149 | 0.0544 | 2.144 | — | 118.63 |
| Vosk-TTS 0.6 | Multi | 2.3 | — | 0.0605 | 3.283 | 0.869/0.571 | 9.99 |
| TeraTTS | Natasha | 1.6 | — | 0.1945 | 3.281 | — | 70.10 |
| UtrobinTTS | Female | 2.1 | 0.0265 | 0.1323 | 2.851 | — | 73.34 |
| UtrobinTTS | Male | 2.1 | 0.0265 | 0.1323 | 3.186 | — | 46.14 |
| XTTS2 | Multi | 2.7 | 0.3458 | — | 3.035 | 0.762/0.468 | 97.05 |
| Vosk-TTS GPT | Multi | 2.1 | 0.2690 | — | 3.381 | 0.814/0.544 | 10.08 |
| Piper | Denis | 3.7 | — | 0.045 | 3.056 | — | 142.91 |
| Piper | Dmitry | 3.6 | — | 0.045 | 2.864 | — | 130.9 |
| Piper | Irina | 1.4 | — | 0.045 | 3.672 | — | 74.98 |
| Piper | Ruslan | 3.0 | — | 0.045 | 2.975 | — | 72.22 |
| BeneGes | Ruslan | 2.4 | — | 0.321 | 2.537 | — | 63.02 |
| EdgeTTS | Dmitry | 0.7 | — | 0.076 (cloud) | 3.565 | — | 32.69 |
| EdgeTTS | Svetlana | 0.7 | — | 0.076 (cloud) | 3.513 | — | 30.60 |
| Yandex | Alexander | 0.6 | — | 0.028 (cloud) | 3.413 | — | 54.10 |
| Yandex | Marina | 0.6 | — | 0.028 (cloud) | 3.482 | — | 49.40 |
| Tortoise Ruslan | Multi | 6.2 | 25.0300 | — | 2.893 | 0.660/0.483 | 14.21 |
| Bark Small | Ru_4 | 10.3 | 1.201 | — | 2.554 | — | 61.71 |
What the author concludes
Clarity beats intonation, and he says why. "Fastspeech2 methods still show best clarity (Silero/Yandex/EdgeTTS). They are not very good in intonation but clarity is hard to beat. For end user it really makes sense actually, you can deal with plain intonation but artifacts are really annoying." The table backs it: the three lowest-CER systems carry FAD between 30 and 170, and the two lowest-FAD systems sit at CER 2.1–2.3.
Data quality beats data quantity, with a worked example. "even a small size gives very good results (CER and UTMOS), if the data is good." Piper Irina, trained on 1 hour of data, takes the highest UTMOS in the whole table (3.672) and a CER of 1.4, against 3.0–3.7 for the three other Piper voices built the same way on different recordings.
Multi-voice costs clarity. "Multi-voice systems seriously suffer from fuzziness (CER 0.7 > 2.0+), something needs to be done about it." Every multi-speaker row — Vosk-TTS, XTTS2, Tortoise — sits at 2.1 or worse, and the single-speaker leaders sit at 0.6–0.7. For a wiki that has spent sixty pages recording zero-shot cloning as a free feature (free-voice-clone-list: 35 of 36 TTS entries claim it), this is a measured price for it in one language.
XTTS2 disappoints him. "XTTS2 results are much worse than I expected. Both similarity and clarity of speech." Similarity 0.762 average but 0.468 minimum. On some reference voices the clone barely resembles the target.
He also flags what he could not measure: "It is necessary to add another metric responsible for the liveliness of speech (F0 correlation? duration?). FAD is relevant, but only works for multivoice systems." By his own account the intonation column is unreliable for exactly the single-speaker systems that win on clarity.
The conflict of interest, stated plainly
Vosk-TTS is the author's own system, and it takes the best FAD (9.99) and the best UTMOS (3.381) in his table. The evaluation stays T2 here rather than dropping to T3 because the method, the metrics and the scoring code are all published and re-runnable, which is what separates a measurement from a claim — but a reader should weight the two columns Vosk-TTS wins differently from the CER column, which it loses.
Why a 2024 table is the current state of the art
Two years is a long time in this spoke and the gap is not evenly distributed:
- Silero has moved twice since. v5 shipped in October 2025 with automatic stress and homograph resolution built in; the table's newest Silero is v4. Its worst number here — Silero 4 Aidar at UTMOS 1.755, the lowest studio-quality score of any system tested — is a measurement of a model two major versions old.
- ElevenLabs is absent entirely, and so is every other frontier commercial system except Yandex and Microsoft's EdgeTTS.
- Piper was archived on 2025-10-06 and continues as a differently-licensed project.
Nobody has re-run it. That is the finding.
Related
russian-tts · silero-tts · yandex-speechkit · tts-front-end · tts-benchmarks · vosk · nickolay-shmyrev · alpha-cephei