Spokes.wiki Search About
Defined Term mechanism stable
Created
Updated

Voice cloning

Synthesizing speech in a target speaker's voice from a short reference sample — ideally zero-shot (no per-voice fine-tuning). A major capability axis that splits the text-to-speech field.

Who has it (and who doesn't)

So "can it clone a voice?" is a clean dividing line: presence of cloning often correlates with larger models and conditioning on reference audio, while the smallest efficiency models drop it.

The dividing line stopped dividing (2026-08-03)

A 36-model census of open TTS (free-voice-clone-list, April 2026) marks 35 of 36 with zero-shot cloning, the sole exception being supertonic-2 at the small, fast end. The list sorts on languages, streaming and licence instead, which is what a capability looks like once it becomes table stakes.

That census also credited kokoro with cloning, contradicting the two roundups this page was built from. Checked and settled the same day against the Hugging Face model card: Kokoro-82M does not support cloning, and its voices/ directory holds 54 .pt voicepack tensors selected by file, with no reference-audio path. So kokoro remains this page's example of a model that drops the capability by design, now on primary evidence rather than on two secondary sources agreeing.

The size correlation is also weakening. kittentts claims cloning at 15M parameters, a fifth of kokoro's footprint, which if it holds cuts the link between cloning and model size that this page was written around. Unverified — the only source is a curated list with no measurements. (Re-checked 2026-09-14: the measurements exist outside the corpus — ClonEval, arXiv 2504.20581, is an open voice-cloning benchmark with a deterministic protocol and public leaderboard, and vendor tables report speaker similarity on Seed-TTS and Minimax-MLS. Not ingested; a candidate for the spoke's growth edges.) (Re-checked 2026-10-11: KittenTTS's own README at v0.8.1 lists eight built-in preset voices and describes no voice-cloning feature at all. So the 15M cloning claim rests on the curated list alone, and the primary source does not make it. See kittentts.)

Reference-sample length has converged too: entries across the catalogue advertise cloning from a three-to-ten-second sample, matching the three seconds luxtts claims. Whatever cloning cost in data, it no longer costs much.

A grounded exemplar: XTTS

The open zero-shot cloner with a peer-reviewed primary is XTTS (Casanova et al., Coqui, INTERSPEECH 2024 — xtts-paper). It pins down two things the roundups only implied. First, cloning travels across languages: XTTS "was trained in 16 languages and achieved state-of-the-art results in most of them" — one zero-shot model cloning a voice across many tongues, not a per-language system xtts-paper. Second, it locates cloning in a second architectural lineage: XTTS "builds upon the Tortoise model," i.e. the autoregressive route to cloning, distinct from the codec-token route (neural-audio-codec) — two paths to the same capability. And it is openly released, which is what makes zero-shot cloning an off-the-shelf commodity (the permissive side of open-weight-tts's license split) and gives the consent problem below its teeth.

Adjacent capabilities

Cloning sits alongside emotion/style control (guided emotion in orpheus; Hume Octave 2 inferring emotion from meaning) and multi-speaker dialogue (Dia, sesame-csm). Together these "controllability" features, not just raw naturalness, increasingly differentiate models.

Note

Voice cloning carries obvious consent/impersonation risk; the wiki tracks it as a technical capability, but it is the capability most entangled with abuse and licensing/consent questions. The catalogues that distribute the capability say nothing about it — see audio-deepfake for the census.

Cloning a voice vs. editing a recording (added 2026-08-06)

voicecraft does something this page's axis does not cover. Cloning conditions a new utterance on a reference voice; VoiceCraft's first task is infilling an existing recording — replacing words inside real audio while the surrounding speaker, prosody and room stay put — with zero-shot TTS falling out as the case where everything is infilled. So "can it clone a voice?" has a sharper sibling: can it alter a recording that already exists? The census above says cloning is table stakes; editing is not, and no leaderboard in this wiki ranks it.

The measured axis is similarity, and it is not the one that gets quoted

audio8-tts is the first model here to publish cloning quality as two separate numbers, and they point in opposite directions. Its ~0.17B checkpoint lands within 0.06 WER of the 4.6B fish-audio-s2-pro on Seed-TTS English — near-parity on intelligibility — while scoring 56.7 speaker similarity against Fish's 64.6 and IndexTTS2.5's 82.3. Word-error rate says the small model arrived; similarity says it did not.

Which matters, because cloning is the product and WER is what roundups print. Nothing else in this wiki carries a similarity score at all: the census sorts on a checkbox, and the tts-arena-leaderboard Elo pools preference votes over everything at once. So the field's one capability axis that stopped dividing models on presence (free-voice-clone-list: 35 of 36) has a quality axis underneath it that almost nobody reports — see tts-benchmarks.

audio8 puts consent-before-impersonation and disclosure-of-synthetic-audio in the licence text of a model that clones, rather than in card prose that binds nobody. synthesis records the census finding — zero occurrences of consent, watermark or misuse across 53 open audio models — and the single OpenRAIL-M exception attached to supertonic-2, which has no cloning. This is the second instance and the first on a cloning model. Two data points, and still not a trend.

text-to-speech · xtts-paper · orpheus · sesame-csm · fish-audio-s2-pro · kokoro · kittentts · supertonic-2 · free-voice-clone-list · open-source-tts-models · audio-deepfake · voicecraft · open-weight-tts · audio8-tts · audio8 · tts-benchmarks

Related pages

Links to

Linked from