VoiceCraft
Puyuan Peng, Po-Yao Huang, Abdelrahman Mohamed and David Harwath’s “token infilling neural codec language model” (arXiv:2403.16973), claiming state-of-the-art on speech editing and zero-shot text-to-speech over real recordings — podcasts, audiobooks, internet video. Weights at 330M and 830M, trained on GigaSpeech XL, over an EnCodec-style codec of 56M parameters, 4 codebooks × 2048 codes. 8,500+ stars, 797 forks, 95 open issues. Code under CC BY-NC-SA 4.0, weights under the Coqui Public Model License 1.0.0, with MIT and Apache-2.0 components. Runs on Python 3.9.16 / PyTorch 2.0.1 / CUDA 11.7 with ffmpeg, espeak-ng and the Montreal Forced Aligner; Colab, Docker and Gradio provided.
T1 — the project’s own repository and paper. Benchmark claims are the authors’; nothing here is independently measured.
The task this wiki did not have: editing, not generating
Every model paged in this spoke so far produces speech — kokoro, orpheus, fish-audio-s2-pro, misotts and the rest synthesize an utterance from text, and voice-cloning is about whose voice comes out. VoiceCraft’s first task is different: take an existing recording and change words inside it, keeping the speaker, the room and the prosody around the edit intact. That is infilling — the codec-token equivalent of masked language modelling — rather than generation from scratch.
It matters for how this wiki’s categories are drawn. TTS asks “can it sound like a person”; editing asks “can a real recording be altered without the alteration being audible”. The second is a strictly stronger claim about a specific piece of evidence, and it is the capability audio-deepfake describes the consequences of. Zero-shot TTS falls out of the same model as the degenerate case: infill everything.
Where it sits in the codec lineage
The architecture is neural-audio-codec as substrate — 4 RVQ codebooks of 2048 codes, the EnCodec design — with the model predicting tokens rather than waveform. What is specific to VoiceCraft is rearranging the token sequence so a causal transformer can condition on audio after the gap as well as before it, which is what editing requires and plain left-to-right decoding cannot do. Compare misotts’ 32 codebooks × 2048: VoiceCraft spends far less codebook depth, which is the usual editing-versus-fidelity trade.
A consent statement, which is the finding
audio-deepfake records a gap: the models are documented, the harms are documented in journalism and regulation, and the distribution layer — repos, model cards, catalogues — mostly ships cloning with no consent language at all. luxtts was flagged for exactly that, and the 36-model census free-voice-clone-list carried almost none.
VoiceCraft is the counter-example. The repository prohibits generating or editing anyone’s speech without their consent, naming political figures and celebrities specifically. It is a licence-and-README statement with no enforcement behind it, and the non-commercial licences (CC BY-NC-SA, Coqui CPML) are the only teeth. Still: the project with the strongest editing capability in this wiki is also the one that wrote the sentence down, which is the opposite of what the pattern predicted.
Note the licensing consequence for open-weight-tts: non-commercial weights are open to read and run, closed to build a product on — a third position beside the permissive (Apache/MIT) and restricted-use (OpenRAIL-M) licences already tracked there.
Currency
The README’s most recent improvement is dated March 2025, so this is a well-starred repository that has been quiet for over a year while the field moved. Treat the state-of-the-art claim as of its paper, not of today; tts-benchmarks holds no entry for it and no leaderboard here ranks editing at all.
Related
text-to-speech · voice-cloning · audio-deepfake · neural-audio-codec · encodec-paper · open-weight-tts · free-voice-clone-list · misotts · speech-audio-ai