SoundStream: An End-to-End Neural Audio Codec (Zeghidour et al., 2021)

The paper that established the RVQ neural audio codec the spoke’s TTS stack runs on — Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, Marco Tagliasacchi (Google), arXiv:2107.03312. The neural-audio-codec page describes residual vector quantization as “the dominant scheme”; this is where that scheme was first put together as a trainable, streamable codec — the ancestor of Mimi (misotts) and EnCodec (musicgen).

What it is

A codec that compresses speech, music, and general audio at bitrates traditionally reserved for speech-only codecs. Its design is the template every later RVQ codec reuses: “a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end,” combining adversarial and reconstruction losses.

The facts that matter for the spoke

• One model, all of audio. A single SoundStream model handles both speech and music across the whole bitrate range — the property that makes codec tokens a general substrate for audio language modeling, not a speech-only trick.
• Variable bitrate from one model. Trained with structured dropout on the quantizer layers, it runs at any rate from 3 kbps to 18 kbps “with a negligible quality loss when compared with models trained at fixed bitrates.” This is the RVQ payoff the neural-audio-codec page describes — stacking codebooks lets one model trade rate for quality without retraining.
• Real time on a phone. “Amenable to a low latency implementation, which supports streamable inference and runs in real time on a smartphone CPU” — the streaming property TTS decoders inherit.
• Beats classical codecs at far lower rate. “SoundStream at 3 kbps outperforms Opus at 12 kbps and approaches EVS at 9.6 kbps” — the headline that made neural codecs credible.

Tier

T1 — peer-reviewed primary (IEEE/ACM TASLP) from the codec’s authors; the canonical origin of the RVQ-codec line. Figures (3–18 kbps, the Opus/EVS comparison, smartphone real-time) from the abstract; noted where the abstract is the source.

Related

neural-audio-codec · encodec-paper · text-to-speech · misotts · musicgen · speech-audio-ai