Neural audio codec (discrete audio tokens)

The representation most modern text-to-speech models generate: instead of predicting a raw waveform, the model predicts discrete audio tokens from a learned neural codec, which a decoder turns into sound. This is what lets a transformer “do speech as language modeling.”

Residual vector quantization (RVQ)

The dominant scheme. Each audio frame is encoded as several codebook indices that sum (residually) to reconstruct the sound, rather than one index from a giant flat vocabulary. The payoff, per misotts: 32 codebooks × 2048 entries → ~10^105 addressable audio tokens “without adding parameters” — far more sonic range than scaling a single vocabulary would allow.

Where RVQ codecs came from

The scheme was assembled as a trainable, streamable codec by SoundStream (Zeghidour et al., Google, 2021 — soundstream-paper): “a fully convolutional encoder/decoder network and a residual vector quantizer, trained jointly end-to-end.” Two of its properties are why codec tokens became the substrate for audio language modeling. First, one model spans speech and music — a single SoundStream codes both, so the tokens aren’t a speech-only trick. Second, structured dropout on the quantizer layers lets one model run at any rate from 3 to 18 kbps with negligible loss versus fixed-rate models — the literal mechanism behind “stack more codebooks for more quality.” It beat classical codecs hard: 3 kbps SoundStream outperformed Opus at 12 kbps soundstream-paper.

Meta’s EnCodec (encodec-paper, Défossez et al., 2022) is the direct successor — the codec musicgen generates into. It kept the RVQ encoder-decoder and added a single multiscale spectrogram discriminator, a loss balancer (which “defines the fraction of the overall gradient” each loss contributes — RVQ codecs are notoriously hard to train), and a small transformer that entropy-codes the token stream for up to ~40% more compression while staying faster than real-time — the music-side echo of “language modeling over the codebook indices” encodec-paper.

Seen in the wild

• Mimi codes via RVQ — misotts (8B; an AR-over-time backbone + AR-over-depth decoder over the codebooks).
• EnCodec is the codec musicgen tokenizes audio into — the music branch’s instance of the same RVQ line encodec-paper.
• Dual-autoregressive + RVQ — fish-audio-s2-pro‘s architecture.
• Codec tokens underlie the broader “TTS on an LLM backbone” pattern (Llama-based orpheus, sesame-csm, Higgs Audio V2) — text and audio both become next-token prediction.

Cross-wiki note

RVQ here quantizes audio into tokens; the related idea of quantizing model weights to low precision is quantization (llm-providers-wiki). Same “vector quantization” family, different target — worth not conflating.

Related

text-to-speech · soundstream-paper · encodec-paper · misotts · fish-audio-s2-pro · orpheus · sesame-csm · musicgen · wavenet