High Fidelity Neural Audio Compression / EnCodec (Défossez et al., 2022)
Meta’s RVQ codec — Alexandre Défossez, Jade Copet, Gabriel Synnaeve, Yossi Adi, arXiv:2210.13438. The direct successor to SoundStream and the codec musicgen generates tokens into, so it sits one layer under the music branch of neural-audio-codec. Open code and models at github.com/facebookresearch/encodec.
What it adds over SoundStream
The same recipe — “a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion” using residual vector quantization — refined in three ways the neural-audio-codec page should note:
- A single multiscale spectrogram discriminator. One adversary over multiple spectrogram scales, “to efficiently reduce artifacts and produce high-quality samples” — a simpler, stronger adversarial setup than the multi-discriminator stacks before it.
- A loss balancer. A novel mechanism that “defines the fraction of the overall gradient” each loss term contributes — it stabilizes training across the many competing reconstruction/adversarial losses that make RVQ codecs hard to train.
- A transformer that compresses the tokens further. Compact Transformer models entropy-code the RVQ token stream for “up to 40%” extra compression while “remaining faster than real-time” — the audio analog of language-modeling over the codebook indices.
It evaluates on 24 kHz monophonic and 48 kHz stereophonic audio and reports being “superior to the baseline methods across all evaluated settings” in MUSHRA subjective tests, across speech, noisy-reverberant speech, and music.
Tier
T1 — peer-reviewed primary from the codec’s authors; the codec under Meta’s audio-gen line (musicgen). Figures (multiscale spectrogram adversary, ~40% transformer compression, 24/48 kHz) from the abstract; noted inline.
Related
neural-audio-codec · soundstream-paper · musicgen · text-to-speech · audio-music-generation