Hot AI News

Neural audio codecs: How speech becomes tokens a language model can write

A neural audio codec compresses sound into short codes and rebuilds it. This is how it works, where it came from, and why modern speech models such as Zonos3-Mini depend on it.

Neural audio codecs: How speech becomes tokens a language model can write
Photo illustration: Aiholics

A neural audio codec is a pair of neural networks that shrink sound into a short list of numbers and then rebuild the sound from that list. The list is made of discrete tokens, much like the word pieces a chatbot reads. That is what lets a language-style model generate speech by predicting the next token.

The idea is old, but the neural version took off in 2021. It now sits under many open and commercial speech models.

Why a speech model needs to turn sound into tokens first

Think of sheet music. A pianist does not store the vibration of every string. The notation is a compact code, and a skilled player can turn it back into sound. A neural audio codec learns its own notation for audio.

Raw audio is huge. CD-style sound has 44,100 samples every second, far too many for a model to predict one by one over minutes of speech. A text model, by contrast, handles a few tokens per second of reading. The codec closes that gap by turning each second of sound into a handful of tokens.

Once audio is tokens, text-to-speech looks like translation: text tokens go in, audio tokens come out, and the decoder part of the codec turns them into a waveform.

Advertisements

An encoder, a bank of codebooks, and a decoder

A wall-mounted screen in an acoustic lab shows a speech waveform compressed into codebook selections and reconstructed as audio.
Photo illustration: Aiholics

A codec has three parts, trained together.

  • The encoder reads the waveform and produces one compact vector for each short slice of time, called a frame.
  • The quantizer replaces each vector with the closest entries from fixed lists called codebooks. The chosen entry numbers are the tokens.
  • The decoder takes the entries back and rebuilds the waveform.

Training rewards the decoder for sounding like the original. Both papers below also used an adversarial setup, in which a second network tries to tell real audio from rebuilt audio, which reduces audible artifacts.

One codebook is too coarse to describe a voice well. The standard fix is residual vector quantization. The first codebook picks its best match. The second codebook then encodes what the first one got wrong. A third encodes the remaining error, and so on. Each extra codebook adds detail, so a frame becomes a small stack of tokens.

Where the idea came from: SoundStream, EnCodec and AudioLM

In 2021, Neil Zeghidour and colleagues at Google published SoundStream, a codec built from a convolutional encoder and decoder plus a residual quantizer. The authors reported that one model could run from 3 kbps to 18 kbps (kilobits per second). It also ran in real time on a smartphone processor. At 3 kbps it beat the Opus codec at 12 kbps in their listening tests.

Randomly dropping some quantizer layers during training is what allowed that range of bitrates. A listener or app can use fewer codebooks for a smaller file or more for better quality.

In 2022, Alexandre Défossez and colleagues at Meta released EnCodec, an open codec for speech and music at 24 kHz mono and 48 kHz stereo. It added a loss balancer and a single multiscale spectrogram discriminator to simplify training. Its public code made codecs a common building block for other researchers.

Also in 2022, Zalán Borsos and colleagues at Google published AudioLM. It treated audio generation as language modeling over tokens and combined two kinds of them. Tokens from a pretrained masked language model carried long-term structure, such as what is being said. Tokens from a neural codec carried fine acoustic detail. Later systems often call these the semantic and acoustic tokens.

Advertisements

How Zonos3-Mini uses one

Zyphra’s open text-to-speech model, covered in our news report on Zonos3-Mini, is a clear current example. According to its model card, speech is handled as tokens from a codec running at 15 Hz and 44.1 kHz. It uses 10 codebooks of 4,096 entries each, one semantic and nine acoustic.

Those figures allow a rough check. Each entry among 4,096 needs 12 bits, so 15 frames per second times 10 codebooks times 12 bits is about 1,800 bits per second, or roughly 1.8 kbps. This is an estimate from the card’s numbers, not a figure Zyphra states. It is lower than the 3 kbps low end SoundStream reported in 2021 for a different design.

The frame rate also explains the long audio. At 15 frames per second, 10 minutes is about 9,000 frames, which fits comfortably in the card’s 32,768-position context. A codec running at 50 frames per second would need more than three times as many.

Limits worth knowing

Compression is lossy. At low bitrates a codec can smear fine detail, such as breath, room sound or singing, and the result depends on the training data. A speech model can only be as clear as its codec’s rebuilt audio.

Stacked codebooks also create a modeling problem. Each frame holds several tokens, so a model must predict them in some order or in parallel. Designs differ here, and the choice affects speed and quality. Zyphra has not described this in the material available for this article, so its approach is best checked in its technical report.

Token-based speech also raises the question of voice identity. A codec carries speaker traits, so cloning relies on how well those traits survive. Evaluation is usually vendor-run: Zyphra’s reported latency and quality scores come from its own benchmark and have not been independently reproduced in what we read.

Advertisements

What to watch next

  • Lower frame rates. Fewer frames per second mean shorter sequences and longer audio per request.
  • Semantic codebooks. More codecs now reserve one codebook for meaning, aiming for clearer speech from smaller models.
  • Open tooling. Codecs that run in a browser or on a laptop make local voice generation cheaper, and they raise consent and misuse questions for voice cloning.

Anyone reading a speech-model card can start with three numbers: frame rate, number of codebooks and codebook size. Together they set the bitrate, the sequence length, and much of the sound quality.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Daniel ReedAI Research, Safety & Ethics Analyst

Daniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like