A neural audio codec turns a sound into a short grid of small integers, and turns the grid back into sound. One second of speech becomes, for example, 75 columns of 8 integers each, and the decoded audio sounds close to the original. Those integers are the audio tokens that every speech and music language model in this chapter reads and writes. So the codec decides how long those models' sequences are, and what they can hear.

Why not give a language model the sound itself? A digital waveform is a list of samples, one reading of air pressure per tick of a clock (§35.1). At 24 kHz and 16 bits per sample, that is 24,000 symbols per second, each one of 65,536 possible values. Ten seconds of speech would be a sequence of 240,000 steps.

The mel spectrogram that text-to-speech models predict, a picture of the sound's energy over time and frequency (§37.1), is much shorter, about 86 frames per second. But each frame is 80 real numbers, not a token from a finite list. It has also lost the phase, so it needs a second network, a vocoder, to become sound (§37.2). And a model that outputs real numbers is trained with a regression loss, not with the next-token cross-entropy that makes language models work. For scale, the words of English speech fit in about 3 to 4 text tokens per second (Défossez et al. 2024).

A codec sits in between. It produces tokens from a finite vocabulary, tens to a few hundred per second, and its decoder turns them back into audio that a listener accepts as the original. This section follows four codecs, SoundStream (Zeghidour et al. 2021), EnCodec (Défossez et al. 2022), DAC (Kumar et al. 2023), and Mimi (Défossez et al. 2024). All four are autoencoders, networks trained to squeeze their input into a small code and rebuild it. In the middle of each sits a residual vector quantizer, a stack of small lookup tables in which each table describes what the tables before it missed.

Earlier chapters promised these tokens. §1.8 said that snapping audio frames to a learned list of codes turns them into tokens. §2.7 said that stacked codebooks let a language model read sound. §1.10 quoted EnCodec's bitrate, $75\times8\times10 = 6{,}000$ bits per second. Here we build the machine behind those statements and derive every number in it.

The section has three parts. The first builds the codec, counts the bits and tokens it spends, and shows why its quantizer must be residual. The second is about training. We make one model serve many bitrates, keep every code in use, balance the losses, and measure how many bits the codes really need. The third splits tokens into semantic tokens, which record what is said, and acoustic tokens, which record how it sounds, and surveys the low-frame-rate tokenizers of 2024 to 2026.

A codec is an encoder, a quantizer, and a decoder

A neural codec has three parts. The encoder shrinks the waveform in time. It reads $R$ samples and emits one vector, so its output is $R$ times shorter than its input. Each output vector is a frame, and $R$ is the total stride, also called the hop. The quantizer replaces each frame by a few integers. The decoder turns the integers back into a waveform.

The quantizer is built from vector quantizers. A vector quantizer keeps a fixed list of $K$ vectors, its codebook. It replaces each frame by the nearest entry and sends only that entry's index, one integer (§32.5). A codec uses $n_q$ codebooks, one per level, so each frame becomes $n_q$ integers. How the levels share the work is the subject of most of this section. Here is the precise statement, with every symbol.

In plain words, the codec shrinks the waveform $R$-fold in time, names each shrunken frame with $n_q$ small integers, and trains a decoder that hears the integers and gives back the waveform.

Here is one real input followed through every stage, with the shapes of EnCodec's released 24 kHz model at 6 kbps. Its encoder starts with a convolution of 32 channels. Each of its four downsampling blocks divides the length by its stride and doubles the channels. A two-layer LSTM, a recurrent network that carries a memory along the sequence (§3.3), then models longer context. A last convolution sets the frame width to $d = 128$.

The same arithmetic at the scale of a real reply shows what a language model faces.

The next two subsections look inside the encoder and the decoder. They ask how much audio each frame depends on, and how long a codec makes you wait. The quantizer comes after we count the bits.

How far one frame can hear

The encoder is a stack of 1-D convolutions, small filters slid along time (§0.8). A convolution with stride $s$ moves its filter $s$ samples at a time, so its output is $s$ times shorter. A convolution with dilation $\Delta$ spaces its filter taps $\Delta$ samples apart, so it sees farther with the same number of weights. All four codecs alternate strided convolutions with residual units, small blocks of dilated convolutions whose output is added back to their input. They differ in the strides, the sampling rate, and what follows the convolutions.

How much audio does one frame depend on? That stretch of input is the frame's receptive field. §0.8 computes it layer by layer. Say layer $l$ has kernel $k_l$, dilation $\Delta_l$, and stride $s_l$. The receptive field grows as $r_l = r_{l-1} + (k_l - 1)\Delta_l\,j_{l-1}$, starting from $r_0 = 1$. Here $j_l = j_{l-1}s_l$ is the jump, the distance in input samples between neighboring outputs of layer $l$, starting from $j_0 = 1$. The cell applies this to SoundStream's encoder. It lists only the layers that widen the view, since the kernel-1 convolutions inside the residual units add nothing.

The jump is 320 samples, the hop between frames. Each frame sees 5,314 samples, about 221 ms of past audio. That spans a syllable but not a sentence. It is one reason EnCodec adds an LSTM after the convolutions and Mimi adds Transformers.

The decoder mirrors the encoder. It uses transposed convolutions, which upsample, with the encoder's strides in reverse order, so it upsamples by the same $R$. In other words it is a vocoder whose input is learned frames instead of mel frames. It is trained the way §37.2 trains HiFi-GAN, with spectral reconstruction losses and with discriminators, networks that learn to tell real audio from decoded audio.

Streaming and latency

A codec is causal when every output depends only on input that has already arrived. A causal codec can run while the audio is still coming in. Its algorithmic latency is the delay that the architecture itself forces, ignoring compute time. The encoder must wait for a full hop of $R$ samples before it can emit a frame, and the decoder then emits $R$ samples for that frame.

The decoder needs one trick to stream. A transposed convolution of stride $s$ produces more than $s$ outputs per input step, because the outputs of neighboring steps overlap. In its streamable mode EnCodec pads every convolution on the left only. It sends out the first $s$ outputs of each step at once and keeps the overlap in memory until the next frame arrives. So it outputs 320 samples (13 ms) as soon as it has received 320 samples. Mimi works the same way at its own scale. After 80 ms of input it emits one frame, which decodes to 80 ms of audio. That one frame is the building block of the 160 ms latency floor of the full-duplex voice model of §37.5.

Bits per second and tokens per second

Everything a language model pays for follows from four numbers. They are the sampling rate, the total stride, the number of codebooks, and the size of each codebook. The bitrate is the number of bits per second that the codes of all frames take. Naming one entry out of $K$ costs $\log_2K$ bits, so a codebook of 1,024 entries costs 10 bits per frame.

In plain words, bits per second are frames per second times bits per frame, and bits per frame are codebooks times bits per codebook. The number of steps a model takes depends on whether it reads a frame's codes one by one or all at once. That choice is the main design question of §37.4.

The cell computes the table for the codecs of this section. Two of them have not appeared yet. The EnCodec inside the MusicGen music generator runs at 32 kHz with total stride 640 and four codebooks of 2,048 (Copet et al. 2023). X-codec2 runs at 16 kHz with stride 320 and a single codebook of 65,536 (Ye et al. 2025).

Two rows deserve a comment. DAC's frame rate is $44{,}100/512 = 86.13$ Hz, so nine 10-bit codebooks give 7,752 bits per second and 91.0 times compression. The DAC paper rounds the frame rate to 86 Hz and reports 91.16, and it summarizes the codec as about 90 times compression at 8 kbps. Mimi's $12.5\times8\times11 = 1{,}100$ bits per second is 349 times smaller than the raw samples.

The frame rate is the number that matters most for a language model. It sets the length of every sequence, and the attention cost of a Transformer grows with the square of that length (§4.7). An interviewer who asks you to compare two codecs wants tokens per second and steps per second, not only bits.

The widget puts these rates on one log scale. Pick a codec from the list, or move the stride, the number of codebooks, and the codebook size, and watch one factor of the product change at a time.

import numpy as np

def codec_rates(fs, strides, n_q, K, seconds):
    raise NotImplementedError

One codebook cannot hold 80 bits per frame

Now the quantizer. A codec at a useful bitrate needs many bits per frame, and a single codebook cannot supply them. Section III.C of the SoundStream paper makes the argument with real numbers.

An interviewer will hand you a codec's sampling rate, strides, and codebooks and ask for exactly this arithmetic. Its conclusion is that the bits of a frame must be split over several small tables. The question is how.

Refining one number in three stages

The phrase "applied to what the previous ones missed" is the whole idea. Here it is once by hand, in one dimension.

Three things in this example carry over to real codecs. First, each stage refines the estimate of the stage before it. Second, stage 3's best move was to add nothing, and it could, because its table contains 0. A stage whose table contains zero can never make the error worse. Third, every prefix of the code decodes on its own. Keep stages 1 and 2 and you have 0.75. Keep stage 1 alone and you have 0.5. That last property is what will let one codec serve many bitrates.

Residual vector quantization

Residual vector quantization, or RVQ, does the same thing with vectors. The first codebook quantizes the frame. The second quantizes what the first missed, the third what the first two missed, and so on. §32.6 defined it for image tokenizers and proved what it guarantees. Here is the definition in this section's notation. The residual entering level $q$ is $\mathbf{r}^{(q)}$. The codebook of level $q$ is $\mathbf{C}^{(q)} \in \R^{K\times d}$, with rows $\mathbf{c}^{(q)}_k$.

In plain words, each level picks the code nearest to what is left and subtracts it. The decoder adds up the chosen codes of the levels it received.

What does this guarantee? The answer uses k-means, which represents each cluster of points by one center. Its centers are fit by Lloyd's algorithm, which alternates between assigning every point to its nearest center and moving every center to the mean of its points. §7.4 teaches both in full and proves that the algorithm stops, and §35.5 uses them to turn speech frames into units.

The proofs are in §32.6, and the reasons fit in a sentence each. A zero code lets a level do nothing, so its best choice is at most as bad as doing nothing. In a k-means level each code is the mean of the residuals assigned to it, and a mean is never farther from its points, in total squared distance, than zero is. Real codebooks are trained together with the rest of the network, so (b) holds only approximately in practice.

What residual stages cost

SoundStream's $2^{80}$ is not a quirk of one design. Every single quantizer at a useful bitrate has the same problem, and residual stages are the cheapest way out. Let $r$ be the bits per frame, and split them evenly over $n_q$ levels, so $\log_2K = r/n_q$. SoundStream, EnCodec, DAC, and Mimi all give every level the same $K$.

In plain words, RVQ turns an exponential cost into a linear one. It pays with a small loss in distortion, for two reasons. Its reconstructions are a structured subset of what one free codebook could place, and it encodes greedily, one level at a time. Storage alone would push you toward absurd codebooks of 2 or 3 entries, so storage is not what sets $K$ in practice. The cell prints the storage at 80 bits per frame.

How much quality do deep stacks of tiny codebooks really lose? Codec papers answer with two measures that run through the rest of this section. ViSQOL is an objective metric. It compares the decoded audio with the original and predicts the 1 to 5 quality score a listener would give (Chinen et al. 2020). MUSHRA is a listening test. Listeners rate several versions of an excerpt from 0 to 100 against a known reference (ITU-R BS.1534). A hidden copy of the reference and a deliberately degraded version are among the versions, to check the listeners. §37.6 treats both properly.

Deep stacks of tiny codebooks train well and lose only a little. Yet practice settles on 1,024 to 2,048 codes per level, for two reasons. Each level is one more code per frame for the language model downstream to predict, and eighty levels would be eighty tokens per frame. And the table shows quality falling, slowly, as the same bits are spread over more and smaller levels.

Watching the levels work

Here is RVQ on synthetic encoder frames in $d = 8$ dimensions, with six levels of 16 codes. Each level is fit by k-means to the residuals of the levels before it. The helpers stay defined for the cells that follow. entropy_bits measures how evenly a level uses its codes, in bits, a quantity we return to when we look for unused codes.

The first level does by far the most work. It cuts the error tenfold, from 23.05 to 2.40, because it captures which of the five clusters a frame belongs to. After that each level removes 40 to 50 percent of what is left, down to 0.14 after six levels. Every level uses its 16 codes almost evenly, with entropy near the full 4 bits. Keep that coarse-to-fine ordering in mind, because the next idea makes it deliberate.

The widget runs the same search on 2-D latents drawn from five clusters, so each level's step can be seen. Press Add levels and watch each path take one big step and then smaller corrections.

Quantizer dropout trains every prefix

A codec trained with all its levels always on has a decoder that has only ever seen full-depth reconstructions. Ask it to decode the first two levels, and you feed it inputs unlike anything it was trained on. SoundStream's fix is to make the depth random during training, so the decoder sees every depth.

With the depth $n$ drawn uniformly, the training objective is the average of the losses at every depth,

In plain words, one decoder is trained for every prefix at once, so at run time the bitrate is just the depth you choose to send. A single SoundStream model trained this way serves 3 to 18 kbps and performs about as well as models trained separately at each bitrate, slightly worse at 3 kbps and slightly better at 9 and 12 kbps.

Why the levels come out coarse to fine

Look at which terms of that sum each codebook can affect. Level $q$'s code vector appears in $\hat{\mathbf{z}}^{(n)}$ only when $n \ge q$. So it influences $\mathcal{L}_q, \dots, \mathcal{L}_{N_q}$, which is $N_q - q + 1$ of the $N_q$ terms. Level 1 is in every term, including $\mathcal{L}_1$, where it is all the decoder gets. So the gradient pushes level 1 to carry whatever matters most for the loss on its own. Level $N_q$ appears in only one term, after every other level has contributed, so it can only add the finest detail.

Greedy residual encoding already leans this way, since each level sees a smaller residual. Quantizer dropout makes the decoder agree. The DAC authors describe the result as levels that go from the most significant bits to the least significant. This is why codec language models such as VALL-E (§37.4) generate level 1 one token at a time and then fill in the finer levels in parallel.

There is a cost. DAC found that dropout on every example lowers reconstruction quality at the full bitrate. Dropout with probability 0.5 keeps nearly all the low-bitrate benefit while closing most of the gap at full bitrate. SoundStream saw the other side. A model trained at 18 kbps without dropout still decoded lower bitrates reasonably, with a quality drop that grew as the bitrate fell.

What a decoder that never saw a prefix gets wrong

Here is why a full-depth decoder fails on a prefix. Think of the decoder as a function $g$ from its input to audio. All the frames that share one coarse code form that code's cell. Given only the code, the best possible output under a squared-error loss is the average target over the cell, $\mathbb{E}[\mathbf{x}\mid\hat{\mathbf{z}}^{(n)}]$. A decoder trained only at full depth has learned $g(\mathbf{z}) \approx \mathbf{x}$ for precise inputs. On a coarse input it outputs $g(\hat{\mathbf{z}}^{(n)})$, the target at the center of the cell. For a curved $g$ that is not the average over the cell.

The cell below makes this concrete. The "waveform" is a nonlinear function of the frame. The decoder is a fixed body of random features with a linear head fit by least squares, the closed-form solution for the best linear map. We fit the head three ways, on full-depth codes only, on a uniform random depth per example, and on DAC's half-and-half mixture. Then we score each head at every depth.

The widget shows the same two decoders on the 2-D latents of the previous widget, quantized by eight levels of 4 codes, with each latent mapped to a curved target. The pink decoder was trained on full depth only, so for a coarse code it outputs the target at the center of the code's cell. The green one is the best decoder for each depth. It outputs the average target over the cell, which is what training on every prefix aims for.

In an interview, quantizer dropout is the answer to "how does one codec serve many bitrates". The coarse-to-fine ordering is the answer to "why does a codec language model treat the first level specially".

import numpy as np

def quantizer_dropout(codes, codebooks, rng, p=1.0):
    B, N_q = codes.shape
    depths = np.full(B, N_q)                       # placeholder, no dropout at all
    recon = sum(codebooks[q][codes[:, q]] for q in range(N_q))
    return depths, recon

Keeping every code alive

A codebook entry that no frame ever picks is dead. It still costs a slot in the bitrate. So a codec whose levels use only half their codes is really running at fewer bits than it claims. This failure is the codebook collapse of §32.5.

The honest way to measure it is the entropy of a level's code usage (§0.5). If code $k$ is chosen with frequency $p_k$, the entropy $H = -\sum_kp_k\log_2p_k$ is the average number of bits the level really carries. It is at most $\log_2K$, reached when all codes are used equally. The perplexity $2^{H}$ is the number of codes the level effectively has. For example, a level of 1,024 codes with perplexity 256 carries 8 bits, not 10. DAC's Figure 1 plots this entropy for each level over a large test set. It finds that a model trained with the usual moving-average codebook updates still under-uses its codes.

Codecs have used three kinds of fixes.

Initialization and restarts. SoundStream starts each codebook from k-means on the first training batch, so the codes begin near the data. It then tracks how many frames each code receives with an exponential moving average, a running average that weights recent batches most, with decay 0.99. Any code whose average falls below 2 is replaced by a frame drawn at random from the current batch. EnCodec follows the same recipe, with moving-average codebook updates (decay 0.99) and replacement of unused entries.

Factorized and normalized codes. DAC borrows two ideas from improved VQGAN image tokenizers (Yu et al. 2021). It looks codes up in a low-dimensional space, which is the factorized part. And it compares only directions, not lengths, which is the normalized part.

Why does this keep codes alive? Write $\hat{\mathbf{u}}$ and $\hat{\mathbf{e}}$ for the normalized projection and code. For unit vectors, $\lVert\hat{\mathbf{u}} - \hat{\mathbf{e}}\rVert^2 = 2 - 2\cos(\mathbf{u}, \mathbf{e})$, so the lookup picks the code with the largest cosine. That removes what kills codes in plain Euclidean VQ. There, a code that starts far from the data in norm is never the nearest one, so it never receives a gradient and never moves. Under cosine the norm no longer matters, only the direction. And in a low-dimensional space the frames' directions cover the unit sphere densely, so few codes point where no frame does.

The low dimension $M$ helps too, because the codes have less space to cover. The DAC authors read the projection as a lookup on the few directions that explain most of the variance, and found $M = 8$ best. With both tricks DAC trains with the plain VQ-VAE losses and needs neither k-means initialization nor restarts.

No codebook at all. Finite scalar quantization, or FSQ (Mentzer et al. 2023, recapped in §32.6), projects a frame to a few dimensions and rounds each one to a fixed grid of integers. The code is the tuple of rounded values. Every code exists by construction, so none can die.

In practice, dead codes are the first thing to check when a codec underperforms, and the check is cheap. An interviewer who asks "how do you know your codebook is healthy" wants the per-level usage entropy, compared with $\log_2K$.

import numpy as np

def factorized_l2_vq(z, W_down, codebook):
    u = z @ W_down
    codes = ((u[:, None, :] - codebook[None])**2).sum(-1).argmin(1)   # Euclidean, unnormalized
    return codes, 1.0

Training losses and EnCodec's balancer

The generator of a codec is the encoder, quantizer, and decoder together. It is trained with the vocoder losses of §37.2 plus a quantizer term. Its discriminators are trained against it. There are four kinds of generator loss.

• Reconstruction. Distances between the input and the output, on the waveform and on spectrograms at several window sizes.
• Adversarial. A reward for fooling the discriminators.
• Feature matching. A distance between the discriminators' internal activations on real and on decoded audio.
• Commitment. A pull of each residual toward the code vector chosen for it.

That is four or five losses of wildly different scales. The adversarial ones also change scale during training as the discriminators improve. Tuning their weights by hand is miserable, because a weight of 3 on one loss means nothing until you know how big its gradient is. EnCodec's balancer fixes this. It rescales each loss's gradient to a fixed size before adding them, so each weight says what share of the push that loss gets.

In plain words, each loss's gradient is first divided by its own typical size, so that it has norm about 1. It is then multiplied by its share $\lambda_i/\sum_j\lambda_j$ of a fixed budget $R_{\text{ref}}$. Two losses of very different scales show what that buys.

The example is one case of two general facts. Multiplying a loss by a constant changes nothing, because the balancer divides the constant out again. And the weights become shares of the gradient's norm at the output.

So each loss gets a fixed share of the gradient budget at the output, whatever its natural scale. With EnCodec's weights, $\lambda_g = 3$ and $\lambda_t = 0.1$ mean that the adversarial loss pushes 30 times harder than the time-domain loss. The shares are of the sum of the norms, not of the norm of the sum, since the parts point in different directions.

Two cautions. The balanced gradient is no longer the gradient of any fixed weighted loss, so the balancer changes the optimization problem, not just its scaling. And it applies only to the generator's losses. DAC, for comparison, uses fixed weights (15 for the mel loss, 2 for feature matching, 1 for the adversarial loss) and no balancer. The ablation in EnCodec's appendix reports that the balancer markedly stabilizes training, though it was run on music only and with a different quantizer.

import numpy as np

def balance(grads, lambdas, state, beta=0.999, R=1.0):
    return sum(l * g for l, g in zip(lambdas, grads)), np.array(state, dtype=float)

How many bits the codes really need

A level with 1,024 codes is charged 10 bits per frame. But its codes are not equally likely. They also depend on the frames before them, because speech changes slowly compared with a 13 ms frame. An entropy coder turns both facts into savings. It gives short bit strings to likely codes and long ones to unlikely codes.

§0.5 gave the rule. An arithmetic coder turns a model's predicted probabilities into bits, and it sends a code with probability $p$ under the model in about $-\log_2p$ bits. So the average length is the cross-entropy of the model against the true distribution, which is at least the true entropy. The theorem applies that rule to a codec's code stream.

In plain words, a model that predicts each level from the past saves what the codes waste in two ways, through uneven usage and through predictability over time. It gives up only the dependence between the levels of one frame. EnCodec does exactly this. A small Transformer (5 layers, 8 heads, 200 channels) predicts all codebooks of frame $t$ in parallel from the earlier frames. It drives a range coder, a fast form of arithmetic coding. Predicting the levels in parallel ignores the information they share inside a frame, in exchange for speed. It cuts the bandwidth by about 25 to 40 percent, for example from 3 kbps to 1.9 kbps, with smaller savings at high bitrates.

The same quantity has a second meaning that matters more for this chapter. $H(\mathbf{c}_t\mid\mathbf{c}_{<t})$, in bits per frame, is the lowest loss that any unconditional language model over these codes can reach. A model that also reads the text can go lower. A codec with a lower entropy rate gives the generator of §37.4 less to predict per second. That is one reason better reconstruction does not automatically mean better generation.

The cell builds a sticky code stream. Each frame repeats the previous code with probability 0.9, and otherwise jumps to a uniformly random one. The cell measures both entropies.

The codes are used evenly, so the marginal entropy is the full 4 bits, and a coder that ignores time saves nothing. Given the previous code the entropy is about 0.8 bits, a fivefold saving. Real codec levels sit between these extremes, and EnCodec's measured 25 to 40 percent tells you roughly where.

import numpy as np

def ideal_bitrate(codes, K, frame_rate, context="none"):
    n_q = codes.shape[1]
    H = np.full(n_q, np.log2(K))                   # placeholder, the nominal bits per level
    return H, frame_rate * H.sum()

Semantic and acoustic tokens

A codec is trained to reconstruct sound, so its codes carry everything audible together. The words, the voice, the room, and the microphone are all mixed into the same integers. That is ideal for reconstruction and awkward for a language model. The model must keep a sentence coherent over hundreds of frames while also getting the hiss of the room right. AudioLM (Borsos et al. 2022) made this distinction precise and measured it.

How do you measure "linguistic content" in a token? Not by reconstruction, which rewards everything audible. The standard test asks whether the representation keeps apart sounds that differ by a single phone, the smallest unit of speech sound. Take two recordings of one sound, $A$ and $X$, and one recording of a different sound, $B$. A good representation puts $X$ closer to $A$ than to $B$. The ABX error is how often it fails.

The across-speaker condition is the telling one. There $X$ has a different voice from $A$ and $B$. A representation that mostly encodes the voice puts $X$ far from both, and the small phone difference between them is lost in that distance.

The semantic rows come from w2v-BERT (Chung et al. 2021), a self-supervised speech encoder trained to predict masked parts of its input (§35.5), with its features clustered by k-means. We can reproduce the pattern of the table with synthetic features. Each token is a weighted phone vector plus a weighted speaker vector plus noise. Semantic-like features give the phone full weight and the speaker a small one. Acoustic-like features weaken the phone and give the speaker a large weight, as a codec does when it spends its bits on the voice. We estimate the ABX error from random triples.

The semantic-like features score about 7 percent in both conditions, because the speaker barely moves them. The acoustic-like features score about 21 percent within a speaker. There the strong voice component is shared by $A$, $B$, and $X$, and it only dilutes the phone contrast. Across speakers they score about 32 percent, because $X$'s different voice pulls it away from both. That is the shape of AudioLM's table.

import numpy as np

def abx_error(emb, phone, speaker, mode="within"):
    raise NotImplementedError

Four ways to make semantic tokens

The table says a generator needs semantic structure to stay coherent and acoustic detail to sound real. There are four ways to get tokens with semantic structure, in roughly historical order. Two of them use distillation, which trains one network's output to imitate the features of another, already trained network, the teacher.

1. Cluster a layer of a self-supervised model. Run a self-supervised speech encoder, cluster the frames of one of its middle layers with k-means, and use the cluster indices as tokens. HuBERT builds its training targets this way (§35.5). Generative spoken language modeling (Lakhotia et al. 2021) did this with three encoders (CPC, wav2vec 2.0, and HuBERT) and 50, 100, or 200 units. It found that the best number of units depends on the encoder and the task. AudioLM uses a 0.6 B-parameter w2v-BERT. It normalizes each dimension of the 7th layer of the model's masked-language-model module to zero mean and unit variance, which it found important, and runs k-means with $K = 1{,}024$. That runs at 25 Hz, so 250 bits per second. These tokens need a separate acoustic codec. AudioLM generates semantic tokens first and acoustic tokens conditioned on them, the hierarchy of §37.4.

2. Distill a teacher into the first level. SpeechTokenizer (Zhang et al. 2023) is a codec of EnCodec's design at 16 kHz and 50 Hz, with 8 levels of 1,024 codes. During training, the output of its first RVQ level is pulled toward the layer-9 features of HuBERT (or their average over layers), a model trained to capture phonetic content. The levels above level 1 then carry what the first one leaves out, mostly the voice.

3. A split quantizer. Mimi distills too, from WavLM (Chen et al. 2021), a self-supervised model of the HuBERT family (§35.5) that turns 16 kHz audio into 1,024-dimensional features at 50 Hz. To reach 12.5 Hz the targets are average-pooled with kernel 8 and stride 4. The pooling looks ahead in time, which is fine, because the targets are needed only in training. The loss is a cosine distance between a projection of the semantic quantizer's output and the pooled WavLM features. Without distillation, the first level of Mimi had an ABX error of 23.3 percent, about as poor as the acoustic tokens in AudioLM's table.

Distilling into level 1 of a single 8-level RVQ hurt audio quality, and the Moshi report explains why. Every later level quantizes the residual of the first. So the first level must keep acoustic information for the others to refine, which conflicts with being phonetic. The fix is a split RVQ. A plain VQ produces the semantic token, and a separate 7-level RVQ quantizes the same latent in parallel. Their outputs are added before the decoder, so the acoustic levels no longer depend on the semantic level's residual. In Mimi's ablation the split raised MUSHRA from 57.8 to 64.0, while the ABX error moved from 6.5 to 8.1 percent, a better trade overall.

4. Supervised tokens from a speech recognizer. CosyVoice (Du et al. 2024) inserts a vector quantizer into the encoder of a speech-recognition model (its SenseVoice model) and trains it with the recognition loss. The tokens keep exactly what recognition needs and little else. CosyVoice 2 (Du et al. 2024) replaces the VQ with FSQ, at 25 tokens per second with 6,561 codes. These tokens are so stripped of acoustics that CosyVoice does not decode them with a codec decoder. A conditional flow-matching model (§37.1), which turns noise into a mel spectrogram step by step, makes the spectrogram from them. It takes the voice from a speaker embedding and a prompt. A vocoder then makes the waveform, HiFi-GAN in the first CosyVoice.

Mimi and the low-frame-rate tokenizers

Put the pieces together and you have Mimi, the codec of the Moshi dialogue model.

• A causal convolutional encoder takes 24 kHz audio to 12.5 Hz frames of width $d = 512$, and a causal Transformer models the latent.
• A split RVQ produces one WavLM-distilled semantic code and seven acoustic codes per frame. Each codebook has 2,048 entries, after a projection from 512 to 256 dimensions.
• A second Transformer and a mirrored convolutional decoder turn the codes back into audio.
• It is trained with quantizer dropout and, in its best variant, with adversarial and feature-matching losses only. For half of the training sequences it skips quantization and passes the unquantized latent to the decoder, a variant of DAC's half-time dropout. That improved objective scores, while the listening tests were inconclusive.

The result is 1,100 bits per second, 100 flattened tokens per second, and a frame every 80 ms.

The tokenizers built for language-model text to speech from 2024 to 2026 all push the same two levers. They take fewer steps per second, and they make the first level more semantic. The table shows where they sit. Beyond the codecs above, it lists WavTokenizer (Ji et al. 2024), BiCodec from Spark-TTS (Wang et al. 2025), VibeVoice's tokenizer (Peng et al. 2025), Qwen-TTS-Tokenizer-12Hz from Qwen3-TTS (Hu et al. 2026), and MOSS-Audio-Tokenizer from MOSS-TTS (Gong et al. 2026). Only values that the papers or the released models state are given, and computed ones are marked. "n.s." means not stated, and the year is that of each first preprint.

Read the table from top to bottom and two trends stand out. The multi-level codecs of 2021 to 2023 ran at 75 to 86 Hz, and those of 2024 to 2026 at 12.5 Hz, with VibeVoice's continuous latent at 7.5 Hz. And the first level stopped being the coarsest acoustic approximation and became a semantic token. Three design families coexist.

• Multi-level codecs with a semantic first level (Mimi, Qwen-TTS-Tokenizer-12Hz) keep enough acoustic levels to decode directly. They need a model that produces a whole frame of codes per step.
• Single-codebook tokenizers (X-codec2, WavTokenizer, BiCodec) let an ordinary text language model generate one token per step. The price is a bigger vocabulary. BiCodec also adds a fixed-length set of global tokens that describe the speaker.
• Continuous latents (VibeVoice, whose tokenizer downsamples 24 kHz by 3,200) avoid quantization altogether. They generate each frame with a diffusion head, a small network that turns noise into the frame's vector, instead of a softmax over codes. §37.4 takes this up.

Neural codecs against classic codecs

Neural codecs also beat the standards at their own job. Two hand-engineered codecs carry most of today's calls. Opus is the open codec of internet voice and video (RFC 6716), and EVS is the codec of mobile telephony standardized by 3GPP.

The catch is compute. A neural decoder has millions to tens of millions of parameters (DAC's has 54 M), far more arithmetic per second of audio than a classic codec. That is why SoundStream reported its real-time speed on a single thread of a smartphone CPU, and EnCodec on a single CPU core.

In PyTorch

The block builds a codec small enough to read in one sitting.

• Strides $(2, 4, 5)$, so $R = 40$, with causal convolutions.
• A causal transposed convolution. It drops the last $s$ outputs, so no output waits for a later frame, as in EnCodec's streaming mode.
• A 4-level RVQ trained with the VQ-VAE codebook and commitment losses. The straight-through estimator copies the gradient past the nearest-code lookup, which has no gradient of its own (§32.5).
• SoundStream's quantizer dropout, with one depth per example.

It runs one training step and checks the shapes, the gradient path through the quantizer, prefix decoding, and causality.

The codes come out with shape (B, n_q, frames), the layout every library uses. Real codecs differ from this one in scale and in their losses, not in this skeleton. With Hugging Face Transformers, the pretrained EnCodec and Mimi models encode and decode as below (not run here, since it downloads weights).

The bandwidth argument of EnCodec chooses the depth of the RVQ prefix, and num_quantizers does the same for Mimi. Mimi's released checkpoint has more levels than the 8 that Moshi uses. Both arguments are quantizer dropout paying off at inference time.

Choosing a tokenizer, and where codecs break

The fairest way to compare tokenizers for generation is at equal token rate. The paper that introduced X-codec2 and the Llasa speech models (Ye et al. 2025) does exactly that. It resynthesizes test speech through about twenty codec configurations and transcribes the result.

Codecs break in predictable ways.

• Better reconstruction is not better generation. This is the lesson of §32.6 again. A codec with more levels or a higher frame rate reconstructs better and gives the generator more to predict. Divide a model's context length by the token rate and you get the seconds of audio it can produce.
• Low frame rates pack more into each step. The LJSpeech reader of §37.1 speaks about 15 characters per second. So a single 80 ms Mimi frame covers more than one character on average, and the semantic level and the generator must carry more per step.
• Domain. A codec trained on speech degrades on music and noise, and the reverse. That is why DAC balances speech, music, and environmental sound in every batch.
• Codebook collapse at scale silently lowers the effective bitrate below the nominal one. Measure per-level usage entropy, as DAC does.
• Semantic teachers carry their training data. WavLM and HuBERT were trained mostly on English speech, so tokens distilled from them may be less phonetic for other languages. Treat this as a reason to measure ABX in your language, not as a known result.
• Streaming needs causality end to end. EnCodec's non-streamable models normalize each one-second chunk with statistics over its whole length. So only causal designs, such as SoundStream, streamable EnCodec, Mimi, and Qwen-TTS-Tokenizer-12Hz, fit the real-time systems of §37.5.
• Watermarks must survive the codec. A watermark embedded in generated audio has to survive being re-encoded by exactly these codecs, which §37.6 tests.

Read more

• Zeghidour et al. (2021). SoundStream: An End-to-End Neural Audio Codec. The paper that introduced residual vector quantization and quantizer dropout to neural audio coding. Section III.C has the 80-bit argument and Table II the depth against codebook-size trade.
• Défossez et al. (2022). High Fidelity Neural Audio Compression. EnCodec, with its streaming design, multi-scale STFT discriminator, loss balancer (Section 3.4), and Transformer entropy coder.
• Kumar et al. (2023). High-Fidelity Audio Compression with Improved RVQGAN. DAC, which diagnoses codebook under-use with per-level entropy, fixes it with factorized and normalized codes, and finds the cost of quantizer dropout.
• Défossez et al. (2024). Moshi: a speech-text foundation model for real-time dialogue. Section 3.3 builds Mimi and its split RVQ, and Tables 3 and 4 are the best ablation of semantic distillation and of objective against subjective metrics.
• Borsos et al. (2022). AudioLM: a Language Modeling Approach to Audio Generation. Section III defines semantic and acoustic tokens and measures them with ABX and ViSQOL.
• Zhang et al. (2023). SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models. Distilling HuBERT into the first RVQ level, and a benchmark of how suitable tokens are for language modeling.
• Ye et al. (2025). Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis. X-codec2, a single 65,536-entry FSQ codebook at 50 Hz, and a comparison of codecs grouped by token rate.
• Jurafsky & Martin. Speech and Language Processing, chapter 17. A textbook treatment of audio tokenizers, VQ, RVQ, and EnCodec, building toward codec-based TTS.

Watch next

▶

▶

▶