Chapter 37 · Section 37.4 · 95 min

Codec language models

A codec language model is a language model whose tokens are pieces of sound. It writes audio the way a text model writes a sentence. It predicts the next token from the ones before it, appends it, and repeats. The tokens come from a neural audio codec, a network that turns a waveform into a short grid of integers and turns the grid back into sound (§37.3). Models built this way clone a voice from three seconds of audio, read text aloud, compose music, and speak inside voice assistants.

Here is the grid in plain words. The codec cuts audio into short frames, $f_r$ of them per second. It describes each frame $t$ with $n_q$ integers called codes, $c_{t,1}, \dots, c_{t,n_q}$ , one per level $q$ . Each code names one of the $K$ vectors in its level's codebook, a learned table. The levels come from residual quantization. Level 1 picks the codebook vector closest to the encoder's vector for the frame. Level 2 picks the vector closest to what level 1 missed, and level 3 corrects what the first two missed. So level 1 is a coarse description of the frame, and each later level is a finer correction. The codec's decoder adds up the chosen vectors of each frame and turns the result back into a waveform.

The obvious plan is the one that worked for text. Write the codes out in one row, frame by frame, and within a frame from level 1 to level $n_q$ . Train a decoder-only Transformer, the model of Chapter 6, to predict the next code of the row. Then prompt it with a few seconds of audio and sample. Two things go wrong.

The first is length. EnCodec, a widely used codec (Défossez et al. 2022), runs at $f_r = 75$ frames per second with $n_q = 8$ levels in its 6 kbps setting. Its row holds $75 \times 8 = 600$ tokens per second, so a 30-second clip is 18,000 tokens, and the cost of attention grows with the square of the length (§4.7). The second problem is worse. A team at Google trained exactly this model while building AudioLM (Borsos et al. 2022) and prompted it with 4 seconds of speech. The continuations kept the speaker's voice and the recording conditions, but the words were babble.

So a codec language model must answer two questions that a text model never faces. What should it generate first? And in what order should it generate the $n_q$ codes of each frame? This section answers both. We build the hierarchy that writes the content before the sound, and voice cloning as the continuation of a prompt. We compare the orders in which a frame's codes can be generated, with an exact rule for when an order is correct, and we build the faster designs that real-time and production systems use. We end with what is specific to audio when sampling (loops that never end) and when post-training (rewards that are easy to game).

Speech changes slowly in what it says and fast in how it sounds

Speech carries two kinds of information, and they change at very different rates. What is said, the sequence of words and speech sounds, changes slowly. A reader of audiobooks says about 15 characters per second, the rate of the LJSpeech dataset of §37.1. English speech can be written with only 3 to 4 text tokens per second (Défossez et al. 2024). How it sounds, meaning the exact voice, the room, the breath, and the fine detail of the spectrum, changes fast and takes most of the bits. That is why EnCodec needs 600 tokens per second.

§37.3 gave each kind of information its own kind of token.

A model that predicts everything in one row spends almost all its steps on the fast part. Meanwhile the slow part, which a listener calls the content, must stay consistent across thousands of positions. The fix is to generate the slow part first and then fill in the fast part conditioned on it. Every design in this section is a version of that idea, and "why semantic tokens first?" is a standard interview question.

Four orders for the codes of a frame

The second question is the order of the codes inside a frame. The chain rule of probability allows any order, as long as each code is predicted from all the codes generated before it. A model may also write several codes in one step, which is faster, but then those codes cannot see each other. Here are the four orders that matter, on a grid small enough to draw.

✏️ A grid of 3 frames and 2 levels, four ways

Take $T = 3$ frames and $n_q = 2$ levels, six codes $c_{t,q}$ in all.

Flatten. One code per step, frame by frame, $c_{1,1}, c_{1,2}, c_{2,1}, c_{2,2}, c_{3,1}, c_{3,2}$ . Six steps. Each code sees every code before it, so sampling is exact.
Parallel. Step $t$ writes $c_{t,1}$ and $c_{t,2}$ together. Three steps. Level 2 cannot see level 1 of its own frame, although level 2 exists only to correct level 1.
Delay. Level 2 runs one step behind level 1. Step 1 writes $c_{1,1}$ . Step 2 writes $c_{2,1}$ and $c_{1,2}$ . Step 3 writes $c_{3,1}$ and $c_{2,2}$ . Step 4 writes $c_{3,2}$ . Four steps, and every level-2 code sees the level-1 code of its own frame.
Coarse first. Steps 1 to 3 write level 1 of each frame, and steps 4 to 6 write level 2 of each frame. Six steps. With more levels, steps 4 to 6 would write all the finer codes of a frame together.

Parallel saves steps by ignoring the strongest dependence in the grid, the one between a correction and what it corrects. Delay saves almost as many steps and ignores a much weaker one, between neighboring frames. A theorem later in the section makes "ignores" exact.

The cell counts the steps for real codecs and clip lengths. The frames column is also the number of steps of any model that writes a whole frame at once. The delay column is the frames plus $n_q - 1$ . The codecs are EnCodec at 24 kHz, the 32 kHz EnCodec of the music model MusicGen, SoundStream (the codec of AudioLM), and Mimi, the 12.5 Hz codec of the Moshi dialogue model (§37.3).

Python · cell 1

Flattening five minutes of Mimi codes takes 30,000 steps, and the delay order takes 3,757. Each sequential step is one more forward pass that the listener waits for, which is why the order of the codes is a design decision and not a detail.

Phonetic units alone lose the voice

An early language model of speech threw the fast part away entirely. Clustering the features of a self-supervised speech model gives discrete units that track phones (§35.5). Generative spoken language modeling built a whole pipeline on such units, with no text anywhere (Lakhotia et al. 2021).

📐 Generative spoken language model (GSLM)

A GSLM has three trained parts and uses no text.

A speech encoder maps audio to a sequence of discrete units. It runs a self-supervised model (CPC, wav2vec 2.0, or HuBERT, from §35.4 and §35.5) and assigns each feature frame to the nearest of $K$ k-means centroids, with $K \in \{50, 100, 200\}$ in the original study. Consecutive repeats of a unit are usually merged.
A unit language model, a Transformer trained by next-token cross-entropy on unit sequences, generates new units or continues a prompt.
A unit-to-speech decoder turns units back into a waveform. The original used a Tacotron 2 style model (§37.1) that reads units instead of letters and writes a mel spectrogram, followed by a WaveGlow vocoder (§37.2).

GSLM produces speech-like continuations from raw audio alone, which was a real result. But its units were chosen to be phonetic, and a phonetic code is by design almost independent of who is speaking. So the decoder has to invent a voice.

The way to measure this is speaker similarity. A speaker-verification model maps each recording to an embedding vector, and the similarity of two recordings is the cosine of their embeddings (§35.6, made precise in §37.6). Two different real recordings of one speaker score 0.754 with the verification model the VALL-E authors used. GSLM's output scored 0.126 against its prompt (Wang et al. 2023, Table 2). A follow-up restored part of the identity by adding a separate low-rate code for pitch and a vector for the speaker. Its authors noted that the three streams together form a codec at 365 bits per second (Polyak et al. 2021).

The lesson is the one AudioLM drew. Semantic units alone lose the voice, and acoustic tokens alone lose the words. A model needs both, with the content first.

AudioLM writes the content first, then the coarse sound, then the fine sound

AudioLM (Borsos et al. 2022) runs two tokenizers on the same 16 kHz audio.

Its semantic tokens $\mathbf{z}^{\text{sem}}$ come from w2v-BERT, a self-supervised speech model trained to predict masked parts of its input (§35.5). AudioLM clusters the features of the seventh layer of its masked-prediction module into $K = 1{,}024$ k-means clusters, at 25 tokens per second.
Its acoustic tokens $\mathbf{z}^{\text{ac}}$ come from SoundStream (Zeghidour et al. 2021), the first codec of the kind in §37.3. Its encoder downsamples by strides $(2, 4, 5, 8)$ , so one frame is $2 \cdot 4 \cdot 5 \cdot 8 = 320$ samples and there are $16{,}000/320 = 50$ frames per second. Each frame has 12 levels of 1,024 codes, which is 600 tokens and 6,000 bits per second.

Two measurements from §37.3 justify having two tokenizers. The ABX error is the chance that a recording of one phone lands closer to a recording of a different phone than to another recording of its own, so lower means more phonetic. ViSQOL estimates perceived quality on a 1 to 5 scale by comparing the rebuilt audio with the original. In AudioLM's Table I the semantic tokens, at 250 bits per second, have ABX errors of 6.7 percent within speakers and 7.6 percent across speakers, but audio rebuilt from them scores a ViSQOL of only 1.1. SoundStream's codes at 2 kbps rebuild to 3.3 and have ABX errors of 22.4 and 28.7 percent. Neither token type is good at both jobs.

So AudioLM generates both, one after the other. Write $z^{\text{sem}}_t$ for semantic token $t$ and $z^{\text{ac}}_{t,q}$ for the acoustic code of frame $t$ at level $q$ .

📐 AudioLM's three stages

AudioLM factorizes the joint distribution of the two token sequences into three autoregressive models, each a decoder-only Transformer with 12 layers, 16 heads, and width 1,024, about 0.3 B parameters.

Semantic modeling, $p(\mathbf{z}^{\text{sem}}) = \prod_t p(z^{\text{sem}}_t \mid z^{\text{sem}}_{<t})$ .
Coarse acoustic modeling. The first 4 levels are flattened frame by frame and modeled as $p(z^{\text{ac}}_{t,q} \mid \mathbf{z}^{\text{sem}}, \text{coarse codes before } (t,q))$ for $q \le 4$ , conditioned on the entire semantic sequence.
Fine acoustic modeling. Levels 5 to 12 are flattened and modeled as $p(z^{\text{ac}}_{t,q} \mid \text{coarse codes of the chunk}, \text{fine codes before } (t,q))$ for $q > 4$ , on non-overlapping 3-second chunks.

The factorization rests on a conditional independence assumption. Semantic tokens do not depend on past acoustic tokens once past semantic tokens are known,

p(z^{\text{sem}}_t \mid z^{\text{sem}}_{<t}, \mathbf{z}^{\text{ac}}_{<t}) \approx p(z^{\text{sem}}_t \mid z^{\text{sem}}_{<t}).

Stage 3 assumes in the same way that fine codes need no semantic tokens once the coarse codes are known. In a flattened stream the codes of level $q$ are offset by $(q - 1)K$ , so codes of different levels never share an id.

In plain words, first decide what is said, as a slow sequence of phone-like tokens. Then decide how it sounds, first the coarse acoustics conditioned on the content, then the fine acoustics conditioned on the coarse ones. Each assumption says that what a stage needs was already summarized by the stage before it.

The first assumption is reasonable because of what the tokens mean. Which phone comes next depends on the words so far, not on the exact timbre they were spoken with. The second is the residual hierarchy again. Levels 5 to 12 encode corrections of corrections, local detail that the coarse codes have already placed.

The stages also cut the sequence lengths. Stage 1 runs at 25 tokens per second, and stage 2 at $4 \times 50 = 200$ . Stage 3 runs at $8 \times 50 = 400$ , but only inside 3-second chunks. AudioLM samples with temperatures 0.6, 0.8, and 0.6 for the three stages. Prompted with 3 seconds of an unseen speaker, it continues in that speaker's voice with coherent content, which GSLM could not do.

A text-to-speech system needs only one more piece, a model from text to semantic tokens. SPEAR-TTS (Kharitonov et al. 2023) added exactly that. It splits text to speech into "reading", from text to semantic tokens, and "speaking", from semantic to acoustic tokens. The speaking stage needs only audio to train. With this split, SPEAR-TTS reported a character error rate competitive with the best systems of its time from only 15 minutes of paired text and speech. VALL-E, next, took the other route and skipped semantic tokens entirely.

VALL-E clones a voice by continuing a prompt

VALL-E (Wang et al. 2023) treats text to speech as language modeling over EnCodec codes at 6 kbps, with 24 kHz audio, $f_r = 75$ , and $n_q = 8$ levels of $K = 1{,}024$ codes. It is a zero-shot system, which means it speaks in a voice it never saw in training, with no fine-tuning. It conditions on two things. One is the phonemes $\mathbf{x}$ of the text, the symbols for its speech sounds that the front end of §37.1 produces. The other is an acoustic prompt, the codes $\tilde{\mathbf{C}} \in [K]^{T'\times 8}$ of a short recording of the target voice, $T'$ frames long, where $[K]$ is the set of $K$ code values. From them it generates the code matrix $\mathbf{C} \in [K]^{T\times 8}$ of the new utterance. Column $\mathbf{c}_{:,q}$ of the matrix is level $q$ over time.

VALL-E uses two kinds of model. An autoregressive (AR) model writes one position per step, each conditioned on the positions before it. A non-autoregressive (NAR) model writes many positions in one step.

📐 VALL-E

VALL-E factorizes the conditional distribution into an AR model over level 1 and an NAR model for each finer level,

p(\mathbf{C} \mid \mathbf{x}, \tilde{\mathbf{C}}) \;=\; \underbrace{\prod_{t=1}^{T} p\big(c_{t,1} \mid c_{<t,1},\, \tilde{\mathbf{c}}_{:,1},\, \mathbf{x};\, \theta_{\text{AR}}\big)}_{\text{AR, level 1}} \;\cdot\; \underbrace{\prod_{q=2}^{8} p\big(\mathbf{c}_{:,q} \mid \mathbf{c}_{:,<q},\, \mathbf{x},\, \tilde{\mathbf{C}};\, \theta_{\text{NAR}}\big)}_{\text{NAR, one pass per level}}.

The AR model is a causal Transformer over the sequence [phonemes, end of text, level-1 codes, end of audio]. Phonemes and codes have separate embedding tables, and each part ends with its own end token. It is trained by next-token cross-entropy on the audio positions. At inference the phonemes are the prompt's transcript followed by the new text, and the prompt's level-1 codes are a prefix that the model continues.
The NAR model is a Transformer with bidirectional attention, so every position sees every other. To predict level $q$ it reads three things. They are the phonemes, the prompt with the embeddings of all 8 of its levels summed per frame, and the target frames with the embeddings of levels $1$ to $q-1$ summed per frame. Each level has its own embedding table. The level being predicted enters through adaptive layer normalization, a layer norm whose scale and shift are computed from an embedding of $q$ . The model predicts all $T$ codes of level $q$ at once.

In plain words, the AR model writes the coarse code sequence one frame at a time, and with it the timing and the length of the utterance. The NAR model then fills in the seven finer levels, one whole level per pass. Each pass looks at everything coarser and at the full prompt. Both models see the prompt's codes, and that is how the voice gets in.

The split follows the codec. Level 1 carries most of the information, including much of the speaker identity. It is also the only level whose length must be decided. An AR model decides length naturally, by emitting an end token. A separate length predictor would have to know how fast each unseen speaker talks. The finer levels inherit their length from level 1, so they can be predicted in parallel. The authors note that this takes the cost of the remaining levels from $O(T)$ steps to $O(1)$ .

Training is plain language modeling. VALL-E was trained on LibriLight, 60,000 hours of English audiobooks from about 7,000 speakers, transcribed into phonemes by a speech recognizer, in random crops of 10 to 20 seconds. The AR model is never shown a separate prompt in training. Every prefix of a training sequence is a prompt for the rest of it. This is in-context learning in the sense of §6.8, where a model infers a task from its context and continues it with no weight update. Here the prompt is evidence about the speaker and the channel, and the best continuation is the one that fits that evidence. The NAR model's prompt in training is a random 3-second segment of the same utterance.

✏️ Cloning a voice from 3 seconds

Clone a voice from a 3-second recording and speak a 10-second sentence. At 75 frames per second the prompt is $3 \times 75 = 225$ level-1 codes and the target is $10 \times 75 = 750$ .

The AR model's input is the prompt transcript's phonemes, the new phonemes, the end-of-text token, and the 225 prompt codes.
It then samples 750 codes plus the end token, 751 sequential steps. Each step attends to the phonemes of both sentences and up to $225 + 750$ codes.
The NAR model runs 7 passes, one per level, each over $225 + 750$ frames.

Generating all $750 \times 8 = 6{,}000$ target codes in the flattened order would take 6,001 steps, counting the end token, instead of $751 + 7 = 758$ .

Decoding matters as much as modeling. VALL-E samples the AR stage, because the authors saw beam search drive the model into an infinite loop, and because sampling gives varied outputs for the same input. The NAR stages are greedy, taking the most probable code at each frame. Loops come back later in the section.

To read the results you need one more metric. The word error rate (WER) runs a speech recognizer on the generated audio and counts the words it gets wrong against the input text, the substitutions, deletions, and insertions, divided by the number of words in the text (§36.1). Lower is better.

🔬 VALL-E 2 and what human parity meant

VALL-E 2 (Chen et al. 2024) kept the architecture and changed two things. Grouped code modeling has the AR model emit a group of consecutive level-1 codes per step, which shortens the sequence by the group size. The authors tested group sizes 1, 2, 4, and 8. A group size of 2 halved the steps and gave a slightly better objective error rate, while group size 1 scored best with listeners. Repetition-aware sampling, defined later in this section, stopped the loops. VALL-E 2 was trained on Libriheavy, a 50,000-hour version of LibriLight with transcripts.

Its authors called it the first zero-shot TTS system at human parity on LibriSpeech and VCTK. The version that reached parity used group size 1. They defined parity as three differences against the ground-truth recordings in their evaluation, a better word error rate, a better CMOS (a side-by-side naturalness score from listeners), and a better SMOS (a similarity score from listeners). Whether such differences are real, and what parity should mean statistically, is a question of listening-test design that §37.6 takes up. The authors also called VALL-E 2 purely a research project with no plans for public access, citing the risk of spoofing and impersonation.

Core

VALL-E's training sequences and loss masks

VALL-E's AR model reads phonemes and audio codes. The paper gives each its own embedding table. An equivalent layout, used here, is one shared vocabulary in which the two ranges of ids do not overlap. Phoneme ids run from $0$ to $V_{\text{text}} - 1$ . The end-of-text token is $V_{\text{text}}$ . Level-1 code $c$ has id $V_{\text{text}} + 1 + c$ . The end-of-audio token is $V_{\text{text}} + 1 + K$ . Write three functions.

valle_ar_example(phones, codes1, V_text, K) builds the training sequence [phonemes, end of text, offset codes, end of audio]. It returns (inputs, targets, loss_mask). inputs is the sequence without its last token, and targets is the sequence without its first token. loss_mask is 1 exactly where the target is an audio code or the end of audio, and 0 elsewhere.
valle_ar_prompt(prompt_phones, phones, prompt_codes1, V_text, K) returns the inference prefix [prompt phonemes, new phonemes, end of text, offset prompt codes], which the model then continues.
valle_nar_input(prompt_C, C, q, E) builds the NAR model's input for one level. Its arguments are the prompt codes of shape (n_q, T_p), the target codes of shape (n_q, T), the 0-indexed level q >= 1 to predict, and per-level embedding tables E of shape (n_q, K, d). It returns the input matrix of shape (T_p + T, d) and a loss mask. Each prompt row sums the embeddings of all levels. Each target row sums the embeddings of levels 0 to q - 1. Every level is looked up in its own table, and the mask is 1 on the target rows. Leave the phonemes out of this function.

import numpy as np

def valle_ar_example(phones, codes1, V_text, K):
    raise NotImplementedError

def valle_ar_prompt(prompt_phones, phones, prompt_codes1, V_text, K):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

One sentence, end to end, as codec tokens

Here is one sentence followed through every stage of VALL-E, with the shape and the count at each step. It is the sentence that §37.1 followed to a mel spectrogram, Dr. Smith paid $5., so the two routes can be compared. There the front end turned it into the words "doctor smith paid five dollars" and 20 phonemes, and the predicted durations gave 1.81 seconds of speech.

✏️ One sentence as codec tokens

Text. The 20 phonemes become 20 ids. A 3-second prompt recording comes with its transcript, about 36 phonemes at an assumed 12 phonemes per second.
Prompt. The codec encodes the prompt into $\tilde{\mathbf{C}}$ with $8 \times 225$ codes. Only its level 1, 225 codes, enters the AR model.
AR stage. The prefix is $36 + 20 + 1 + 225 = 282$ tokens. The model samples one level-1 code per step and stops when it emits the end of audio, here after $1.81 \times 75 \approx 136$ codes. That is 137 sequential steps.
NAR stage. Seven passes, one per level from 2 to 8, each over $225 + 136 = 361$ frames, fill in the rest of $\mathbf{C} \in [K]^{136\times 8}$ .
Decoding. The codec decoder looks up the codebook vector that each code names and adds the 8 vectors of each frame into one latent vector. Its convolutional layers then turn each latent frame into 320 samples. The 1,088 codes, $136 \times 80 = 10{,}880$ bits at 10 bits per code, become $136 \times 320 = 43{,}520$ samples, 1.81 seconds at 24 kHz.

In all, $137 + 7 = 144$ sequential model calls produce the sentence, against 1,089 for the flattened order. The mel route of §37.1 produced 156 frames of 80 real numbers each. This route produces 136 frames of 8 integers each. The voice came from the prompt's codes, and the length came from when the AR model chose to stop.

The widget follows the sentence through this route, level-1 codes first and then the seven finer levels, on the same time axis as the mel route of §37.1. Switch routes to compare the two.

The cell runs the same pipeline with stand-ins for the two trained models, so every array has its real shape. A trained AR model would sample each code from its prediction given the prefix. A trained NAR model would take the most probable code for each frame.

Python · cell 2

Interleaving patterns describe every order in one language

VALL-E's split is one answer to the question of order. MusicGen, a text-to-music model from Meta (Copet et al. 2023), turned the question into a family of answers with one language for all of them. An interleaving pattern says which codes of the grid are predicted at each step, and so which earlier codes each prediction may look at. We call the order of the flattened row, frame by frame and coarse to fine within a frame, the frame-major order.

📐 Codebook interleaving pattern

Let $\Omega = \{(t, q) : t \in [T],\, q \in [n_q]\}$ be the positions of a code grid, where $[T]$ means $\{1, \dots, T\}$ . A pattern is a sequence of sets $\mathcal{P} = (P_1, \dots, P_S)$ that partitions $\Omega$ , so every position belongs to exactly one set. A model that uses it makes $S$ sequential steps. At step $s$ it predicts every code in $P_s$ at once, each from its own output distribution, conditioned on all codes in $P_1 \cup \dots \cup P_{s-1}$ . The five common patterns and their step counts are these.

Pattern	Step $s$ predicts	$S$
Flatten	the single code at position $s$ of the frame-major order	$T n_q$
Parallel	$\{(s, q) : q \in [n_q]\}$	$T$
Delay	$\{(s - q + 1, q) : q \in [n_q],\ 1 \le s - q + 1 \le T\}$	$T + n_q - 1$
Coarse first	$\{(s, 1)\}$ for $s \le T$ , then $\{(s - T, q) : q \ge 2\}$	$2T$
VALL-E	$\{(s, 1)\}$ for $s \le T$ , then $\{(t, s - T + 1) : t \in [T]\}$	$T + n_q - 1$

The delay pattern is the one MusicGen used, and it is easiest to see as a shift. Write the grid with one row per level. Slide row $q$ to the right by $q - 1$ positions and pad the gaps. Every column of the shifted grid is now one step. Column $s$ holds level 1 of frame $s$ , level 2 of frame $s - 1$ , and so on. So the codes predicted together always come from different frames.

The cell prints, for a grid of 4 frames and 3 levels, the step at which each code is produced. The code counts levels and frames from 0, as the exercises do.

Python · cell 3

The diagonal stripes of the delay pattern are visible in the printout. So is the price of VALL-E's pattern, whose last two steps each produce a whole level of all frames.

Warm-up

Delay and undelay

Write delay(codes, pad), which maps a code grid of shape (n_q, T) to the delayed grid of shape (n_q, T + n_q - 1). In the delayed grid, level q (0-indexed) is shifted right by q columns, and every empty position holds pad. Column s of the delayed grid is what a delay-pattern model predicts at step s. Then write undelay(delayed), which inverts it. Keep the dtype of the input.

import numpy as np

def delay(codes, pad):
    raise NotImplementedError

def undelay(delayed):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

When a pattern is exact

Every pattern except flattening draws some codes together, and codes drawn together cannot see each other. How much does that cost? The answer is exact, and it needs two quantities from §0.5. The entropy $H(c)$ of a random code is its average surprise, $-\mathbb{E}[\log p(c)]$ , in nats when the log is natural. The KL divergence $D_{\mathrm{KL}}(p \Vert \tilde p)$ is the average extra surprise of using a model $\tilde p$ when the data come from $p$ . It is never negative, and it is zero only when $\tilde p = p$ .

Suppose a model is trained with a pattern by maximum likelihood, each code in $P_s$ scored against its true value given the codes of earlier steps. Write $\mathbf{c}_{<s}$ for the codes in $P_1, \dots, P_{s-1}$ . The expected cross-entropy is smallest at the true distribution, so the best such model outputs the true conditional $p(c_i \mid \mathbf{c}_{<s})$ of each single code $i \in P_s$ . Now sample from those exact conditionals step by step, drawing the codes of one step independently of each other. Call the distribution of the result $\tilde p_{\mathcal{P}}$ ,

\tilde p_{\mathcal{P}}(\mathbf{C}) = \prod_{s=1}^{S}\ \prod_{i\in P_s} p(c_i \mid \mathbf{c}_{<s}).

In plain words, $\tilde p_{\mathcal{P}}$ is the best that any model trained with this pattern can do. The question is how far it is from the truth $p$ .

The answer uses the total correlation of a set of codes, the information they carry about each other. It is the sum of their separate entropies minus their joint entropy, so it is zero exactly when they are independent. Two codes that are always equal fair coins have separate entropies of $\log 2$ each and a joint entropy of $\log 2$ , so their total correlation is $\log 2$ . §34.4 used the same quantity to price one parallel step of masked image generation. The answer for a whole pattern adds up its steps.

∴ When a pattern is exact

Let $p$ be any distribution over code grids, $\mathcal{P}$ a pattern, and $\tilde p_{\mathcal{P}}$ the distribution sampled with exact per-code conditionals. Then

D_{\mathrm{KL}}\big(p \,\Vert\, \tilde p_{\mathcal{P}}\big) \;=\; \sum_{s=1}^{S} \mathbb{E}_{\mathbf{c}_{<s}\sim p}\Big[\operatorname{TC}\big(\mathbf{c}_{P_s} \mid \mathbf{c}_{<s}\big)\Big], \qquad \operatorname{TC}(\mathbf{c}_A \mid \mathbf{c}_{<s}) = \sum_{i\in A} H(c_i \mid \mathbf{c}_{<s}) - H(\mathbf{c}_A \mid \mathbf{c}_{<s}),

where the entropies are computed under $p$ for the given context. Consequently

$\tilde p_{\mathcal{P}} = p$ if and only if, for every step $s$ , the codes in $P_s$ are conditionally independent given $\mathbf{c}_{<s}$ (for every context with positive probability), and
a pattern is exact for every distribution $p$ if and only if each of its steps holds a single code.

∂ Proof of the pattern theorem

By the chain rule over steps, $p(\mathbf{C}) = \prod_s p(\mathbf{c}_{P_s} \mid \mathbf{c}_{<s})$ . Dividing by $\tilde p_{\mathcal{P}}$ and taking logs,

\log\frac{p(\mathbf{C})}{\tilde p_{\mathcal{P}}(\mathbf{C})} = \sum_{s=1}^{S}\Big[\log p(\mathbf{c}_{P_s}\mid\mathbf{c}_{<s}) - \sum_{i\in P_s}\log p(c_i\mid\mathbf{c}_{<s})\Big].

Take the expectation under $p$ , one step at a time. Given a context $\mathbf{c}_{<s}$ , the codes $\mathbf{c}_{P_s}$ are distributed as $p(\mathbf{c}_{P_s}\mid\mathbf{c}_{<s})$ , whose marginals are the $p(c_i\mid\mathbf{c}_{<s})$ . So the bracket's expectation is $-H(\mathbf{c}_{P_s}\mid\mathbf{c}_{<s}) + \sum_i H(c_i\mid\mathbf{c}_{<s})$ , the total correlation. Averaging over contexts gives the identity.

Each term is the KL divergence between a joint distribution and the product of its marginals. So it is nonnegative, and it is zero exactly when the codes of the step are conditionally independent, which gives statement 1. For statement 2, a step with one code has total correlation zero for every $p$ . If a step holds two codes $i \ne j$ , take a $p$ in which $c_i$ and $c_j$ are equal fair coins independent of everything else. Their total correlation is $\log 2$ , so the pattern is inexact for that $p$ . ∎

In plain words, a pattern costs exactly the information that codes drawn in the same step carry about each other, given what came before, summed over the steps. Flattening pays nothing, because it never draws two codes together. Every other pattern is a bet that the codes it draws together are close to independent given what it has already seen.

MusicGen's authors state one direction for the parallel pattern, that it is a proper model only if the codes of each frame are conditionally independent given the past. The theorem adds the converse and covers every pattern. Note also what the theorem does not require. A pattern need not follow the frame-major order to be exact. Coarse first with $n_q = 2$ draws one code per step, so it is exact, even though each level-1 code is predicted before the level-2 codes of all earlier frames. It is the chain rule in a different order.

The theorem tells us where each pattern places its bet.

Parallel draws all levels of a frame together. Residual quantization makes them strongly dependent, since level $q$ is by construction a correction of the levels below it.
Coarse first, with three or more levels, draws levels 2 to $n_q$ of a frame together, the same dependence one level up.
Delay draws codes of different frames together, and each code sees every coarser code of its own frame. Its only bet is that codes of nearby frames are nearly independent given everything drawn in earlier steps.
VALL-E's NAR passes draw one level of all frames together, given every coarser level of every frame, including future frames. The bet is that fine codes are local detail once the coarse codes are fixed. SoundStorm, later in the section, makes the same bet.

The next cell measures these bets exactly in a toy world of codes. It has 3 frames, 3 levels, and 3 codes per level, small enough that the joint distribution over all $3^{9} = 19{,}683$ grids fits in a table. Level 1 (index 0 in the code) is a sticky chain over time, with a weak link to the previous frame's level 2. Level 2 depends strongly on level 1 of its own frame and weakly on its own previous value. Level 3 depends strongly on level 2 and weakly on level 1. For each pattern, the cell computes $\tilde p_{\mathcal{P}}$ from the exact conditionals and its KL divergence from $p$ .

Python · cell 4

The widget draws the grid for 8 frames and 4 levels, counted from 0 at the bottom. Press Generate to watch a pattern write its codes step by step. Click a code to see three things. They are the codes the model has already seen when it predicts that code, the frame-major predecessors it has not seen, and the codes drawn in the same step. In the delay pattern, click a code high in the grid. It sees the coarse codes of later frames, a lookahead that the frame-major order never gives.

The theorem predicts the ranking that MusicGen measured on real music.

✏️ Thirty seconds of music, six orders

MusicGen's codec is EnCodec at 32 kHz with a stride of 640 samples, so $f_r = 50$ Hz, with $n_q = 4$ levels of 2,048 codes. Thirty seconds is $T = 1{,}500$ frames and 6,000 codes. Quality was measured by the Fréchet audio distance (FAD), the distance between the statistics of embeddings of generated audio and of real audio, so lower is better. It is the audio version of the FID of §34.6. MusicGen's authors trained a 300 M model for each pattern (Copet et al. 2023, Table 4), and on the in-domain test set the results were these.

Pattern	Steps	FAD
Flatten	6,000	0.86
Delay	1,500	0.96
Partial flattening	3,000	1.32
Partial delay	1,500	1.51
Coarse first	3,000	1.98
Parallel	1,500	2.58

The paper counts the delay pattern as 1,500 steps, and it is exactly $T + n_q - 1 = 1{,}503$ . Delay comes within 0.1 FAD of flattening at a quarter of the steps, and the ranking is the one the theorem predicts. The four patterns that draw levels of the same frame together are clearly worse. Partial flattening predicts level 1 of a frame and then levels 2 to 4 of that frame together. Partial delay delays levels 2 to 4 all by the same amount. Coarse first draws levels 2 to 4 of a frame together, and parallel draws all four.

An interviewer who asks "why does the delay pattern work?" wants this argument. The codes it draws together come from different frames, and each of them already sees the coarser codes of its own frame.

Interview

Which predictions does each pattern starve?

Write pattern(name, T, n_q) for the names "flatten", "parallel", "delay", "coarse_first", and "valle". It returns a list of steps, each a list of (t, q) with frames and levels 0-indexed, following the table above. Assume n_q >= 2.

Then write missing_dependencies(steps). It returns a dict that maps every code (t, q) to the set of its frame-major predecessors that are not in a strictly earlier step. A code's frame-major predecessors are all codes of earlier frames and the coarser codes of its own frame. Codes in the same step cannot see each other. A missing dependency is not an error by itself, since coarse first with two levels misses some and is exact. But it shows where a pattern places its independence bets.

def pattern(name, T, n_q):
    raise NotImplementedError

def missing_dependencies(steps):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Two Transformers, one for time and one for depth

Delay is inexact, and flattening is exact but slow. A third design keeps the exactness and most of the speed. It gives the two axes of the grid two different models, a large one that steps through time and a small one that steps through the levels of a frame. The idea comes from image generation, the RQ-Transformer of Lee et al. (2022), and audio models used it before Moshi did. We follow the form written down for Moshi (Défossez et al. 2024), a dialogue model that §37.5 builds on this design.

📐 RQ-Transformer (temporal and depth Transformers)

Let $\mathbf{V}_s = (V_{s,1}, \dots, V_{s,n_q})$ be the codes of step $s$ , with $\mathbf{V}_0$ a fixed start value. A large temporal Transformer reads the past steps and outputs a context vector. A small depth Transformer predicts the codes of the current step one level at a time,

\mathbf{z}_s = \operatorname{Tr}_{\text{temp}}(\mathbf{V}_0, \dots, \mathbf{V}_{s-1}) \in \R^{d}, \qquad \boldsymbol\ell_{s,1} = \operatorname{Lin}(\mathbf{z}_s), \qquad \boldsymbol\ell_{s,k} = \operatorname{Tr}_{\text{depth}}(\mathbf{z}_s, V_{s,1}, \dots, V_{s,k-1}) \quad (k > 1).

Here $\operatorname{Lin}$ is a linear layer, and $\boldsymbol\ell_{s,k} \in \R^{K}$ are the logits for level $k$ . The model is trained so that $\operatorname{softmax}(\boldsymbol\ell_{s,k}) \approx p(V_{s,k} \mid \mathbf{V}_{<s}, V_{s,<k})$ . The temporal Transformer's input at step $s$ is the sum of $n_q$ learned embeddings of the codes of $\mathbf{V}_{s-1}$ . The depth Transformer's input at position $k$ is $\mathbf{z}_s$ plus an embedding of $V_{s,k-1}$ , so it runs $n_q - 1$ steps per frame. Moshi's depth Transformer has 6 layers, width 1,024, and 16 heads, and each position $k$ has its own linear layers (projections and feed-forward).

In plain words, the big model reads the history once per frame and summarizes it in $\mathbf{z}_s$ . A linear layer reads the first code off that summary. The small model then spells out the frame's other codes from coarse to fine. Each code sees the summary and the codes already chosen in its frame.

This is the flattened chain rule with no independence bet, so by the theorem it is exact. What changes is the cost. The large model runs $T$ steps instead of $T n_q$ . Its KV cache, the stored keys and values of past positions (§6.5), holds $T$ positions instead of $T n_q$ . The small model runs $n_q - 1$ steps per frame over a sequence never longer than $n_q - 1$ , so its cache is thrown away after every frame.

✏️ Counting the steps of an RQ-Transformer

Say one step of the large model costs $C_{\text{big}}$ and one step of the small model costs $C_{\text{small}}$ . Then a frame costs $C_{\text{big}} + (n_q - 1) C_{\text{small}}$ , instead of $n_q C_{\text{big}}$ for flattening.

Take Mimi's codes, with $f_r = 12.5$ and $n_q = 8$ . Flattening needs 100 steps of the large model per second of audio, 30,000 for 5 minutes. The RQ-Transformer runs the large model 12.5 times per second, 3,750 times for 5 minutes. The small model takes $7 \times 12.5 = 87.5$ steps per second, each over at most 7 positions. Moshi's temporal Transformer has 7 B parameters and its depth Transformer is a small fraction of that, so almost all the compute now scales with frames, not codes.

Moshi adds one refinement, the delay pattern's idea applied inside the frame. Mimi's first level is semantic and the other seven are acoustic (§37.3). Moshi shifts the acoustic levels by an acoustic delay of $\tau_{\text{ac}}$ frames. With Mimi's codes written $A_{t,q}$ , the step's codes are $V_{s,1} = A_{s,1}$ and $V_{s,q} = A_{s-\tau_{\text{ac}},q}$ for $q \ge 2$ . With $\tau_{\text{ac}}$ of 1 or 2, the authors report that quality "greatly improves". The reason is that the large temporal model, rather than the small depth model, now sees a frame's semantic code before it has to describe that frame's sound.

Later systems use the same split. Qwen3-TTS (Hu et al. 2026) predicts the first of its 16 levels at 12.5 Hz with the large backbone and the other 15 with a small multi-token prediction module in the depth role. MOSS-TTS (Gong et al. 2026) offers both a delay-pattern model and one with a small autoregressive module inside each frame.

The reference block builds both pieces at toy size. The first is a VALL-E style AR model. It is a causal Transformer over [phonemes, end of text, level-1 codes, end of audio] whose loss is masked to the audio targets, the layout of Exercise 37.4a. The second predicts the 4 levels of one frame from a context vector, level 0 with a linear layer and levels 1 to 3 with a small causal depth Transformer. A check confirms that the prediction for level $k$ cannot see the code of level $k$ .

🔥 In PyTorchfor a machine with PyTorch

Filling the grid in parallel

An autoregressive model over frames still takes one step per frame. Masked parallel decoding removes that too. §34.4 built it for images (Chang et al. 2022), and here is its rule in brief. Start with every position masked. A bidirectional model predicts every masked position at once. The predictions it is most confident about, those with the highest probability, are kept, and the rest are masked again. After iteration $i$ of $I$ , the number of positions still masked is $\lceil N\cos(\tfrac{\pi}{2}\tfrac{i}{I})\rceil$ out of $N$ . This cosine schedule commits few tokens in the first iterations, while little of the grid is known, and many at the end.

SoundStorm (Borsos et al. 2023) applies this rule to AudioLM's acoustic stages. For each 50 Hz frame, its input is the sum of two kinds of embedding. One is the embedding of the conditioning semantic token, repeated from 25 Hz to 50 Hz. The others are the embeddings of the frame's 12 acoustic codes. So the bidirectional model attends over frames rather than codes, and its sequence does not grow with $n_q$ . The model is a 350 M Conformer, a Transformer with convolution layers added. It decodes one level at a time, coarse to fine. Within a level it runs the confidence-based schedule over the frames, with a greedy choice in the last iteration of each level.

Training mirrors decoding. For each example it draws a prompt boundary, a level $q$ to train, and a masking ratio from the cosine schedule. Frames before the boundary are a voice prompt and are never masked. At level $q$ , each later frame is masked with that probability. Every finer level is masked on all later frames, and the coarser levels stay visible. The loss is the cross-entropy on the masked tokens of level $q$ only.

📌 SoundStorm's speed

SoundStorm uses 16 iterations for level 1 and one greedy pass for each of the other 11 levels. So it needs $16 + 11 = 27$ forward passes to produce 30 seconds of audio, $30 \times 50 \times 12 = 18{,}000$ tokens, which AudioLM's acoustic stages produce one at a time. The authors report audio of the same quality as AudioLM's acoustic stages, with a more consistent voice and recording conditions, two orders of magnitude faster. It produces 30 seconds of audio in 0.5 seconds on a TPU-v4. Using 16 iterations instead of greedy decoding on level 1 raised a predicted MOS, the score of a network trained to imitate listener ratings, by 0.1 to 0.2. More iterations on the finer levels gave no statistically significant gain.

The schedule of levels is the theorem again. Codes of level 1 depend on each other strongly, so they get many small steps. Given all of level 1, the finer codes are close to conditionally independent local detail, so one parallel step per level loses little.

Research

SoundStorm's training mask and decoding passes

Write soundstorm_mask(T, n_q, t_p, q, frame_mask), which returns two boolean arrays of shape (n_q, T), masked and loss. Frames 0 to t_p - 1 are the prompt and are never masked. At level q (0-indexed), a frame t >= t_p is masked when frame_mask[t] is true. Every finer level is masked on all frames t >= t_p, and coarser levels are not masked. The loss covers exactly the masked tokens of level q.

Then write soundstorm_schedule(T, iters), where iters[l] is the number of decoding iterations of level l. Levels are decoded in order. Within a level with I iterations, the number still masked after iteration i is $\lceil T\cos(\tfrac{\pi}{2}\tfrac{i}{I}) - 10^{-9}\rceil$ . The small constant guards against rounding, so that the count is 0 after the last iteration. Return a list with one (level, n_committed) pair per forward pass, where n_committed is the number of tokens that pass unmasks.

import numpy as np

def soundstorm_mask(T, n_q, t_p, q, frame_mask):
    raise NotImplementedError

def soundstorm_schedule(T, iters):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Hybrids put a language model on the content and a flow on the sound

Many of the strongest text-to-speech systems of 2025 and 2026 take the slow-then-fast idea to its end. A language model generates a single stream of semantic tokens, one at a time. A separate continuous generator then turns them into sound. Usually it is flow matching (§33.6, used for TTS in §37.1), which learns a velocity field that carries noise to data along straight paths. Sometimes it is diffusion.

CosyVoice (Du et al. 2024) is the cleanest example, in four parts.

Its tokens are supervised semantic tokens. They are made by inserting a vector quantizer into the encoder of a multilingual speech recognizer (§37.3), so they keep what recognition needs.
An LLM maps text to these tokens.
A conditional flow-matching model maps the tokens, a speaker embedding (an x-vector, §35.6), and the prompt's mel spectrogram to a mel spectrogram.
A neural vocoder, HiFi-GAN (§37.2), makes the waveform.

CosyVoice 2 (Du et al. 2024) made the LM a pretrained text LLM, Qwen2.5-0.5B, with no separate text encoder and no speaker embedding in the LM. Its tokens come from finite scalar quantization, which rounds each coordinate of a short vector to one of a few levels, at 25 tokens per second. Its flow-matching decoder is chunk-aware and causal, so it can stream. Its mel spectrogram runs at 50 frames per second for 24 kHz audio, so each speech token covers two mel frames. The flow takes 10 function evaluations, 10 forward passes of its network per chunk. CosyVoice 3 (Du et al. 2025) scaled the data from ten thousand to one million hours and the model to 1.5 B parameters.

✏️ One sentence, three ways

Take the same 1.81 seconds of speech through three designs.

Design	Tokens the LM writes	Sequential LM calls	Then
VALL-E, EnCodec at 75 Hz	136 frames of 8 codes	137 AR steps and 7 NAR passes	codec decoder, 43,520 samples
Temporal plus depth, Mimi at 12.5 Hz	23 frames of 8 codes	23 large steps and 161 small steps	codec decoder, $23 \times 1{,}920 = 44{,}160$ samples
Hybrid, CosyVoice 2 at 25 Hz	45 semantic tokens	about 45 steps	90 mel frames by a flow with 10 evaluations, then a vocoder, $90 \times 480 = 43{,}200$ samples

The temporal plus depth design takes the fewest large-model steps. The hybrid's LM takes about twice as many, but each of its steps writes one token of content and nothing about the sound. The voice is not in its tokens at all. It enters in the flow, through the speaker embedding and the prompt's spectrogram.

Why do hybrids work so well? Each half does what it is good at. The slow, discrete part (which words, with what timing, with what prosody) is a long-range sequence problem. Autoregressive LMs are strong there, and the text knowledge of a pretrained LLM transfers. The fast, continuous part is local and high-dimensional. Autoregression over its codes would be slow and would compound errors, while a parallel generator that sees whole chunks is fast and accurate. The hybrid also lets the acoustic stage condition directly on the prompt's spectrogram, so the voice does not have to be squeezed through the semantic tokens.

VibeVoice (Peng et al. 2025) goes one step further and makes the tokens continuous. Its tokenizer compresses 24 kHz audio 3,200 times, to 7.5 latent frames per second. A Qwen2.5 LLM, with 1.5 B or 7 B parameters, reads text and past latents. Its hidden state at each step conditions a small diffusion head of 4 layers that generates the next continuous latent, which the authors call next-token diffusion. The low frame rate is what makes long audio fit. Ninety minutes at 7.5 Hz is $90 \times 60 \times 7.5 = 40{,}500$ frames, inside its 64K context, and the model generates up to 90 minutes with up to 4 speakers.

Streaming text in, speech out

A voice assistant does not have the whole sentence when it starts speaking, because the text LLM is still writing it. CosyVoice 2 handles this inside the LM's own sequence, by alternating short runs of text and speech.

📐 Bi-streaming interleave

Given text tokens $x_1, \dots, x_{T_{\text{txt}}}$ and speech tokens $u_1, u_2, \dots$ , CosyVoice 2's streaming sequence for an interleave ratio $N{:}M$ is built in three parts.

It starts with the start token $\mathtt{S}$ .
Then come blocks of $N$ text tokens, each followed by $M$ speech tokens, until the text runs out. The last block may have fewer than $N$ text tokens.
Then come the turn-of-speech token $\mathtt{T}$ , the remaining speech tokens, and the end token $\mathtt{E}$ .

The model is trained by next-token prediction on this sequence, with two substitutions. Where the next token is text that follows a speech token, the target is a filling token $\mathtt{F}$ instead. Other positions whose next token is text carry no loss. CosyVoice 2 uses $N = 5$ and $M = 15$ . At inference, when the model emits $\mathtt{F}$ , the system appends the next $N$ text tokens as they arrive.

In plain words, the model reads a little text and speaks a little. Whenever it has spoken its share, it says "more text, please" with the filling token. When no more text will come, the turn-of-speech token switches it to plain continuation until it ends.

The ratio keeps the text ahead of the speech. At 25 speech tokens per second and roughly 3 to 4 text tokens per second of English speech, a sentence needs about 6 to 8 speech tokens per text token. That is more than $M/N = 3$ . So each block of text covers more audio than the model speaks before it reads the next block, and the model never has to speak words it has not read.

Core

The bi-streaming sequence

The setup defines the special tokens SOS = "S", TURN = "T", EOS = "E", FILL = "F", and IGNORE = None, a target that carries no loss. Write bistream(text, speech, N=5, M=15), returning (inputs, targets), two lists of equal length.

The full sequence is S, then blocks of N text tokens each followed by the next M speech tokens until the text runs out, then T, the remaining speech, and E. inputs is the full sequence without its last token. targets[k] is the next token of the full sequence, with two exceptions. It is FILL when the next token is text and the current one is speech. It is IGNORE when the next token is text and the current one is not speech. You may assume there are at least M speech tokens for every text block.

def bistream(text, speech, N=5, M=15):
    seq = [SOS] + list(text) + [TURN] + list(speech) + [EOS]
    return seq[:-1], seq[1:]

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Music, and guidance on token logits

MusicGen (Copet et al. 2023) is a single-stage codec LM for music. It models the 4 levels of a 32 kHz EnCodec at 50 Hz with the delay pattern. It reads the text caption through a pretrained text encoder, T5-base, and cross-attention, and it can also follow a melody. The models have 300 M, 1.5 B, and 3.3 B parameters. They were trained on 20,000 hours of licensed music and sample with top-k 250 at temperature 1.

The melody conditioning needs no labels. A chromagram of a reference track gives the energy of each of the 12 pitch classes (C, C sharp, D, and so on) in every frame. MusicGen keeps only the dominant pitch class per frame. The authors added this bottleneck because the raw chromagram let the model copy the reference.

Like image token models, MusicGen uses classifier-free guidance on its logits, a way to make a model follow its condition more strongly than it learned to. During training the text condition is dropped with probability 0.2. So one network learns both the conditional logits $\boldsymbol\ell_c$ , given the caption, and the unconditional logits $\boldsymbol\ell_u$ , given no caption. At each step it samples from

\operatorname{softmax}\big(\boldsymbol\ell_u + \gamma_{\text{cfg}}(\boldsymbol\ell_c - \boldsymbol\ell_u)\big), \qquad \gamma_{\text{cfg}} = 3 \text{ in MusicGen}.

In plain words, start from the logits the model would use with no caption, and move $\gamma_{\text{cfg}}$ times as far as the caption pushes them. With $\gamma_{\text{cfg}} = 1$ this lands on the conditional logits, which is plain conditional sampling. With $\gamma_{\text{cfg}} = 3$ it goes twice as far again, past what the caption alone would do.

§34.4 worked out what this rule does. Write $p_c$ and $p_u$ for the softmax of the two logit vectors. Guidance samples each token from the renormalized product $p_c^{\gamma_{\text{cfg}}}p_u^{1-\gamma_{\text{cfg}}}$ , the per-token version of the Bayes argument for diffusion guidance in §33.5. That holds token by token, not for the whole sequence, because each step's normalizer depends on the prefix. The rule must combine logits or log-probabilities, never probabilities, which can turn negative. And it runs the model twice per step, with and without the caption, which doubles the batch and the KV cache but not the number of steps. The same guidance also helps text LLMs (Sanchez et al. 2023). The cell shows its effect on one codebook's distribution.

Python · cell 5

The caption alone raises the odds of the 64 favored codes by a factor of $e$ and moves their share from 0.029 to 0.076. Guidance with $\gamma_{\text{cfg}} = 3$ raises the odds by $e^{3}$ , to a share of 0.377, and lowers the entropy from 9.55 to 8.73 bits. That is the familiar trade of diversity for adherence to the text.

The reference call through Hugging Face transformers is a few lines. At 50 frames per second, 256 new tokens is about 5 seconds of audio.

Sampling audio tokens without loops

Everything in §6.4 applies to audio tokens. Temperature divides the logits before the softmax. Top-k keeps the $k$ most probable tokens before sampling. Top-p, or nucleus sampling, keeps the smallest set of most probable tokens whose total probability reaches $p$ , called the nucleus. What is specific to audio is how decoding fails.

Loops. The model repeats a few codes, heard as a stuck syllable, a buzz, or endless silence. VALL-E saw beam search fall into infinite loops, and VALL-E 2 reported the same for nucleus sampling with a small top-p.
Stopping errors. The end token comes too early, which cuts the sentence, or never comes, which runs on into babble.
Hallucinated content. Words are skipped, repeated, or added. This is the codec LM's version of the attention failures of Tacotron in §37.1.

Loops have a mechanism. In text LMs, Xu et al. (2022) found a self-reinforcement effect. The more times a sentence has been repeated in the context, the more probable the model makes the next repetition. An audio code that repeats has the same pull, and a run of silence or of one steady sound is the easiest pattern to continue. Once the probability of repeating reaches the top-p threshold, the nucleus contains only the repeated code, and no amount of sampling can leave. Sampling from the full distribution escapes. But on every step it also lets in the long tail of unlikely codes, which costs quality.

The cell runs a toy codec LM over 16 codes for 300 steps at several values of top-p. Its probability of repeating grows with the length of the current run, from 0.6 when the last code has just appeared to 0.9 once it has appeared four times in a row. Of the remaining probability, 0.4 goes to one "next sound" code and 0.6 is spread evenly over the other 14 codes. A tail draw is a code whose probability was below 0.05.

Python · cell 6

VALL-E 2's fix is to truncate as usual and to watch for repetition.

📐 Repetition-aware sampling (RAS)

The inputs are the model's next-code distribution $\mathbf{p}$ , the previously generated codes $c_1, \dots, c_{n-1}$ , a top-p value, a window $K_{\text{win}}$ , and a threshold $t_r$ . Repetition-aware sampling first draws a candidate $\hat c$ from the nucleus of $\mathbf{p}$ . It then computes the candidate's repetition ratio over the preceding window,

r = \frac{1}{K_{\text{win}}}\sum_{k=1}^{K_{\text{win}}} \mathbb{1}[c_{n-k} = \hat c],

counting positions before the start of the sequence as non-matches. If $r > t_r$ , it discards $\hat c$ and draws the code from the full distribution $\mathbf{p}$ instead. Otherwise it returns $\hat c$ . VALL-E 2 used $K_{\text{win}} = 10$ and $t_r = 0.1$ .

In plain words, sample carefully from the head of the distribution as long as the output is varied. The moment the chosen code already fills more than a tenth of the last ten slots, take one draw from the whole distribution instead. Its small chance of a different code is what ends the loop.

RAS changes the output only on steps that repeat. On varied speech it is plain nucleus sampling, and its cost is one extra draw, negligible next to a forward pass. It is what let VALL-E 2 decode with a very small top-p, even 0. The authors identify that as the key to a word error rate below that of the ground-truth recordings. One detail matters if you implement it from the paper. Its Algorithm 1 writes the sum from $k = 0$ , which would count the candidate itself, while dividing by $K_{\text{win}}$ . With $t_r = 0.1$ that version falls back whenever the candidate appears even once in the window. We count only the $K_{\text{win}}$ preceding codes. That matches the division, and the paper's prose, which speaks of the candidate's ratio in the preceding codes.

The widget's toy model caps its repeat probability at 0.85 on the sticky preset. The line under the strip is the model's chance of repeating the last code, and the dashed line is top-p. The widget opens with RAS off at top-p 0.8. Within a few codes the line climbs to the dashed line, and the strip becomes one long outlined loop, because the nucleus holds only the repeated code. Turn RAS on and the loops end, since every fallback leaves with probability at least 0.15. Then switch RAS off and set top-p to 1. The runs look much the same, and the tail draws go up.

That is the whole effect of RAS. On repetitive stretches it samples exactly like the full distribution, so its loops last as long as those of top-p 1. Everywhere else it keeps the nucleus. On the very sticky preset, where the model repeats with probability 0.97, every sampler leaves long runs, because each draw escapes only 3 percent of the time. The fix there is a better model, not a better sampler.

Interview

Repetition-aware sampling

Write ras_sample(probs, history, rng, top_p=0.8, window=10, threshold=0.1), returning (token, fell_back). The setup provides nucleus_probs, which applies top-p with the threshold rule of §6.4.

Draw a candidate with rng.choice(K, p=nucleus_probs(probs, top_p)), where K = len(probs).
Compute its repetition ratio, the number of matches among the last window codes of history, divided by window even when the history is shorter than the window.
If the ratio exceeds threshold, return a fresh draw rng.choice(K, p=probs) from the full distribution and True. Otherwise return the candidate and False.

Use rng for every draw and draw nothing else, so that without a fallback the result equals plain nucleus sampling under the same seed.

import numpy as np

def ras_sample(probs, history, rng, top_p=0.8, window=10, threshold=0.1):
    token = int(rng.choice(len(probs), p=nucleus_probs(probs, top_p)))
    return token, False

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Stopping, and controlling duration

An autoregressive codec LM decides the length of an utterance by when it emits its end token. That is flexible but not controllable, and dubbing and video need an exact duration. IndexTTS2 (Zhou et al. 2025) adds a mode in which the number of tokens to generate is given explicitly, next to a free mode. MOSS-TTS offers token-level duration control. Non-autoregressive systems, MaskGCT and the flow-matching models of §37.1, set the length directly. That gives control for free and moves the problem to estimating a good length.

In free mode, stopping errors are usually treated in two ways. One is a length prior at inference, which refuses an end token far before the length the text implies and forces one far after it. The other is the post-training rewards of the next section, since a cut sentence raises the word error rate.

Post-training with rewards a computer can score

Speech generation has unusually cheap automatic judges. A speech recognizer gives a word error rate against the input text (§36.1). A speaker-verification model gives the similarity to the prompt voice (§35.6). An emotion classifier says whether the requested emotion came out. A MOS predictor estimates the mean opinion score, the 1 to 5 naturalness rating that listeners would give (§37.6).

Each judge can serve as a reward for the methods of Chapter 15 to Chapter 18. The codec LM is the policy $\pi_\theta$ , and a frozen copy of it is the reference $\pi_{\text{ref}}$ that a KL penalty keeps the policy close to. Three methods recur.

DPO (§17.2) fits preference pairs directly, with no reward model and no sampling during training.
GRPO (§18.4) samples a group of $G$ outputs for the same input and scores each by its reward relative to the group's mean.
GSPO, in the same section, is GRPO with one importance ratio per sequence instead of one per token.

The same report documents the failure. To reach a lower WER, the RL model tended to speak more slowly and pronounce more clearly, at a cost in naturalness. The authors had already observed that a lower WER usually means more "standardized" speech that is easier for the recognizer. Such speech keeps less of the prompt's accent and expressiveness, and listeners judge it less similar to the prompt. This is reward overoptimization (§16.5) in its speech form. The proxy measures how well one recognizer understands the speech. The goal is natural speech in a particular voice. Pushing on the proxy moves the output away from the goal in directions the proxy cannot see.

Every cheap judge can be gamed in its own way. A WER reward pays for slow, over-articulated, accent-free speech. A similarity reward can be raised by copying the prompt's room and noise along with the voice. A MOS predictor rewards whatever acoustic features it happens to have learned to like. Practical recipes therefore combine several rewards and keep a KL penalty to the reference model. They gate every release on human listening tests against the reference model, with accent and prosody checks on expressive prompts. The box gives the recipes of 2025 and 2026, and a way to spend the same judges at inference instead.

🔬 Post-training recipes of 2025 and 2026, and search at inference

The recipes follow the pattern above. Qwen3-TTS first runs DPO on human preference pairs, then GSPO with rule-based rewards. GLM-TTS (Cui et al. 2025) runs GRPO with several rewards, the character error rate, speaker similarity, emotion, and laughter. CosyVoice 3 introduced a differentiable reward model, which scores the speech tokens directly so that gradients flow through it. Luna-TTS applies GRPO to a diffusion LM, with policy ratios computed over the sampled denoising trajectory.

Rewards can also be spent at inference instead of in training. Llasa (Ye et al. 2025) samples many candidates and searches over them with speech understanding models as verifiers. It reports that this moves the outputs toward each verifier's preferences. An emotion verifier gives better emotion, a speaker verifier better timbre consistency, and a recognizer better accuracy. This is best-of- $n$ and verifier search (§18.2, §18.7) applied to audio, with the same caveat. A verifier is a reward model, and selecting hard on it overoptimizes it.

Where text to speech stands in 2026

The list below dates the main text-to-speech systems of 2025 and 2026 and says how each is built. Each one combines designs from this section, a language model over tokens, a flow or diffusion stage, a low frame rate, or post-training with rewards.

📌 Where TTS stands in 2026

Each system is dated by its first arXiv version and described as its authors describe it.

CosyVoice 3 (May 2025). Hybrid LM plus flow matching, a tokenizer trained on several supervised speech tasks, a differentiable reward model, one million hours of data in 9 languages and 18 Chinese dialects, and 1.5 B parameters.
IndexTTS2 (June 2025). An autoregressive model that can be told exactly how many tokens to generate, for dubbing, and that separates emotion from speaker identity.
VibeVoice (August 2025). Next-token diffusion over 7.5 Hz continuous latents, up to 90 minutes and 4 speakers.
GLM-TTS (December 2025). An autoregressive text-to-token model plus a diffusion token-to-waveform model, 100,000 hours of training data, GRPO with multiple rewards, and mixed phoneme and text input for pronunciation control.
Qwen3-TTS (January 2026). More than 5 million hours in 10 languages, a dual-track LM, and a first audio packet in as little as 97 ms with the 12.5 Hz tokenizer and the 0.6 B model.
MOSS-TTS (March 2026). Autoregressive modeling over a causal 12.5 Hz residual tokenizer with variable bitrate, with token-level duration control.
Luna-TTS (August 2026). A diffusion language model adapted from an autoregressive text LLM, trained on one million hours, refining the whole residual grid in parallel. On Seed-TTS-Eval it reports a WER of 1.49 percent and a SIM of 0.768 on the English set, and a CER of 0.73 percent and a SIM of 0.797 on the Mandarin set.

The benchmark in that list, Seed-TTS-Eval, was released with the Seed-TTS report and scores a system by its error rate and its speaker similarity (SIM) on fixed English and Mandarin test sets.

🔬 How Seed-TTS-Eval scores a system

Seed-TTS-Eval transcribes English with Whisper-large-v3 and Mandarin with Paraformer-zh, and it scores Mandarin by the error rate over characters (CER), which the report's tables label WER. It computes SIM, the cosine speaker similarity, often written as a percentage, with a WavLM-large speaker-verification model. For scale, Seed-TTS's own model reported English WER 2.249 and SIM 0.762, against 2.143 and 0.730 for the human recordings. In Mandarin it reported 1.115 and 0.796, against 1.254 and 0.750. In a side-by-side listening test against the recordings, rated from $-2$ to $+2$ for similarity and expressiveness, it scored a CMOS of $-0.07$ in English and $-0.08$ in Mandarin. The authors treat any CMOS smaller than 0.1 in size as an insignificant difference.

Two lessons follow for reading such tables. Synthetic speech can score a higher similarity to the prompt than a different real recording of the same speaker does. It copies the prompt's speaking style and recording conditions, which the other recording does not share. And numbers from different evaluation setups are not comparable, a point §37.6 develops.

Where codec language models break

⚠️ How codec LMs fail, and the bugs that look like model failures

Hallucination on hard text. Autoregressive codec LMs skip, repeat, or add words, most often on repetitions and tongue twisters, which is why Seed-TTS built its hard set. RAS, RL with a WER reward, and semantic first levels reduce it but do not remove it.
Errors that compound. Every committed code becomes context. A wrong code early in an utterance can derail the rest, which is the motivation Luna-TTS gives for refining the whole grid in parallel.
Drift in long audio. Speaker and style wander over minutes. Long contexts at low frame rates, as in VibeVoice, and re-prompting per segment help.
The prompt carries the room. Cloning copies noise, reverberation, and the microphone along with the voice, because the model continues all of the prompt's evidence, as VALL-E's authors noted for the acoustic environment. Clean the prompt, or condition on a speaker embedding when the room is unwanted.
Loss on the wrong positions. Putting loss on text or prompt positions, or, in a shared vocabulary, forgetting the offsets that keep text and audio ids apart (Exercise 37.4a), trains a model that predicts its own input. In the delay pattern, the padded positions must hold a special token that carries no loss.
Beam search. It maximizes probability, and the most probable continuation of audio is often a loop or silence. Sample instead.
Latency. A frame-by-frame AR stage cannot emit its first audio until a frame's codes are done, and an NAR stage over a whole level cannot stream at all. §37.5 works out the budget.
Misuse. Cloning from 3 seconds is exactly the capability that §37.6 has to govern with consent, classifiers, and watermarks.

The table puts the families of this section side by side, with the number of sequential model calls for $T$ frames of $n_q$ codes.

Family	Examples	Sequential calls	Streams natively	Typical failure	Duration	Voice or style from
Flattened AR codec LM	AudioLM's acoustic stages	$T n_q$	yes	slow, long contexts	end token	prompt continuation
AR plus NAR	VALL-E, VALL-E 2	$T + n_q - 1$	level 1 only	loops in the AR stage	end token	prompt continuation
Delay pattern	MusicGen	$T + n_q - 1$	yes	inexact within-step draws	chosen number of steps	text, melody, continuation
Temporal plus depth	Moshi, Qwen3-TTS 12.5 Hz	$T$ large plus $T (n_q - 1)$ small	yes	AR failures	end token	prompt or speaker embedding
Masked parallel	SoundStorm, MaskGCT	fixed, 27 for 30 s in SoundStorm	no	needs the total length	set in advance	prompt infilling
Hybrid LM plus flow or diffusion	CosyVoice, Seed-TTS	LM tokens plus flow steps per chunk	with chunk-aware flow	LM hallucination	end token	prompt plus embedding
Continuous AR	VibeVoice	$T$ , each with a diffusion head	yes	long-form drift	end token	prompt
Flow-matching NAR	F5-TTS (§37.1)	fixed ODE steps	no	length estimate	set in advance	infilling

🧭 What to remember

A codec LM must decide what to generate first and in what order to generate each frame's codes. Generating slow content before fast acoustics, as AudioLM's semantic, coarse, and fine stages do, keeps the words coherent and the voice intact.
VALL-E clones a voice by continuing a 3-second prompt's codes. An AR model over level 1 decides the content and the duration, and an NAR model fills levels 2 to 8 one level per pass. It samples because beam search loops.
A codebook pattern sampled with exact conditionals is off from the truth by exactly the total correlation of the codes it draws together. Flattening is exact. Delay, 1,503 steps for 30 s of MusicGen, is nearly exact because it never draws two levels of one frame together.
The RQ-Transformer keeps the exact chain rule at one large-model step per frame, and SoundStorm's level-by-level masked decoding produces 30 s of audio in 27 passes.
Hybrids of an LM over semantic tokens and a flow or diffusion model for the acoustics are among the strongest TTS systems of 2025 and 2026, with bi-streaming interleaves for text that arrives as it is written.
Decoding audio needs repetition-aware sampling against self-reinforcing loops. Post-training uses cheap rewards such as WER and similarity that are easy to game, so human listening tests stay the judge.
Next, §37.5 puts these generators inside a conversation, where the answer must start within a human turn gap and stop when the user interrupts.

Borsos et al. (2022). AudioLM: a Language Modeling Approach to Audio Generation. Section III gives the token analysis, the babbling experiment, and the three-stage hierarchy.
Wang et al. (2023). Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. VALL-E. Section 4 defines the AR and NAR models and the prompting, and Table 2 gives the results.
Chen et al. (2024). VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers. Algorithm 1 is repetition-aware sampling, and the introduction defines what the authors mean by human parity.
Copet et al. (2023). Simple and Controllable Music Generation. MusicGen. Section 2.2 introduces codebook patterns and exactness, and Table 4 compares them.
Défossez et al. (2024). Moshi: a speech-text foundation model for real-time dialogue. Section 3.4.1 writes the RQ-Transformer and the acoustic delay for audio.
Borsos et al. (2023). SoundStorm: Efficient Parallel Audio Generation. Level-by-level masked decoding of RVQ codes, with its training mask and the 27-pass count.
Du et al. (2024). CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models. The hybrid LM plus chunk-aware flow matching, and the bi-streaming interleave.
Anastassiou et al. (2024). Seed-TTS: A Family of High-Quality Versatile Speech Generation Models. The evaluation protocol behind Seed-TTS-Eval, RL on WER and similarity, and the reward hacking it caused.

Made for larger screens

Codec language models

Speech changes slowly in what it says and fast in how it sounds

Four orders for the codes of a frame

Phonetic units alone lose the voice

AudioLM writes the content first, then the coarse sound, then the fine sound

VALL-E clones a voice by continuing a prompt

VALL-E's training sequences and loss masks

One sentence, end to end, as codec tokens

Interleaving patterns describe every order in one language

Delay and undelay

When a pattern is exact

Which predictions does each pattern starve?

Two Transformers, one for time and one for depth

Filling the grid in parallel

SoundStorm's training mask and decoding passes

Hybrids put a language model on the content and a flow on the sound

Streaming text in, speech out

The bi-streaming sequence

Music, and guidance on token logits

Sampling audio tokens without loops

Repetition-aware sampling

Stopping, and controlling duration

Post-training with rewards a computer can score

Where text to speech stands in 2026

Where codec language models break

Read more

Watch next