This section is about learning speech from audio alone. A model listens to tens of thousands of hours of speech that nobody transcribed. From that audio it learns to turn every 20 ms of sound into a vector that says what is being said. After that, a few minutes of transcribed speech are enough to teach it to write the words down. The model that showed this works, and works well, is wav2vec 2.0 (Baevski et al. 2020), and its ideas run through every speech encoder since.

Here is why that matters. A recognizer trained the ordinary way needs audio paired with its transcript, and lots of it. Whisper, the recognizer of §36.4, learned from 680,000 hours of such pairs. Most of the world's roughly 7,000 languages have little or no transcribed speech. Yet nearly every language has plenty of untranscribed audio, from radio, podcasts, recorded stories, and phone calls. A method that learns from the audio and needs only a little text decides whether a language gets a recognizer at all.

The result that made the field take notice is easy to state. It is measured on LibriSpeech, the standard benchmark of read English audiobooks. LibriSpeech has 960 hours of transcribed training audio and two test sets, a "clean" one and a harder "other" one with less clear speakers. The score is the word error rate (WER), the number of words the recognizer substitutes, deletes, or inserts, divided by the number of words in the true transcript. wav2vec 2.0 was pretrained on 53,200 hours of untranscribed audiobook speech. It was then fine-tuned on ten minutes of transcribed audio, just 48 recordings. It reached 4.8 percent WER on the clean test set and 8.2 percent on the other. Months earlier, the best published result with that little labeled data was 16.3 and 25.2 percent, from a two-stage system by the same group (Baevski, Auli and Mohamed 2019).

§35.3 built the standard audio front end by hand. A log-mel spectrogram pools the spectrum into bands spaced the way the ear hears pitch, then takes a logarithm. That section ended with front ends that learn a few of their own settings, such as the filter shapes or the loudness compression. wav2vec 2.0 learns all of it. As §1.8 sketched, a stack of 1-D convolutions reads the raw samples and emits one vector every 20 ms, and a Transformer reads those vectors. Everything is learned from a task that needs no labels.

This section builds the whole model one part at a time. We count the frames the convolutions produce. We derive what fraction of the audio the masking hides. We show how a discrete codebook can be trained by gradient descent, and why each of the two losses has the form it has. Then we turn the model into a recognizer with ten minutes of labels, and count what the whole thing costs at real scale.

Hide a stretch of speech, then pick it out of a lineup

The training task fits in one sentence. Hide some stretches of the audio from a Transformer, and ask it to pick out what was hidden from a lineup of decoys taken from the same recording. Three ideas from earlier in the book combine to make it.

• Tell the true example apart from random decoys. word2vec learned word vectors by scoring the real neighbor of a word above randomly drawn words (§1.5).
• Hide part of the input and predict it from both sides. BERT trained a Transformer on text this way (§7.1). Masked autoencoders do the same on images (§28.5).
• Make "pick the true one" a softmax cross-entropy. Give each candidate $j$ a score $s_j$, the cosine similarity between the model's prediction and the candidate, divided by a temperature $\tau$. The loss is $-\log\big(e^{s_{\text{pos}}}/\sum_j e^{s_j}\big)$, the cross-entropy of choosing the true candidate. This is the InfoNCE loss of §7.2.

The naive version of this task fails on speech in three ways, and wav2vec 2.0 makes one change for each.

• Hide spans, not single frames. Speech changes smoothly over tens of milliseconds. One hidden 20 ms frame can be guessed by averaging its two neighbors, so the model would learn to interpolate, not to understand.
• Make the targets discrete. Suppose the model must match the hidden frame's exact vector. Then it can win by matching low-level detail, such as the microphone, the background hum, or the speaker's timbre. Forcing each target through a small codebook throws that detail away.
• Draw the decoys from the same utterance. An utterance is one recording of one person speaking. Decoys from other recordings could be rejected by voice or microphone alone. Decoys from the same recording share both, so only the content tells them apart.

Here is the whole task at the size of a napkin.

The model at a glance

wav2vec 2.0 has three learned parts, trained together. The paper trains two sizes, BASE with 95 million parameters and LARGE with 317 million. The table gives the shapes for one utterance of $n_{\text{samples}}$ samples at 16 kHz, in the BASE size. We call each output of the convolutions a latent frame, a 512-number summary of 25 ms of audio.

The context vector $\mathbf{c}_t$ is also projected to width 256, and the loss compares it with $\mathbf{q}_t$ at every masked step. One warning about letters. Here $\mathbf{q}_t$ is the quantized target of frame $t$, as in the paper, not an attention query. And $\mathbf{c}_t$ is the context network's output, not a codebook entry.

The convolutional feature encoder

§35.2 showed that the short-time Fourier transform is itself a convolution. A fixed bank of complex sinusoid filters, each 400 samples long, slides over the waveform in steps of 160 samples. wav2vec 2.0 replaces that one fixed layer with seven learned ones.

Two questions come first. How much audio does each latent frame see, and how many frames does a clip give? Two numbers answer them. A frame's receptive field is the number of input samples it depends on. The jump is the distance, in input samples, between the starts of neighboring frames. §0.8 derived how both grow through a stack of layers. Start with $r_0 = j_0 = 1$ for the raw samples. A layer with kernel $k_l$ and stride $s_l$ gives

In plain words, each new layer adds $k_l - 1$ extra taps, and each tap is one old jump further along the input. The stride multiplies the jump. An unpadded layer also maps $n$ inputs to $\lfloor (n - k)/s\rfloor + 1$ outputs.

How many frames does a clip give? A frame exists exactly when its whole 400-sample window fits inside the clip. Windows start every 320 samples. So a 1,000-sample clip has windows starting at samples 0 and 320, since the next one, at 640, would end at 1,039. That gives 2 frames, and $\lfloor (1000 - 400)/320 \rfloor + 1 = 2$. The general rule is worth remembering.

So long audio gives 50 frames per second, one per 320 samples, while a one-second clip gives 49. The paper quotes an output rate of "49 Hz", which is the count for one second, not the rate. Frame $t$, counting from 0, covers samples $320t$ to $320t + 399$, so its center sits at $20t + 12.5$ ms, not at $20t$. That half-frame offset matters whenever you align frames with phone labels or word timestamps, and "how many frames does one second give?" is a common warm-up question in interviews. The cell checks the table and the formula against a layer-by-layer count for every length up to 40,000 samples.

The 15.625 s and 20 s rows are the training crops of BASE (250,000 samples) and LARGE (320,000 samples). So the Transformer reads 781 and 999 frames per example.

The first layer deserves a second look. Its kernel of 10 samples at stride 5 spans 0.625 ms. That is far too short to resolve pitch or formants, the resonances of the vocal tract that shape each vowel (§35.1). So this layer is not a learned spectrogram. It is a bank of 512 tiny learned filters. The deeper layers combine their outputs over longer and longer stretches, the way the deeper layers of an image network combine edges into textures. Only after all seven layers does a frame see as much audio as one log-mel frame.

def conv_stack_geometry(kernels, strides, n_samples, dilations=None):
    raise NotImplementedError

The context network

After the feature encoder, the latent frames are layer-normalized and then go two ways. One copy, taken before any masking, feeds the quantizer, which makes the targets. The other copy is projected from 512 numbers to the model width $d$, partly masked, and fed to a Transformer encoder. That Transformer is the context network $g$, and it outputs $\mathbf{c}_1,\dots,\mathbf{c}_T \in \R^{d}$.

A Transformer needs to know where each frame sits, and wav2vec 2.0 tells it with a convolution instead of a table of position vectors. A 1-D convolution with kernel 128 slides over the frames, its output goes through a GELU, and the result is added to the input. A convolution's weights depend only on the offset between an output frame and the frames it reads, never on the absolute position. So this is a relative position signal, in the sense of §5.4. Its kernel covers 128 frames, 2.56 s, and the attention layers then reach across the whole utterance. In practice this means there is no table with a maximum length, so the model reads clips longer than any it was trained on.

The convolution is grouped. Its 768 channels are split into 16 groups of 48, and each output channel reads only the 48 input channels of its own group. That cuts the weights 16-fold, to $768\cdot(768/16)\cdot128 \approx 4.7$ million in BASE instead of 75.5 million.

The parameter counts are easy to check, and checking them shows where the weights live.

The totals, 95.0 and 317.4 million, match the paper's 95 and 317 million. The convolutional encoder holds only 4.2 million weights. But its early layers run at high frame rates, 3,199 outputs per second after the first layer, so it is far from free in compute. The real-scale example near the end counts its cost.

Masking spans of latent frames

Now the training task itself. Before the Transformer, a random set of frames is hidden. Each hidden frame is replaced by one learned vector, the same for every hidden position, after the projection to width $d$. The quantizer still reads every latent $\mathbf{z}_t$ as it was before masking, because that unmasked latent defines the target. Targets are needed only at masked frames, and the reference code quantizes only those.

The masking hides spans, and spans can overlap. A tiny case shows what that does.

What fraction of frames does this hide, and how long are the masked stretches? The tempting answers are $p\ell = 0.65$ and $\ell = 10$. Both are wrong, for the reasons the example showed.

In plain words, about half of the frames are hidden, in stretches that average almost 15 frames, about 0.3 s. That matches the paper's report of about 49 percent masked with a mean span of 14.7 frames (299 ms). The median run is still 10 frames. A run is a single span exactly when none of the $\ell$ frames after its start is another start, which has probability $(1-p)^{\ell} = 0.51$. So just over half of the runs have length exactly 10, and only a span clipped at the end of the sequence can be shorter.

The cell checks both formulas by simulation, under two samplers. One is the idealized model. The other is the sampler the reference code uses. It draws $\lfloor pT + u\rfloor$ distinct starts, with $u$ uniform on $[0, 1)$ to round at random, from the first $T - \ell$ positions, so that no span runs off the end.

Both samplers land on the closed forms. The masked fractions agree within about half a percentage point and the mean runs within about a fifth of a frame. The finite length and the edges cause the small gaps. HuBERT, in §35.5, uses $p = 0.08$ with the same $\ell$, which masks about 57 percent. The widget below draws one mask over the 781 frames of a BASE crop, with a tick above every start. Frames hidden by more than one span are drawn darker. New draw lays a fresh mask down from left to right, and the histogram and the numbers under it are averages over 300 draws.

Both answers follow from the theorem. With $\ell = 1$ a run is a geometric number of back-to-back starts, with mean $1/(1-p) = 1.33$ at $p = 0.25$. So almost every hidden frame has a visible neighbor on each side, and averaging the two neighbors nearly solves the task. With $\ell = 20$, half the frames are hidden near $p = 0.035$, and the mean run is about 29.7 frames, almost exactly twice the 14.7 at $\ell = 10$. At a fixed masked fraction the mean run grows in proportion to $\ell$, because the start probability needed falls roughly as $1/\ell$.

Why spans of 10? The paper's ablations answer it (Appendix A and Table 5). They used a reduced BASE setup with $p = 0.075$ and fine-tuned each model on 10 hours of labels. Each model was scored by its average WER on LibriSpeech's two development sets, clean and other, which are held out for tuning. Spans of 10 gave 7.97, against 8.33 for spans of 8, 8.19 for 12, and 8.43 for 15. Shorter spans make the task easier and the representation worse, and the authors report that spans of a single frame make the task trivial. Non-overlapping spans of fixed length 10 were clearly worse at 9.15, with $p$ lowered so that the same number of frames was masked. That suggests the variety of merged run lengths helps. The same ablation setup and the same score, average dev WER with 7.97 as the baseline, appear in every ablation quoted below.

import numpy as np

def sample_span_mask(T, p, ell, rng):
    raise NotImplementedError

def span_stats(p, ell):
    return p * ell, float(ell)       # naive placeholders, wrong once spans overlap

Discrete targets from a product quantizer

The targets are not the latent frames themselves. Each latent $\mathbf{z}_t$, as it was before masking, is turned into a discrete choice from learned codebooks, and the target $\mathbf{q}_t$ is built from the chosen entries. Why discrete? The paper's ablation, which we read in full after the losses, found that continuous targets let the model match detail that is useless for recognition. Quantized targets did not.

A single codebook big enough to name every sound would be expensive to search and hard to keep fully in use. Product quantization splits one big choice into $G$ independent small ones. Each of $G$ codebooks with $V$ entries contributes one choice, so together they name $V^G$ combinations. §7.4 used the same idea to compress vectors for search.

The hard part is the pick. Suppose each group takes the entry with the largest logit, $i_g = \arg\max_v l_{t,g,v}$. A small change in the logits does not change which entry wins, so the target does not move. The gradient of the loss with respect to the logits is then zero almost everywhere, and the codebook choice can never learn. The vector-quantized autoencoder of §32.5 hit the same wall with its discrete codes. It got past it with the straight-through estimator. That estimator copies the gradient of the quantized vector onto the unquantized one, as if quantizing were the identity. wav2vec 2.0 uses a cousin of that trick built on random sampling.

The Gumbel-softmax relaxation

The trick is the Gumbel-softmax, which §32.5 builds in full, with a sample worked by hand, for the image tokenizer of DALL-E. Here is the short version. It starts from the Gumbel-max trick of §0.4. Add independent standard Gumbel noise $\epsilon_v = -\log(-\log U_v)$, with $U_v$ uniform on $(0, 1)$, to each logit and take the argmax. The winner is then distributed exactly as $\operatorname{softmax}(\mathbf{l})$. So a random choice becomes a fixed function of the logits and some noise. The argmax still has zero gradient almost everywhere, so the relaxation replaces it by a softmax with a temperature. Two groups found it at the same time, Jang, Gu and Poole (2017) and, as the Concrete distribution, Maddison, Mnih and Teh (2017). wav2vec 2.0 uses its straight-through form.

In plain words, draw a noisy score for every entry and make a soft vote with a temperature. Use the hard winner going forward, and pretend you used the soft vote going backward. The winner does not depend on the temperature, since dividing every noisy score by the same positive number keeps their order. So the forward code is an exact draw from $\operatorname{softmax}(\mathbf{l})$ at every $\tau_{\text{g}}$. The temperature sets only how soft $\boldsymbol\pi$ is. As §32.5 explains, $\boldsymbol\pi$ tends to the one-hot $\mathbf{y}_{\text{hard}}$ as $\tau_{\text{g}} \to 0$ and to the uniform vector $\tfrac1V\mathbf{1}$ as $\tau_{\text{g}} \to \infty$. A small temperature stretches the gaps between the noisy scores, so the largest takes almost all the weight, and a large one shrinks every gap toward zero.

The gradient shows what each temperature costs. Differentiating the softmax, as in §0.2, gives

At high temperature $\boldsymbol\pi$ is spread out. The gradient is smooth and reaches every entry, but the soft vector the backward pass pretends to use is far from the hard one the forward pass used. At low temperature the two vectors agree, but the Jacobian is almost zero for most noise draws. A nearly one-hot $\boldsymbol\pi$ makes the bracket vanish, as for the saturated softmax of §4.3. For the rare draw where two noisy scores nearly tie, the Jacobian is huge, because of the $1/\tau_{\text{g}}$.

So straight-through Gumbel is biased. Its backward pass is the gradient of a different function from the one evaluated forward, and its average is not the gradient of the expected loss. It is a practical compromise, and the temperature trades that bias against variance. The cell measures both effects on a group of four entries.

The argmax frequencies match the softmax to about three decimals, which is the Gumbel-max trick at work. As the temperature falls, the soft sample approaches one-hot, and the mean largest entry rises from 0.47 to 0.95. At $\tau_{\text{g}} = 2$ the gradient scale is about the same for every draw, with a median of 0.34 and a 99th percentile of 0.37. At $\tau_{\text{g}} = 0.1$ the median draw passes almost no gradient (0.002), while one draw in a hundred passes more than 5. That spread is the variance that low temperatures cost.

wav2vec 2.0 therefore lowers the temperature during training, a schedule called annealing. It starts at $\tau_{\text{g}} = 2$, where every entry receives gradient. It multiplies the temperature by $0.999995$ after every update, until a floor of 0.5 for BASE or 0.1 for LARGE. Reaching the floor takes $\ln(0.5/2)/\ln 0.999995 \approx 277{,}000$ updates for BASE and $\ln(0.1/2)/\ln 0.999995 \approx 599{,}000$ for LARGE, so the anneal spans most of training.

The anneal never changes the codes. They are drawn from $\operatorname{softmax}(\mathbf{l})$ at every temperature, and only the gradient that trains the logits depends on $\tau_{\text{g}}$. This makes a good interview check, because a common wrong belief is that a lower temperature makes the codes themselves less random. In the widget, 256 frames each choose one of 16 entries. Drag the temperature, or press Anneal to lower it from 2 to 0.1. The top panel, one frame's soft vector, sharpens, while the bottom panel, how often each entry is chosen, stays put. The switch under the slider previews the diversity loss defined later in this section, which pushes the batch toward using every entry.

import numpy as np

def gumbel_quantize(logits, codebooks, tau, noise, hard=True):
    raise NotImplementedError

The objective

Every piece is now in place, so we can state the training objective precisely.

In plain words, at every hidden frame, pick the true target out of 101 candidates from the same utterance by cosine similarity. On the side, keep every codebook entry in use. The next two parts take the losses one at a time and justify each choice in them.

The contrastive loss, and why the distractors come from the same utterance

$\mathcal{L}_m$ is the InfoNCE loss of §7.2, so its gradient and the role of its temperature carry over unchanged. What is new is where the candidates come from, and that choice decides what the model learns.

Every frame of one utterance shares a speaker, a microphone, a room, and a noise floor. Suppose the distractors came from other utterances. The model could then pick the positive by asking which candidate sounds like this recording, which needs no knowledge of what was said. Distractors from the same utterance share all of those cues with the positive, so the cues cancel. Only the content of the hidden stretch separates the true target from the rest.

The paper tested this (Table 13 of its appendix). It replaced half of the 100 same-utterance distractors with distractors from other utterances in the batch. The average dev WER rose from 7.97 to 8.79, which the authors attribute to candidates from other utterances being easy to tell apart.

We can watch the shortcut happen in a toy world. Each utterance has one speaker. Each frame's input is a phone vector in one subspace, plus that speaker's vector in an orthogonal subspace, plus a little noise. Phones last 3 to 8 frames. A family of linear encoders, one for each angle $\theta$, multiplies the phone coordinates by $\cos\theta$ and the speaker coordinates by $\sin\theta$. So the "phone share" $\cos^2\theta$ runs from 1, a pure phone encoder, to 0, a pure speaker encoder. The encoder's output is also the target. The "context" vector is the average of a frame's two neighbors. We score each encoder with the contrastive loss under each choice of distractors.

Read the two columns as two different teachers. With same-utterance distractors, the loss is lowest, about 1.6, when the encoder keeps mostly phone information. It climbs past chance (3.04) as the encoder turns to the speaker, because then every candidate looks the same. With distractors from other utterances, the ranking flips. A speaker-heavy encoder reaches a loss near 0.04, far below the 1.1 of the pure phone encoder. Gradient descent would happily learn to recognize voices and ignore phones. Same-utterance sampling makes that shortcut useless.

One number in the table needs explaining. The pure phone encoder does worse with same-utterance distractors, 1.6 against 1.1. Phones recur within an utterance, so some same-utterance distractors share the positive's phone and tie with it.

Three further details complete the loss.

• Distractors come only from masked frames, just like the positive. The paper found that drawing them from any frame did not help and cost more compute.
• The reference code draws the $K$ distractor positions uniformly with replacement from the other masked steps. It drops a distractor whose target vector is identical to the positive's, since two frames can quantize to the same code, and an identical candidate is not an alternative.
• The cosine, rather than a dot product, keeps every score between $-1/\tau$ and $1/\tau$. So neither network can lower the loss by inflating vector lengths.

import numpy as np

def sample_distractors(mask, K, rng):
    raise NotImplementedError

def contrastive_loss(c, q, mask, neg_idx, tau):
    raise NotImplementedError

The diversity loss, and why it is the entropy of the average

The contrastive task trains the quantizer too, since the targets are its output. Its gradient also flows through the quantizer into the feature encoder, and the paper found that cutting this path hurt (8.41 against 7.97, Table 13). Left alone, a learned quantizer tends to use only a few of its entries. §32.5 called this codebook collapse. Here it would make the targets nearly constant, carrying little information about the frame. The diversity loss pushes back by rewarding even use of every codebook entry.

The key detail is which entropy. Recall that entropy $H(\mathbf{p}) = -\sum_v p_v \log p_v$ measures how spread out a distribution is (§0.5). It is 0 for a one-hot vector and $\log V$ for a uniform one. A tiny case shows which distribution's entropy we want.

So $\bar{\mathbf{p}}_g$ is the noise-free softmax averaged over the batch, and $\mathcal{L}_d$ is minus its entropy, scaled by $1/(GV)$. Maximizing the entropy of the average asks that, across many frames, every entry is used about equally often. It does not ask any single frame to be uncertain. The alternative, the average of per-frame entropies, would push every frame toward a uniform choice. That is the opposite of what a quantizer should do, since a good quantizer is confident about each frame and diverse across frames.

A natural way to report codebook use is the perplexity $e^{H(\bar{\mathbf{p}}_g)}$, the effective number of entries in use. It ranges from 1, total collapse, to $V$, uniform use. At the uniform optimum, $\mathcal{L}_d = \frac{1}{GV}\cdot G\cdot(-\log V) = -\log V/V$, which is $-0.018$ for $V = 320$. The reference code optimizes $(GV - \sum_g e^{H(\bar{\mathbf{p}}_g)})/(GV)$ instead, which is 0 at uniform use and $1 - 1/V$ at total collapse. It has the same minimizer, since $e^{H}$ grows with $H$, and the paper calls the two equivalent. Their gradients differ, though. The derivative of each group's term with respect to its entropy is $e^{H}$ times larger in the code's form, from 1 at collapse to $V = 320$ at uniform use. So the published $\alpha = 0.1$ belongs to the code's form, and the same $\alpha$ on $\mathcal{L}_d$ as written gives a much weaker pull.

The third row is the one to remember. Every frame is certain of its code, yet the codebook is fully used and $\mathcal{L}_d$ is at its minimum. A loss built from per-frame entropies would have called this the worst case.

How much does it matter? In the paper's reduced ablation setup, the average dev WER was 8.48 with $\alpha = 0$, 8.34 with $\alpha = 0.05$, 7.97 with $\alpha = 0.1$, and 8.58 with $\alpha = 0.2$, where training became slightly unstable. So the diversity loss buys about half a point of WER, and its weight needs tuning in both directions. In practice, watch the codebook perplexity during training, since a fall toward a handful of codes is the first sign of collapse.

import numpy as np

def diversity_loss(logits, mask=None):
    raise NotImplementedError

Why quantize the targets but not the inputs

Now the design choice from the start of the section can be judged with data. The paper trained four variants in its reduced setup and fine-tuned each on 10 hours of labels (Table 4, average WER on dev-clean and dev-other over three seeds).

Quantizing the input hurts most. The 102,400 codes throw away information the context network needs to build good representations, which is why the earlier two-stage systems lagged. Quantizing the target helps, and the paper gives the mechanism. With continuous targets, the accuracy at picking the true target rose from 62 to 78 percent, yet the downstream WER got worse. The continuous targets kept details of the particular recording, such as the speaker and the background, that make the task easier and the representation less useful.

So the quantizer acts as an information bottleneck on what counts as the right answer, while the input keeps everything. There is direct evidence that the bottleneck keeps phonetic content. In its Appendix D, the paper counts which phones co-occur with each code on TIMIT, a small English dataset labeled phone by phone, and many codes specialize to particular phones. "Why quantize the targets but not the inputs?" is one of the questions interviewers ask most about this model.

Before moving on, go back to the speaker shortcut of the toy simulation. It has an exact version, and working it out shows how much the shortcut would pay at the real $K$ and $\tau$.

import numpy as np

def shortcut_losses(a, b, rho, K, m, tau):
    raise NotImplementedError

How it is trained, and what keeps it stable

The objective is only half of a recipe. The rest is fixed by the pretraining setup the paper describes and by the released fairseq configurations. These settings matter as much as the objective when a run goes wrong. They fall into three groups, the optimizer and its schedule, two regularizers on the feature encoder, and dropping whole Transformer blocks.

The optimizer. Training uses Adam (§0.6). The learning rate rises linearly over the first 8 percent of updates, a phase called warmup, to a peak of $5\times10^{-4}$ for BASE and $3\times10^{-4}$ for LARGE. It then decays linearly to zero. So warmup lasts 32,000 of BASE's 400,000 updates. LARGE on LibriSpeech trains for 250,000 updates and first meets its peak at update 20,000. The released configurations add $\beta_2 = 0.98$ and Adam's constant $\epsilon_{\text{Adam}} = 10^{-6}$ (not the Gumbel noise $\boldsymbol\epsilon$). They also add a weight decay of 0.01 applied directly to the weights rather than through the gradient, the AdamW form of §0.6. They train in 16-bit floating point (fp16) with dynamic loss scaling (§8.5). That multiplies the loss by a large factor so that small gradients do not round to zero, and backs the factor off when values overflow.

Two regularizers on the encoder. Pretraining on LibriSpeech's 960 hours, which made the released BASE model, adds two regularizers on the feature encoder. Both act on the output of its last convolutional block, before the layer normalization that both branches read.

• The gradient flowing back into the encoder's output is multiplied by 0.1 (fairseq calls this feature_grad_mult). The forward pass is unchanged. Every encoder parameter receives its gradient through that output, so each one gets a tenth of its true gradient.
• An L2 penalty $\lambda\cdot\operatorname{mean}(\mathbf{h}^2)$ is added to the loss, where $\mathbf{h}$ holds all activations of that last block in the batch. The released BASE configuration uses $\lambda = 10$.

The LibriVox LARGE recipe uses neither. It stabilizes the encoder with layer normalization in every block on a normalized waveform, as described after the encoder's definition. Its released configuration also normalizes the input of each Transformer sublayer, the pre-LN arrangement of §5.2, and turns dropout off.

The paper gives no reason for the two encoder regularizers beyond calling them regularization. The penalty has a clear job. Layer normalization divides each frame by its own standard deviation. So, up to the small constant inside it, neither loss can see how large the encoder's output is. §0.7 makes the same point about weights before a normalization. Nothing in the objective holds that size in place, and fp16 cannot represent a value above 65,504. The penalty supplies the missing pull toward small activations. It is computed after the gradient factor, so its own gradient into the encoder is scaled by 0.1 too. The gradient factor is subtler, and the box below works out what it does under Adam. The practical lesson is that a 0.1 gradient factor under Adam is not a learning rate ten times smaller.

The cell runs Adam on one parameter with and without the factor and compares the distance moved with the formula from the box.

The simulated ratios match the steady-gradient formula. Only parameters with gradients near $\epsilon_{\text{Adam}}$ or below feel the factor fully.

Dropping whole blocks. The third ingredient skips Transformer blocks at random, a method called LayerDrop (Fan, Grave and Joulin 2019). At every training step, each block is skipped entirely with probability $p_{\text{ld}}$, for the whole batch. Otherwise it is used unchanged, with no rescaling. It is a close cousin of stochastic depth, which skips residual blocks at random in vision Transformers (§27.4), and it keeps the model from relying on any single block. Pretraining uses $p_{\text{ld}} = 0.05$ for BASE, 0.2 for LARGE on LibriSpeech, and none for LV-60k. Fine-tuning uses 0.05 for BASE and 0.1 for LARGE.

The real way, in PyTorch

Here is the whole pretraining step at toy scale, with every piece of the section in place. That includes the training recipe, meaning the gradient factor and activation penalty on the encoder, LayerDrop, and AdamW with its warmup schedule. The released models differ in size, not in structure.

The printed contrastive loss sits near chance, $\log 9 \approx 2.20$, and barely moves. Two things hold it there. Three updates at the start of a 32,000-update warmup use a learning rate below $10^{-7}$, and random noise has no content for the context to predict. On real speech, over a real schedule, the loss falls steadily below chance as the model learns. With the Hugging Face library, a pretrained model gives you the latents and every layer's context vectors directly.

Fine-tuning with a few minutes of labels

After pretraining, the quantizer and the projection heads are thrown away. A new, randomly initialized linear layer maps each context vector $\mathbf{c}_t$ to scores over characters. For LibriSpeech there are 29 character tokens plus a word-boundary token.

The model is then trained with connectionist temporal classification (CTC), which §36.2 derives in full. For now, here is what it does. CTC takes, for every frame, probabilities over the characters plus a special blank symbol that means "no new character here". It returns the log-probability of the transcript, summed over every way of spreading the transcript's characters across the frames. So it needs only the transcript, never a label for each frame.

Three choices protect the pretrained weights from overfitting ten minutes of data.

• The convolutional feature encoder stays frozen for the whole of fine-tuning.
• For the first 10,000 updates only the new output layer trains, so that its random early gradients do not wreck the Transformer.
• The latent frames are masked during fine-tuning too, in the spirit of SpecAugment (§35.3), which hides random stretches of time and bands of frequency. Spans of 10 time steps are replaced by the mask vector, and spans of 64 channels are set to zero. The paper reports that this delays overfitting and helps most with the smallest labeled sets.

Decoding then adds a language model trained on LibriSpeech's text, combined with the acoustic scores by beam search (§36.5). The paper tries two. One is a 4-gram model, which predicts each word from the three before it by counting. The other is a 20-block Transformer.

The small labeled sets come from Libri-light (Kahn et al. 2020), a benchmark that carves 10-minute, 1-hour, and 10-hour training subsets out of LibriSpeech. Papers that report on these subsets use the same splits, so their results compare directly. Libri-light also supplies the unlabeled audio, about 60,000 hours of LibriVox recordings, which wav2vec 2.0's preprocessing trims to the 53,200 hours of LV-60k.

The paper's headline table is worth reading as a curve. All rows are LARGE pretrained on LV-60k, the 53,200 hours of LibriVox, and decoded with the Transformer language model (Tables 1 and 2). The dev columns are the sets used for tuning, and the test columns are the final scores.

Two readings matter in interviews. First, the curve is flat. Going from 10 minutes to 960 hours multiplies the labels by 5,760, yet cuts test-other WER only from 8.2 to 3.3. The representation already holds most of what recognition needs.

Second, the language model does much of the work at the low end. Without any language model, the same 10-minute model scores 40.2 and 38.7 percent on test-clean and test-other (Table 9). It hears the sounds but cannot spell, writing "al" for "all" and "wil" for "will" (Table 11's most frequent errors). With 960 hours of labels the numbers without a language model are 2.2 and 4.5. So when you see a low-resource result, ask which language model it used. One caution if you read the paper. Its results section quotes 5.2 and 8.6 for the 10-minute model, which disagrees with its own Table 1, while the abstract and the conclusion use the table's 4.8 and 8.2.

The cell redoes the counts.

Scaling out, from 53 to 1,600 languages

The method transfers across languages with no change, and that is where it matters most. XLSR-53 (Conneau et al. 2020) pretrained one LARGE model on 56,000 hours in 53 languages, with one quantizer shared across them. It found that the learned discrete units are shared between languages, more so between related ones.

XLS-R (Babu et al. 2021) scaled to 436,000 hours of public speech in 128 languages, with models of 0.3, 1, and 2 billion parameters. MMS (Pratap et al. 2023) pretrained wav2vec 2.0 models of about 300 million and 1 billion parameters on about 491,000 hours in 1,406 languages. Its data combined public corpora with new recordings of read religious texts that reach over a thousand languages. From them it built a single recognizer for 1,107 languages and a language identifier for 4,017. Its abstract reports more than halving Whisper's word error rate on 54 languages of FLEURS, a benchmark of the same read sentences in 102 languages. It did so while training on a small fraction of Whisper's labeled data.

The line continues at the frontier. Omnilingual ASR (Omnilingual ASR team 2025) pretrained a 7-billion-parameter wav2vec 2.0 encoder on 4.3 million hours of speech in more than 1,600 languages. It then put a language-model-style decoder on top, so that a new language can be added from a handful of examples. Five years after wav2vec 2.0, its objective still trains some of the largest open speech encoders.

In practice, for a language with ten hours of transcripts, starting from one of these checkpoints is the default recipe.

Where it breaks, and what came next

The method has known failure modes.

• Domain shift. A model pretrained on read English audiobooks transfers less well to conversational or noisy speech. Adding unlabeled audio from the target domain to pretraining closes 66 to 73 percent of the gap between fine-tuning on in-domain and on out-of-domain labeled data (Hsu et al. 2021). That finding is part of what pushed later work toward more varied data, such as WavLM's mix of sources and the multilingual models above.
• Codebook use. With too small a diversity weight, codebook use drops and WER rises (8.48 at $\alpha = 0$).
• Distractors and noise. The contrastive task is sensitive to where its distractors come from. Taking half of them from other utterances hurt (8.79 against 7.97), and doubling $K$ to 200 did not help (8.12). The quantizer needs its Gumbel noise, too, since training without it was worse and unstable across seeds (8.73).
• The top layers. The top Transformer layers specialize to the pretraining task (Pasad et al. 2021). So they are not the best features for downstream use. §35.5 measures this layer by layer with the probing methods of §23.2.
• Few labels lean on the language model. CTC fine-tuning on few labels depends heavily on an external language model. At 10 minutes of labels the error was about 40 percent without one.

Read more

• Baevski, Zhou, Mohamed and Auli (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. The model of this section. Read Sections 2 and 3 for the method, and Appendices A and F for the masking statistics and the ablations quoted here.
• van den Oord, Li and Vinyals (2018). Representation Learning with Contrastive Predictive Coding. The origin of InfoNCE and of contrastive prediction of latent audio frames.
• Schneider, Baevski, Collobert and Auli (2019). wav2vec: Unsupervised Pre-training for Speech Recognition. The first wav2vec, a convolutional encoder on the raw waveform trained contrastively.
• Baevski, Schneider and Auli (2019). vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations. The quantized predecessor, with both Gumbel-softmax and k-means quantizers.
• Jang, Gu and Poole (2017). Categorical Reparameterization with Gumbel-Softmax, and Maddison, Mnih and Teh (2017), The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. The relaxation and the straight-through estimator, with their bias and variance discussed.
• Babu et al. (2021). XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. wav2vec 2.0 at 2 billion parameters and 128 languages.
• Pratap et al. (2023). Scaling Speech Technology to 1,000+ Languages. MMS, the same recipe across 1,406 languages, with the comparison against Whisper.
• Omnilingual ASR team (2025). Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages. A 7-billion-parameter wav2vec 2.0 encoder pretrained on 4.3 million hours, with CTC and decoder recognizers built on it. Its part on self-supervised pretraining shows how the objective of this section was scaled.

Watch next

▶

▶

▶