This section is about training a speech encoder on audio alone, with no transcripts, by filling in blanks. You hide, or mask, some stretches of a recording and train a Transformer to say what was hidden. Text models already learn this way, by predicting hidden words. Speech has no words to name, so the method first makes up labels for it. It groups short slices of audio into clusters, and the model learns to predict the cluster number of each hidden slice. The made-up labels can be poor, and the model still learns the sounds that speech is built from. Why that works is the main idea of this section.

It matters for three reasons. Masked prediction of made-up labels is the standard way to pretrain a speech encoder when transcripts are scarce. The cluster numbers themselves become the discrete speech tokens that spoken language models read and write. And "why do bad targets work?" is a question interviewers ask about self-supervised learning.

Here is the problem the method solves. Text comes already cut into tokens, so a text model can hide a token and predict it with an ordinary softmax over the vocabulary. That is masked language modeling, the training task of BERT (§7.1). The speech model of the previous section, wav2vec 2.0 (§35.4), also hid stretches of audio. But it had no vocabulary, so it had to learn its targets inside the model. That took a trained quantizer (a layer that maps each frame to one of a fixed set of codes), a temperature annealed over hundreds of thousands of updates, an extra loss to keep the quantizer from using only a few codes, and 100 decoy frames per masked frame.

So try a shortcut. Make up a vocabulary for speech, almost any vocabulary, and train exactly as for text. Hsu et al. (2021) did that. They summarized every 10 ms of audio by 39 numbers that describe its spectrum, grouped those summaries into 100 clusters, and trained a Transformer to predict the cluster of each masked frame. The clusters are poor stand-ins for phones, the 40 or so basic sounds of English such as the /k/ in "cat". Knowing a frame's cluster removes only about a quarter of the uncertainty about its phone. Yet the model, HuBERT (Hidden-Unit BERT), matched or beat wav2vec 2.0 at almost every amount of transcribed speech the authors tried for the final recognizer, from 10 minutes to 960 hours.

This section builds HuBERT and answers the questions it raises. What exactly is the loss? How does the clustering algorithm, k-means, make the units, and why does it stop? How do you measure whether units are any good? Why do consistent but wrong units still teach phones? How does clustering the model's own layers make better units? What did the successors (WavLM, data2vec, BEST-RQ, w2v-BERT) change? And which layer of the finished encoder should you use for which job? The same idea, hide part of the input and predict something about it, also trains image encoders (masked image modeling) and video encoders (§29.4).

BERT for speech needs a vocabulary

HuBERT is built from two pieces of earlier models. The first is wav2vec 2.0's front end (§35.4). Seven strided 1-D convolutions read 16 kHz audio and emit one 512-number vector every 320 samples, that is, every 20 ms. Each vector is a frame, and it summarizes 400 samples (25 ms) of sound. One second of audio becomes 49 frames. Some runs of frames are masked, hidden from the Transformer by replacing each with one learned mask vector, and the Transformer reads the result. HuBERT keeps this front end exactly. Its BASE and LARGE models also keep wav2vec 2.0's Transformer sizes.

The second piece is masked language modeling (§7.1). Hide some tokens and run a Transformer over the whole sequence. At each masked position, score the model by the cross-entropy of the true token, the negative log of the probability the model gave it. Only the masked positions are scored. HuBERT keeps this too. The missing piece is the token.

A speech frame has no token. Two recordings of the same word never give the same numbers. There is no dictionary of sound units, and nothing marks where one sound ends and the next begins. HuBERT's answer is to label every frame before training starts. A clustering algorithm runs once, offline. It groups frames that look alike and gives each frame the number of its group. These numbers are the units. The paper calls them hidden units, the H of HuBERT, because no one observes them. Write $[K]$ for the set $\{1, \dots, K\}$ of unit numbers. Frame $t$ gets a unit $u_t \in [K]$, and the Transformer learns to predict $u_t$ at masked frames from the visible frames around them.

The first clustering runs on MFCCs (mel-frequency cepstral coefficients). An MFCC vector is a compact summary of the spectrum of one 25 ms slice of audio, computed every 10 ms. §35.3 builds it from the log-mel spectrogram. It has 13 coefficients plus their first and second differences over time, 39 numbers in all. MFCCs come at 100 per second and the encoder's frames at 50, so each encoder frame takes the unit of every second MFCC frame.

The picture holds the two facts the rest of the section proves. The loss must sit on masked frames, so that the label can only be inferred from context. And the labels must be consistent, the same function of the sound everywhere in the data, even if that function is far from the true phones.

The HuBERT objective

Here is the loss in words before symbols. The Transformer produces one output vector per frame. A learned matrix projects it to a smaller space. The model keeps one learned vector per unit, called a codeword embedding. It scores every unit by the cosine similarity between the projected output and that unit's codeword. Cosine similarity is the dot product of two vectors divided by both their lengths. It is a number between $-1$ and $1$ that depends only on the angle between them. The scores are divided by a small temperature $\tau$, which sharpens the softmax, and turned into probabilities. The loss is the cross-entropy of the true unit, counted only at masked frames.

Now the symbols. Write the frames of one utterance as $\mathbf{Z} = (\mathbf{z}_1, \dots, \mathbf{z}_T)$, with $T$ the number of frames, and their units as $u_1, \dots, u_T$, made beforehand by clustering. Choose a set of masked frames $\mathcal{M} \subset [T]$ by the span masking of §35.4. It picks random start frames and hides a run of frames from each start. Replace each masked frame by a learned vector $\mathbf{z}_{\text{mask}}$ to get the corrupted input $\tilde{\mathbf{Z}}$. The Transformer maps $\tilde{\mathbf{Z}}$ to outputs $\mathbf{o}_1, \dots, \mathbf{o}_T \in \R^d$.

In plain words, project each output frame, compare it by cosine with one learned vector per unit, multiply by $1/\tau = 10$, and pay the log-loss of the true unit, but only where the input was hidden.

For HuBERT BASE the widths are $d = 768$ and $d_e = 256$ (Table I of the paper). Dividing cosines by $\tau = 0.1$ is the same design as wav2vec 2.0's logits and the contrastive losses of §7.2. Every logit lies in $[-10, 10]$. So no frame can push its loss toward zero by growing a vector's length, and the softmax cannot overflow.

The real difference from wav2vec 2.0 is the denominator. There, the true target competed with 100 decoys sampled from the same utterance. Here, it competes with every unit, a fixed vocabulary of $K$ classes. The loss is a plain $K$-way classification, with no sampling, no quantizer, and no extra loss to keep codes in use. That is what makes HuBERT simpler and steadier to train, and it is the first contrast an interviewer expects you to draw between the two models.

To use the model for speech recognition, throw away the projection and the codewords. Add a fresh softmax layer over 26 letters, space, apostrophe, and a blank symbol. Then fine-tune the Transformer with CTC, a loss that sums over every way the letters could line up with the frames (§36.2 derives it). The convolutional front end stays frozen.

The cell checks these numbers. It also adds made-up cosines for the four visible frames, to show what the other two choices of $\alpha$ would compute.

The visible frames have a far smaller loss (0.146 against 0.817), and that is a warning sign. A visible frame is easy, because the model can read it and reproduce whatever the clustering did to it. We will see that this easy part is worse than useless when the units are poor.

import numpy as np

def hubert_loss(o, A_proj, E, targets, mask, tau=0.1, alpha=1.0):
    raise NotImplementedError

Discovering units with k-means

The units come from k-means, about the simplest clustering algorithm there is. It represents each cluster by one point, its centroid, and labels every frame with its nearest centroid. §7.4 teaches it in full, with an example worked by hand and the proof that it stops, where it splits a database of vectors into cells for fast search. Here is what this section needs, in the notation of frames and units.

Each step can only lower $J$ or leave it alone. The assignment step gives every frame its closest centroid, so no term of $J$ can grow. The update step cannot raise $J$ either, because the mean of a set of points is the one point with the smallest total squared distance to them. Because of the tie rule, a frame changes cluster only when its new centroid is strictly closer, so every change of the partition strictly lowers $J$. So no partition can repeat. There are finitely many partitions, at most $K^n$, so the algorithm stops. §7.4 proves both facts in full.

Stopping is all that is guaranteed, not a good result. Where it stops depends on the starting centroids, and a bad start can trap it far from the best partition. For example, two centroids can end up sharing one big cluster while two small, distant clusters are forced to share one centroid. Finding the global minimum is NP-hard in general, meaning no known algorithm solves every instance fast. So practice relies on good starting points and several restarts.

The standard starting rule is k-means++ seeding (Arthur and Vassilvitskii 2007). Pick the first centroid uniformly from the data. Pick each next one at random with probability proportional to $D(\mathbf{x})^2$, the squared distance from $\mathbf{x}$ to the nearest centroid chosen so far. Far-away regions, the ones a uniform start tends to miss, become likely picks. The paper proves that the seeding alone, before any Lloyd step, has expected objective at most $8(\ln K + 2)$ times the optimum. We state this without proof. Lloyd's steps can only lower it further, and the best of several independent seedings is rarely bad.

HuBERT used scikit-learn's MiniBatchKMeans, which updates the centroids from batches of 10,000 frames instead of the full data. It used k-means++ seeding with 20 random starts. For the later iterations, which cluster 768-dimensional Transformer features, it fitted k-means on a random 10 percent of the training audio. Mini-batch updates lose the guarantee that $J$ never rises, because each step sees only a sample. In exchange they make clustering tens of millions of frames affordable. In an interview, "prove that k-means converges, then say why it still needs restarts" is a standard pair of questions. The argument above, proved in full in §7.4, answers the first, and the paragraphs on restarts and seeding answer the second.

import numpy as np

def kmeans(X, K, n_iter, rng, init=None):
    raise NotImplementedError

The synthetic speech we will cluster

To watch all of this happen, we need frames whose true phones we know. The next cell makes 40 utterances of 100 frames each. An utterance is a run of phone segments over 8 phones, each segment 3 to 9 frames long (60 to 180 ms at 20 ms per frame). A frame is a 12-number vector. The phone decides only 4 of its directions. The other 8 directions carry large frame-local nuisance, variation that changes from frame to frame and says nothing about the phone, like breath noise or jitter in loudness. A random rotation mixes the two, so no coordinate is pure phone. Real MFCCs also carry much variation that is not about the phone, and that is the property these stand-ins copy. The speaker argument adds one offset shared by a whole utterance. It is off here, and the widget below lets you turn it on.

The objective falls from about 81,000 to 59,000 and never rises, as Lloyd's algorithm guarantees. Bad starting points need a different picture, one big cluster and four small, distant ones.

Uniform seeding never finds the five clusters. Its centroids mostly land in the big cluster. Lloyd's algorithm splits that cluster, and the small ones end up sharing centroids, at 8 to 14 times the optimal objective (best 7.9, median 13.7). k-means++ finds the optimum in 13 of 20 runs and fails in the rest, which is why HuBERT takes the best of 20 starts.

Measuring units with purity and PNMI

How good are the units we just made, units1? To answer that, we need the true phone of every frame. For real speech, the HuBERT authors got phone labels by forced alignment. A supervised recognizer is given the transcript and finds the time span of each phone in the audio. These labels serve only to evaluate units, never to train.

Three numbers compare units with phones. Phone purity asks whether each unit is mostly one phone. Cluster purity asks whether each phone mostly lands in one unit. PNMI, phone-normalized mutual information, asks what fraction of our uncertainty about the phone disappears once we are told the unit. Uncertainty here is entropy, $H(y) = -\sum_i p(i)\log p(i)$, the average surprise of a random label, measured in nats with natural logarithms (§0.5). Given phones $y_t \in [P]$ and units $u_t \in [K]$ for $t \in [n]$, all three come from one table that counts how often each pair occurs.

In plain words, phone purity rewards units that each hold one phone, cluster purity rewards phones that each use one unit, and PNMI measures how much the unit tells you about the phone, counting every frame and not only each unit's majority.

All three numbers depend on $K$, the number of units, and that is a trap. Split every unit in two and phone purity can only rise or stay, since part of a unit is at least as pure as the whole. With one unit per frame, phone purity is 1. PNMI is no protection, because one unit per frame also gives $H(y\mid u) = 0$ and PNMI of 1. Cluster purity moves the other way and collapses. So compare unit inventories only at the same $K$, on held-out frames, and read the two purities together. HuBERT's paper does this. It compares features only at equal $K$ (100, 500, or 1,000) and scores them on LibriSpeech's development set, audio the k-means never saw. At a fixed $K$, PNMI is the most useful single number.

The paper's numbers set the scale. k-means on MFCCs gives (cluster purity, phone purity, PNMI) of (0.099, 0.335, 0.255) with 100 units and (0.031, 0.356, 0.287) with 500 units. A quarter of the phone uncertainty, as promised. On features from a HuBERT layer, the same algorithm reaches PNMI of about 0.6 to 0.7, as we will see.

import numpy as np

def unit_quality(phones, units):
    return 1.0, 1.0, 1.0      # placeholder

Here is our version for the rest of the section, applied to the synthetic units at several values of $K$.

With 16 units, PNMI is 0.136, worse than real MFCC clusters. Raising $K$ from 8 to 256 lifts phone purity from 0.26 to 0.56 and PNMI from 0.12 to 0.50, while cluster purity collapses to 0.04. Most of that rise comes from $K$ itself, not from better clustering. Part of it is pure estimation error. With 4,000 frames and 256 units, the upward bias of the estimate alone is about $7\cdot255/8000 = 0.22$ nats, a tenth of $H(y) = \ln 8 = 2.08$, so about 0.1 of the 0.50.

The widget runs k-means on a smaller synthetic set you can see, 720 frames of six phones from 12 utterances, with heavy frame-local nuisance. k-means groups them into $K = 12$ units. Each dot is a frame, drawn in the two directions where the frames spread most and colored by its true phone, and each cross is a centroid. The table shows how each unit's frames split across phones, so a pure unit is a column with one dark cell. You can switch between three kinds of features. MFCC-like frames are raw frames as above. 5-frame mean averages each frame with its four neighbors. Layer-like features come from one round of the refinement we build later in this section. The widget always shows the units k-means converges to, and Run k-means replays the steps from the k-means++ seeds. The slider sets the speaker variation, an offset shared by every frame of an utterance. Besides the phone PNMI, the widget shows a speaker PNMI, the same formula with the utterance in place of the phone. It is high when the units also encode who is speaking.

Why noisy units still teach

Now the central question. MFCC units have PNMI of about 0.25. Why does predicting them teach a model anything about phones?

One experiment answers half of it. HuBERT's Table V trains BASE models for 100,000 steps with three values of $\alpha$ and several kinds of targets. It then fine-tunes each on 10 hours of transcribed speech. It reports the word error rate (WER), the share of words the recognizer substitutes, deletes, or inserts, as a percentage of the words in the true transcript. LibriSpeech, the standard English benchmark, is 960 hours of read audiobooks with transcripts. Its development and test sets each come in a "clean" half and an "other" half with speakers that are harder to recognize. The WERs below are on dev-other. The last row of the table uses chenones, supervised labels from a conventional recognizer that is given the transcript (8,976 context-dependent letter states).

With MFCC targets, the model trained only on visible frames is useless, at 96 percent WER. Even mixing in half visible loss nearly doubles the error. The six-frame example hinted at why. A visible frame's MFCC unit is a fixed function of that frame, and the model can see the frame. The cheapest way to predict it is to learn the clustering itself, a function of 25 ms of audio, and nothing in that task rewards looking at the neighbors. Hiding the frame removes the shortcut.

As the targets improve, the gap closes. With supervised labels, the visible loss stops hurting, because imitating the labeler is then imitating something worth learning.

The other half of the answer is why the masked model learns phones from targets that are not phones. Start with a small case.

The general statement needs names for the pieces. Let $P$ be the number of phones, $y_t \in [P]$ the true phone and $u_t$ the unit of frame $t$, and $\tilde{\mathbf{Z}}$ everything the model sees when frame $t$ is masked. Think of the labeling as a noisy channel that receives a phone and sends out a unit. Describe it by its channel matrix $\boldsymbol\Gamma \in \R^{K\times P}$, with $\Gamma_{jk} = \mathbb{P}(u_t = j \mid y_t = k)$, so each column is the distribution of units for one phone. In the example, the columns are $(0.6, 0.3, 0.1)$ and $(0.1, 0.3, 0.6)$. Write $\mathbf{p}_y(\tilde{\mathbf{Z}}) \in \R^P$ for the vector of phone probabilities $\mathbb{P}(y_t = k \mid \tilde{\mathbf{Z}})$ and $\mathbf{p}_u(\tilde{\mathbf{Z}}) \in \R^K$ for the unit probabilities $\mathbb{P}(u_t = j\mid\tilde{\mathbf{Z}})$.

In the example, the singular values of $\boldsymbol\Gamma$ are 0.82 and 0.50, so an error of 0.01 in the unit probabilities can become an error of up to 0.02 in the phone probabilities.

Now connect this to training. Cross-entropy is minimized, in expectation, by the true conditional distribution, because its excess over the minimum is a KL divergence (§0.5). So a model that drives $\mathcal{L}_m$ to its floor outputs $\mathbf{p}_u(\tilde{\mathbf{Z}})$ at every masked frame. By the theorem, that output contains $\mathbf{p}_y(\tilde{\mathbf{Z}})$, everything the context says about the phone of the masked frame, behind a fixed linear map. The model never sees a phone label, yet the best solution to its task computes phone probabilities from context.

In plain words, masked prediction needs targets that are consistent and fine enough, not correct. Wrong but consistent units are a fixed, invertible scrambling of the phones, and undoing a fixed scrambling is a job for a linear layer. The theorem also says exactly what breaks the lesson.

• Too coarse. If two phones always get the same units, $\boldsymbol\Gamma$ loses rank, and nothing forces the model to tell them apart. Fewer units than phones guarantees this. That is one reason HuBERT clusters into 100 or 500 units, well above the roughly 40 phones of English.
• Inconsistent. If the labeling changes between utterances, there is no single $\boldsymbol\Gamma$, and averaging different channels blurs them toward a matrix of low rank. Running k-means separately on each utterance would give sensible clusters with meaningless numbers.
• Leaky. If the unit depends on something else the context reveals, assumption (i) fails. The speaker and the recording conditions (microphone and room) are the usual culprits. They stay constant across an utterance, so the context reveals them. MFCC clusters depend on them, so the model is rewarded for encoding them too. WavLM, later in this section, attacks this by mixing utterances during training.
• Poorly conditioned. A noisy channel can have full rank and still have a small $\sigma_P$. Then the model must predict the units very precisely before their phone information is usable. This is the price of bad targets with finite data and a finite model, and it is why better units still help a lot in Table V, even though the theorem says MFCC units suffice in the limit.

The assumptions are idealizations, so test the prediction on our synthetic data. There assumptions (i) and (ii) hold by construction for every kind of target below, because each frame's label depends only on its own phone and its own noise. The theorem says exactly what to measure. Fit a masked predictor of the units, read phones out of its outputs through the fixed map $\boldsymbol\Gamma^{+}$, and see how much phone accuracy survives.

The masked predictor is linear. It reads the mean of a frame's four neighbors, never the frame itself, and is fitted by least squares onto one-hot vectors of the targets, a linear stand-in for a classifier. Its outputs play the role of $\mathbf{p}_u$. The cell counts $\boldsymbol\Gamma$ on other utterances and computes $\mathbf{p}_y = \boldsymbol\Gamma^{+}\mathbf{p}_u$, as in the theorem. For a sharper test it makes ten times more synthetic speech with the same phones (the first 40 utterances are the ones above) and labels it with the same 16 centroids. The predictor fits on 100 utterances, $\boldsymbol\Gamma$ is counted on the next 100, and accuracy is measured on the last 200.

Why not train a second classifier, a probe, on the predictor's outputs instead? Because here it would prove nothing. A linear predictor with 16 outputs that reads a 13-number input (12 features and a constant) passes the whole input through, whatever its targets, so a trained probe scores as well on random labels as on phones. Reading out through the fixed $\boldsymbol\Gamma^{+}$ closes that loophole.

The prediction holds, row by row. Guessing gives 0.125. Labels with half the frames replaced at random have PNMI 0.26, close to real MFCC units, yet give the same phone accuracy as the true phones, 0.569 against 0.568. Their channel is consistent and well conditioned ($\sigma_P = 0.46$). Merging phones in pairs looks better by the usual measures (PNMI 0.67, cluster purity 1.0) but drops the accuracy to 0.35, because the merge destroys rank and the two phones of a pair can no longer be told apart. Random labels have full rank only through counting noise ($\sigma_P = 0.012$), and the readout falls to chance (0.098).

Our k-means units land in between, at 0.398. They are consistent and full rank, but $\sigma_P = 0.015$, so an error in the predicted unit probabilities can grow almost 70 times on its way back to phones. This is the poorly conditioned case of the theorem. A finite linear predictor fitted on finite data cannot predict these units precisely enough, and the accuracy is fragile too, anywhere from 0.27 to 0.40 when the utterances are split differently. Correctness was never the point. Consistency, rank, and conditioning were. Better conditioned units are what iterative refinement produces, as the next part of this section shows.

In practice this tells you what to check when you build targets. Use more units than the distinctions you care about, fix the clustering once for the whole corpus, and watch for units that track the speaker. An interviewer who asks "why do k-means targets work?" wants this argument.

import numpy as np

def channel_matrix(phones, units, P, K):
    raise NotImplementedError

def recover_phone_posteriors(p_u, Gamma, tol=1e-6):
    raise NotImplementedError

Iterative refinement

Iterative refinement means making new units from the trained model itself. Better units train a better model, and a better model's hidden states (the vectors its layers produce) should cluster into better units. So cluster one of its layers, and train again on those clusters. HuBERT runs that loop, and the schedule is worth knowing exactly.

The layer choices come from measurement. The paper clustered every layer of both BASE models with 100, 500, and 1,000 units and scored the results (its Figure 2). For the iteration-1 model, unit quality peaks in the middle layers, around layer 6, and drops sharply in the last few. The authors suggest that those top layers learned to imitate the poor MFCC labels they were trained on. For the iteration-2 model, quality mostly rises with depth, which is why layer 9 was chosen for the next generation. At the same $K$, both models' features beat MFCCs on all three measures.

Why should hidden states cluster better than the input? Because of what the masked loss rewards. To predict a masked frame's unit, a layer must carry what the neighbors say about that frame's phone, and that information is shared across nearby frames. Frame-local nuisance does not help predict a neighbor's unit, so nothing rewards keeping it.

The next cell runs the loop in miniature on our synthetic frames, whose first-generation units have PNMI 0.136. The "model" is again linear. It is a projection $\mathbf{B}$, fitted by least squares to predict each frame's current unit from the mean of its four neighbors, the masked view. When features are extracted, nothing is masked. So the layer-like feature of frame $t$ is the same projection applied to the mean of the five frames around and including $t$. A linear map is a crude stand-in for a Transformer layer, but it keeps the two things that matter. It pools over neighbors, and it stretches the directions that predict the units while shrinking the rest, which changes what k-means sees as near. The cell also prints $\sigma_P$ of each generation's channel, the number that sets in the theorem how much the phone readout amplifies errors. For comparison, it runs the loop with a predictor that reads the frame itself, starts it once from random labels, and clusters the five-frame mean with no predictor at all.

With the masked objective, PNMI climbs from 0.136 to 0.60 and then 0.68 in two generations, and $\sigma_P$ from 0.024 to 0.41, so the units become usable in the sense of the theorem. Predicting each frame's unit from the frame itself, the $\alpha = 0$ extreme, improves far more slowly (0.30, then 0.55), because the cheapest predictor of a nuisance-driven label keeps the nuisance. The gain is not just averaging either. Clustering the five-frame mean directly reaches only 0.51, and the same loop started from random labels reaches 0.53, which is what smoothing alone gives. The k-means targets, poor as they are, add the rest. The miniature shows Table V's pattern, a masked objective rescuing poor targets. It also shows diminishing returns in PNMI, the second generation gaining less than the first. Each real generation costs a full pretraining run, so HuBERT stopped at three.

One comparison in the paper isolates the input. DiscreteBERT (Baevski, Auli, and Mohamed 2019) optimizes almost the same objective, masked prediction of discovered units. But its model reads discrete units, made by an earlier quantizer called vq-wav2vec, instead of the waveform. With 10 minutes of labels and a 4-gram language model (one that scores each word by the three before it), it reaches 25.2 percent test-other WER, against 15.3 for HuBERT BASE. Turning the input into discrete units throws away information the model needs. §35.4 found the same when wav2vec 2.0 was fed quantized inputs. The rule, and a common interview answer, is discrete targets and continuous inputs.

HuBERT at real scale

The cost of HuBERT is easiest to see in numbers. The next cell works out three budgets from the paper's settings. The first is training. BASE uses batches of at most 87.5 s of audio per GPU on 32 GPUs, and 100,000 steps take about 9.5 hours. The second is the memory of the clustering step. The third is the bitrate of the units, which matters once units become tokens.

The descendants, WavLM, data2vec, BEST-RQ, and w2v-BERT

HuBERT's recipe, masked prediction of discrete targets from continuous input, became the template for most self-supervised speech models. Each successor changes one thing, the input, the targets, or how the targets are made.

WavLM, denoising and a better Transformer

WavLM (Chen et al. 2022) keeps HuBERT's loss and its targets, k-means units of HuBERT BASE features (layer 9 of the released second-iteration model, for WavLM Base+ and Large). What it changes is the input. In 20 percent of training utterances, part of the audio is overlaid with a second sound. Usually that sound is another utterance from the same batch. For WavLM Base+ and Large, one overlay in ten is a noise recording instead. Either way, the model must still predict the units of the original, clean primary utterance.

The overlay length is drawn uniformly from one sample up to half the utterance. Its gain sets the energy ratio of primary to secondary to a random $r$ dB, drawn from $[-5, 5]$ for speech and $[-5, 20]$ for noise, with the scaling formula of §35.1. Because the overlay never covers more than half, the primary speaker is always the one who talks longer, so "which speaker do I transcribe" has a clear answer.

This is masked prediction plus denoising. To predict the primary speaker's units through an overlapping voice, the model must separate the two speakers and track which one is primary. That is what speaker verification, diarization (deciding who spoke when), and speech separation need, and what HuBERT's audiobooks, one reader at a time, never asked for. WavLM also scales the data from 60,000 to 94,000 hours.

Next to HuBERT's recipe these changes are small, so WavLM's results show what they buy. The gains are largest on the tasks that must know who is speaking.

data2vec, regressing a teacher's layers

data2vec (Baevski et al. 2022) drops discrete targets altogether. A student network sees the masked input. A teacher with the same architecture sees the unmasked input. The teacher's weights are an exponential moving average (EMA) of the student's, updated after each step as $\theta_{\text{teacher}} \leftarrow m\,\theta_{\text{teacher}} + (1-m)\,\theta_{\text{student}}$ with $m$ close to 1. So the teacher is a slowly moving copy of the student, the same device that self-distillation in vision uses (§28.4).

The target at a masked frame $t$ is the average of the teacher's top 8 Transformer blocks at that frame, each normalized first, $\mathbf{y}_t = \frac18\sum_{l=L-7}^{L}\hat{\mathbf{a}}^{(l)}_t$, where $\hat{\mathbf{a}}^{(l)}_t$ is block $l$'s normalized output and $L$ the number of blocks. The student predicts $\mathbf{y}_t$ with a squared-error (L2) loss. Because the teacher improves as the student does, the targets keep getting better. This is iterative refinement done continuously instead of in three discrete generations.

Normalizing the targets, together with the slow EMA, is what keeps the student from collapse, the failure where it outputs a constant that trivially predicts a constant teacher (§28.3). Predicting in feature space rather than predicting the input is also the idea behind joint-embedding predictive architectures, which predict the embeddings of masked regions of an image.

BEST-RQ, a quantizer that never learns

BEST-RQ (Chiu et al. 2022) takes the channel theorem to its limit with a labeler that is random and frozen. If consistency matters more than correctness, the labeler need not be learned at all. BEST-RQ projects each stack of frames with a fixed random matrix and labels it with the nearest of a fixed set of random unit vectors.

The two forms agree because, for unit vectors, $\lVert\mathbf{a} - \mathbf{b}\rVert^2 = 2 - 2\cos(\mathbf{a}, \mathbf{b})$. Xavier initialization draws entries with variance $2/(\text{fan-in} + \text{fan-out})$, a standard choice of scale. The scale does not change any label, since only directions are compared.

The model is a Conformer, a Transformer whose blocks add a convolution module beside the attention so each layer mixes local and global context (Gulati et al. 2020, built in full in §36.1). It reads the log-mels after a convolution that shortens the sequence 4 times, and predicts the labels at masked positions, with masked frames replaced by Gaussian noise of standard deviation 0.1. A random projection followed by a nearest-codeword lookup is consistent, since the same input always gets the same label. A random projection also roughly preserves the distances between inputs, so similar sounds tend to get the same label. (The Johnson-Lindenstrauss lemma makes that precise.) By the channel theorem, that is most of what masked prediction needs.

Having nothing to learn in the quantizer also lets the recipe scale. Google's Universal Speech Model, USM (Zhang et al. 2023), pretrains a 2B-parameter Conformer encoder with BEST-RQ on 12 million hours of unlabeled YouTube audio in over 300 languages. It makes one change, several independent random codebooks predicted by separate softmax heads, which the authors found steadier to train.

import numpy as np

def best_rq_labels(features, R, C, stack):
    raise NotImplementedError

w2v-BERT, both losses at once

w2v-BERT (Chung et al. 2021) removes HuBERT's offline stages by learning the units inside the model. A convolutional subsampler reads log-mels. A first stack of Conformer blocks, the contrastive module, is trained with wav2vec 2.0's contrastive loss, picking the true quantized frame out of decoys (§35.4). Its quantizer also assigns every frame a code. A second stack, the masked prediction module, reads the first stack's output and predicts those codes at masked positions with cross-entropy, as HuBERT does. The two losses are added with equal weights and everything trains end to end, so the targets improve as the quantizer does. Its 0.6B-parameter XL model, fine-tuned on all 960 hours of LibriSpeech, reached 1.5 and 2.9 WER on test-clean and test-other without a language model.

The table puts the family side by side.

In an interview, the one-line summary is that the field moved from learned contrastive targets to fixed classification targets, and then found the targets barely need to be learned at all.

What each layer knows

A pretrained speech encoder gives you one sequence of vectors per layer. Counting the input to the first Transformer layer, a 12-layer BASE model gives 13 and a 24-layer LARGE model gives 25. Which one to use depends on the task. The evidence comes from probes, small classifiers trained on frozen features whose accuracy measures what a layer makes easy to read out (§23.2 covers the method and its pitfalls), and from correlation analyses. The findings are empirical and correlational, measured mostly on English audiobooks, so treat them as strong guides, not laws.

The practical answer is SUPERB's protocol (Yang et al. 2021). It evaluates a frozen encoder on ten tasks, each with a small task-specific head that reads a learned mixture of all layers.

In plain words, let the task pick its layers. After training, $\operatorname{softmax}(\mathbf{w})$ is a cheap readout of which layers a task uses, and that is how the WavLM finding above was measured. Two cautions apply. Layers with larger activations get more say for the same weight unless the states are normalized, which is why WavLM rescales the weights by the layers' magnitudes before reading them. And a weight is not a causal claim. A layer can get little weight because a neighboring layer carries the same information.

import numpy as np

def probe_accuracy(F, labels, train, test, lam=1e-2):
    raise NotImplementedError

def layer_report(H, phones, speakers, train, test):
    raise NotImplementedError

Units as tokens, and where these encoders stand

The units have a second use, as tokens for language models over speech. Cluster a good layer, merge runs of repeated units, and a recording becomes a sequence of integers at a few hundred bits per second. That is a "pseudo-text" a language model can learn from. Generative spoken language modeling (Lakhotia et al. 2021) did this with 50, 100, or 200 k-means clusters of three encoders' features, HuBERT, wav2vec 2.0, and CPC, plus log-mels as a baseline (contrastive predictive coding, an earlier model that learns by picking the true future frame out of decoys). It trained a language model on unit sequences from audio alone, and turned the generated units back into speech.

AudioLM (Borsos et al. 2022) used the same idea with w2v-BERT. It takes k-means with 1,024 clusters on layer 7 of the masked prediction module, at 25 frames per second (250 bits per second). It pairs these semantic tokens, which capture what is said, with the acoustic tokens of a neural audio codec, a network that compresses audio into discrete codes and decodes them back. The acoustic tokens keep the speaker's voice. Semantic and acoustic tokens, and the language models over them, are the subject of §37.3 and §37.4.

In 2026, self-supervision remains the default where labels are scarce or the target is not text. When transcripts exist at enormous scale, training on them, even noisy ones collected from the web as Whisper does (§36.4), gives an encoder already aligned with text. When they do not, masked prediction is how you learn from audio alone. So the encoders inside large audio language models are mostly trained with supervision, while self-supervised encoders remain the usual choice for many-language recognition, speaker tasks, and speech tokenizers.

§35.6 picks up how these encoders feed classifiers, speaker systems, and language models.

In PyTorch

The block below is HuBERT's training step at toy scale. It has a convolutional stem with a total stride of 20 samples, a learned mask vector, a convolutional position embedding as in wav2vec 2.0 (without it, attention could not tell a masked frame's neighbors from distant frames), a two-layer Transformer, cosine logits against learned codeword embeddings at $\tau = 0.1$, and cross-entropy on masked frames only. The audio is synthetic, a run of tones at eight pitches, and each frame's unit is its tone. So context really does predict a masked frame's unit, and the loss can fall below $\ln K$, the loss of a uniform guess.

The loss starts above $\ln K$, because random codewords at $\tau = 0.1$ already give confident and mostly wrong logits. On the training clips it falls well below $\ln K$ within 60 steps. The last line checks that this is not memorization. On clips the model never saw, it names the tone of a hidden frame about 31 percent of the time, against 12.5 percent for guessing, and it can only do that from the visible frames around it.

At real scale you would not train HuBERT to get units. You would download it. The recipe below extracts layer 9 of the released BASE model with Hugging Face transformers and fits HuBERT's k-means settings with scikit-learn. It needs the model download, so it is shown and not run. In this API, hidden_states[0] is the input to the first Transformer layer and hidden_states[l] the output of layer l, matching the layer numbers used in this section.

Read more

• Hsu et al. (2021). HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. The model, the masked-only ablation (Table V), the unit-quality metrics, and the layer-by-layer clustering analysis.
• Chen et al. (2022). WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. Denoising and utterance mixing (Algorithm 1), gated relative position bias, and the SUPERB layer-weight analysis.
• Chiu et al. (2022). Self-supervised Learning with Random-projection Quantizer for Speech Recognition. BEST-RQ, with Table 5 separating quantizer quality from pretraining quality.
• Baevski et al. (2022). data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language. Regressing an EMA teacher's averaged layers, one recipe for three modalities.
• Chung et al. (2021). W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training. Learning the units and predicting them in one model.
• Zhang et al. (2023). Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages. BEST-RQ at 2B parameters and 12 million hours, with the BEST-RQ against w2v-BERT scaling comparison (Table 6).
• Huang et al. (2024). NEST: Self-supervised Fast Conformer as All-purpose Seasoning to Speech Processing Tasks. BEST-RQ targets on an 8x-subsampled FastConformer, with WavLM-style augmentation.
• Seamless Communication et al. (2023). Seamless: Multilingual Expressive and Streaming Speech Translation. Its description of w2v-BERT 2.0 for SeamlessM4T v2 covers the scale-up to 4.5 million hours and why the random-projection loss was dropped.
• Pasad, Chou, and Livescu (2021). Layer-wise Analysis of a Self-supervised Speech Representation Model. The autoencoder pattern and the acoustic-to-word hierarchy, with the follow-up across eleven models, Comparative Layer-wise Analysis of Self-supervised Speech Models.
• Yang et al. (2021). SUPERB: Speech processing Universal PERformance Benchmark. The frozen-encoder benchmark and its weighted layer sum.
• Arthur and Vassilvitskii (2007). k-means++: The Advantages of Careful Seeding. The $D^2$ seeding and its $8(\ln K + 2)$ guarantee.

Watch next

▶

▶

▶

▶