This section turns audio encoders into products. An encoder from the earlier sections of this chapter reads a clip and returns one vector for every short slice of time, called a frame. A product needs an answer instead. Which sounds are in the clip? Is this the phone's owner speaking? Did someone just say the wake word? Which recording in the library matches the words a user typed? Getting from frames to answers takes three decisions. You choose how to pool many frame vectors into one, which loss to train with, and which metric to judge by. Those three decisions are what this section teaches.

Here is what the earlier sections give us. A log-mel front end (§35.3) gives one frame every 10 ms, 100 a second, each a vector of 80 or 128 band energies. A learned encoder such as wav2vec 2.0 (§35.4) or HuBERT (§35.5) reads the raw waveform and gives one frame every 20 ms. That is 50 a second in long audio and 49 for a one-second clip, because the first frame needs 25 ms of audio before it exists. The encoder of Whisper, the speech recognizer of §36.4, also gives 50 a second. §35.5 also showed that the layers of such an encoder know different things. In base-sized models, who is speaking shows most clearly in the lower layers, and which sound is being said in the middle and upper ones.

We follow four tasks. Audio tagging names the sounds in a clip, such as "dog bark" and "traffic" on a YouTube video. Speaker verification decides whether two recordings come from the same voice, which is how a phone unlocks for its owner. Keyword spotting listens all day for one phrase, the wake word. Audio-text retrieval finds "rain on a tin roof" in a sound library from the typed words. At the end we look inside today's audio language models, where the same frames become tokens for an LLM and the frame rate sets the cost.

Each task has a number worth carrying into an interview, and most of them surprise people. The best single audio taggers score a mean average precision of only about 0.50 on AudioSet. A speaker verifier with an error rate under 1 percent still rejects its owner about one try in seven when impostors must almost never get in. And an hour of speech costs an audio LLM between 22,500 and 180,000 tokens, depending on one design choice. By the end you will be able to derive each of these.

Throughout, a clip's frames are $\mathbf{h}_1, \dots, \mathbf{h}_T \in \R^{C}$, $T$ vectors of $C$ numbers each ($C$ is called the number of channels in convolutional models). $K$ is the number of classes and $N$ the number of clips.

From a sequence of frames to one vector

A classifier or a verifier needs one vector per clip, so the first decision is how to pool the frames. Start with a tiny example. One channel of a four-frame clip reads $0, 0, 4, 0$, a short loud event in the third frame, like a single bark. Its mean is 1, its maximum is 4, and its standard deviation is $\sqrt{3} \approx 1.73$. The mean dilutes the bark by the length of the clip. The maximum keeps it at full strength. The standard deviation records that the channel moved a lot, which a steady value of 1 would not.

That example covers the four common choices. Mean pooling averages the frames, which suits sounds that fill the clip. Max pooling takes each channel's largest value over time, which suits events that are short and loud. Attention pooling learns a weight for every frame, so that silence and noise can be ignored. Statistics pooling keeps the standard deviation as well as the mean. How a feature varies over an utterance, such as how much the pitch moves, says a lot about who is speaking.

In plain words, each channel gets its own softmax over time. The channel that tracks vowels can listen to the vowels, and the channel that tracks hissing sounds can listen to those. When all the scores are equal, every $\alpha_{t,c} = 1/T$ and the weighted statistics become the plain ones. This is the attention of §4.1, a softmax-weighted average of stored items, with the frames as the items and one learned query per channel.

Padding is where pooling goes wrong in practice. Clips in a batch have different lengths, so the short ones are padded with extra frames to fill the batch. A mean that runs over the padding is wrong by an amount that depends on which other clips share the batch. That is the worst kind of bug, because it changes between training and serving. The cell below pools a padded batch correctly and shows what the naive mean does when the padding holds garbage.

Which frames to pool is the other half of the decision. A self-supervised speech encoder has many layers, and §35.5 showed that speaker information sits low and phonetic content higher up. So speaker models built on WavLM or HuBERT do not read the last layer. They read a learned mix of all of them, $\sum_{l} w_l\,\mathbf{h}^{(l)}_t$, where $\mathbf{h}^{(l)}_t$ is layer $l$'s vector for frame $t$ and the weights $w_l$ are a softmax of one learned number per layer. This weighted layer sum is the recipe of the SUPERB benchmark (Yang et al. 2021), which freezes a speech encoder and tests it on ten tasks with small heads. A supervised tagger such as the Audio Spectrogram Transformer below is different, since its last layer was trained for the task and is the one to read.

In an interview, the pooling question usually arrives as a bug report. "Our speaker model's scores change when we change the batch size." The answer is padding inside the pooling statistics.

Sound classification, one label or many

The datasets that define the field

Three datasets cover most of what papers report, and they pose different kinds of problem.

ESC-50 and Speech Commands give each clip exactly one label. They are ordinary $K$-way classification, with a softmax over the classes, cross-entropy, and accuracy. AudioSet gives a clip as many labels as it has sounds, which is called multi-label. A street clip can contain speech, a car, a dog, and music at once. The evaluation label file lists 51,804 positive labels on its 20,371 clips, about 2.5 per clip, and the whole dataset averages about 2. Multiple labels change the loss, the metric, and how you handle imbalance.

Sigmoids, not a softmax

A softmax makes the class probabilities sum to 1, so raising one class must lower the others. Here is what that does to a clip with both speech and music in it. Give the model logits $2, 2, -3$ for speech, music, and dog. The softmax returns $0.498, 0.498, 0.003$, which reads as "speech or music, a coin flip". The right answer is "speech, yes, and music, yes". An independent sigmoid on each logit, $\sigma(z) = 1/(1 + e^{-z})$, returns $0.88, 0.88, 0.05$, which says exactly that. So multi-label models put a sigmoid on each class and train each class as its own yes-or-no problem.

Accuracy fails for the same reason. Most of AudioSet's 527 classes are rare, so a model that says "no" to every class on every clip is right on almost all (clip, class) pairs. The metric has to look at how well each class's positives are ranked instead. It is built from two quantities from §0.11. Precision is the fraction of the clips you called positive that really are. Recall is the fraction of the truly positive clips that you found.

In plain words, each class is its own search problem ("rank the clips that contain a dog first"), AP scores that ranking, and mAP weighs every class equally. The rare "harmonica" counts as much as "speech", and that equal weighting is the point. When no two clips share a score, AP is the precision at each positive's rank, averaged over the positives.

Ties need one rule. Clips with equal scores are grouped into one threshold, as in the definition and in scikit-learn's average_precision_score. Then the result does not depend on the order in which tied clips happen to be listed.

The cell below builds a small AudioSet-like problem. Class frequencies follow a power law, so a few classes are common and most are rare, and the model is better on common classes than on rare ones, as real models are. It shows three things. First, accuracy is useless here, because predicting nothing at all is right on 97 percent of (clip, class) pairs. Second, micro-averaged AP, which pools all (clip, class) pairs into one ranking, is dominated by the common classes and flatters the model. Third, missing labels lower AP even when the model is right.

Macro mAP is 0.479 and micro AP is 0.700 for the same scores, so always check which one a paper reports (AudioSet papers report macro). The last line is AudioSet's label-noise problem in miniature. Annotators missed 40 percent of the true positives of the ten rarest classes. Those clips still score high, they now count as false positives, and the measured AP of those classes falls from 0.145 to 0.097 with no change to the model. AudioSet's own quality assessment found poor label accuracy in a substantial number of classes and started rerating them. So part of the gap between 0.5 mAP and perfection is the labels, not the models.

Two training habits follow from these facts. Balanced sampling draws rare classes more often than their natural rate, since otherwise the rare sigmoids see almost no positives. Mixup (§27.4) blends two spectrograms and their label vectors with the same weight, a regularizer that suits multi-label targets well. The Audio Spectrogram Transformer below trains on the full set with balanced sampling, mixup at ratio 0.5, and SpecAugment masks (§35.3) of up to 192 frames and 48 mel bins (Gong, Chung and Glass 2021).

What an interviewer asks here is why AudioSet's best number looks like a failing grade. The answer has four parts. There are 527 classes, most of them rare. The metric is a per-class ranking score averaged with equal weight, not accuracy. Many true sounds were never labeled. And everyone trains on a different surviving subset of the videos.

import numpy as np

def mean_average_precision(scores, labels):
    """scores, labels: (N, K). Returns (mAP, ap_per_class) with NaN for classes without positives."""
    raise NotImplementedError

The Audio Spectrogram Transformer

A spectrogram is an image with one channel

A log-mel spectrogram is a grid of numbers, frequency bands by frames, so it can be treated as a grayscale image. The convolutional taggers that came first did exactly that, and so does the model that replaced them. The Audio Spectrogram Transformer, AST (Gong, Chung and Glass 2021), runs a Vision Transformer on the spectrogram.

Recall how a Vision Transformer reads an image (§27.2). It cuts the image into square patches, flattens each patch, and maps it to a vector with one shared linear map. That map is the same as a convolution whose kernel and stride are the patch size (§27.1). It adds a learned position vector to each patch vector. It puts one extra learned vector, the CLS token, in front, and after the Transformer the CLS token's output summarizes the whole input.

AST applies this to a $128\times 1024$ log-mel spectrogram, 128 mel bands computed with a 25 ms Hamming window every 10 ms, so 1,024 frames are 10.24 s. It uses $16\times16$ patches, each projected to 768 numbers, and a 12-layer, 12-head Transformer encoder. A linear layer with a sigmoid per class reads the CLS output. The one audio-specific choice is that neighboring patches overlap by 6 in both time and frequency, so a new patch starts every 10 bands or frames.

This is the output-size formula of a convolution with no padding from §0.8, applied on both axes. It has one consequence that is easy to miss. When $n - P$ is not a multiple of $S$, the last few bands or frames are in no patch at all, and the model never sees them.

Borrowing ImageNet weights

A Vision Transformer trained from scratch on AudioSet does much worse, because ViTs need a lot of data. AST starts instead from DeiT, a ViT trained on ImageNet's 1.3 million labeled photos with help from a CNN teacher (§27.4). Three things do not fit, and each fix is a small piece of reasoning an interviewer can ask you to reproduce.

The input has one channel, not three. DeiT's patch embedding has a weight tensor $\mathbf{W}\in\R^{768\times3\times16\times16}$, one $16\times16$ kernel per output number and input color channel. Suppose you copied the gray spectrogram $\mathbf{x}$ into all three channels. The patch map is linear, so the result would be $\sum_{c=1}^{3}\mathbf{W}_{:,c}\star\mathbf{x} = \big(\sum_c\mathbf{W}_{:,c}\big)\star\mathbf{x}$, where $\star$ is the strided patch convolution and $\mathbf{W}_{:,c}$ holds the kernels for input channel $c$. So one kernel equal to the sum of the three is exactly equivalent to the copied input, at a third of the cost. The AST paper says it averages the three kernels, which is the same thing scaled by $1/3$, while its released code sums them. The input normalization (AST scales spectrograms to mean 0 and standard deviation 0.5) and fine-tuning absorb the scale, so either choice works.

The position grid has the wrong shape. DeiT at $384\times384$ pixels has a $24\times24$ grid of position embeddings, and AST needs $12\times101$. AST cuts the frequency axis down to 12 rows and bilinearly interpolates the time axis up to 101 columns, so that neighboring patches still get similar positions. Starting the position embeddings from random values instead lowers mAP on the balanced subset from 0.347 to 0.305 (the paper's Table 4).

DeiT has two special tokens. It was trained with a distillation token as well as a CLS token, and AST averages the two into one summary. The released code keeps both tokens in the sequence and averages their two outputs, so it actually reads 1,214 tokens, one more than the 1,213 counted above.

What the overlap costs

Here audio and images part ways, because the overlap multiplies the number of tokens. Recall the cost of one Transformer layer from §5.6. On $n$ tokens of width $d$ it is $24nd^2 + 4n^2d$ FLOPs. The first term is the projections and the MLP, and the second is the two $n\times n$ attention products (§4.7), which grow with the square of the token count. The cell evaluates the count for AST and for ViT-B.

AST reads 6.2 times as many tokens as ViT-B, so its attention products cost $(1213/197)^2 \approx 38$ times as much and its whole forward pass about 7.5 times as much (260 GFLOPs against 35). Dropping the overlap cuts the cost by 2.7 times and, in AST's own ablation, loses only 0.008 mAP (0.451 against 0.459 on the full set). Audio-MAE (Huang et al. 2022) found no gain from overlap at all (47.3 mAP either way). It lists 130.5 and 48.6 billion FLOPs for the two grids, which matches our count if each multiply-add is counted once, as many vision papers do. At 60 s, attention is 61 percent of the compute and growing with the square of the length. That is why long audio needs either chunking or a model whose cost grows linearly.

The widget draws AST's patch grid over the first 1.6 s of a synthetic spectrogram, with the first two patches in time outlined so that their overlap shows. Drag the stride and the clip length, or switch the frame hop, and watch the token count and the share of the compute spent on attention.

An interviewer will ask you to count AST's tokens on a whiteboard and then ask what overlap buys. The honest answer is very little accuracy for almost three times the compute.

import numpy as np

def ast_tokens(n_mels, n_frames, patch=16, stride=10):
    """Returns (n_f, n_t, n_patches) for square patches with the same stride on both axes."""
    raise NotImplementedError

def encoder_flops(n_tokens, d=768, L=12):
    """Returns (total forward FLOPs, FLOPs of the two n x n attention products), 2 FLOPs per multiply-add."""

Where AST sits among audio encoders

AST has neighbors an interviewer will expect you to place, and they come in three kinds. Convolutional networks are cheap and still common baselines. Hierarchical Transformers attend within local windows and merge patches as they go, as Swin does for images (§27.5), so their cost grows linearly with length. Self-supervised encoders pretrain on unlabeled AudioSet audio by hiding patches and predicting them, the audio version of the masked image models of §28.5, and they no longer need ImageNet at all. The table lists the models worth knowing, and the box below says what each one changed.

Tagging in time, sound events and music

AudioSet's labels are weak. They say a dog barks somewhere in the 10 seconds, not when. Sound event detection asks for the when, an onset and an offset for each event, and it usually has to learn this from weak labels anyway. Learning from a label on a whole bag of items (here, the frames of a clip) without labels on the items is called multiple-instance learning.

The recipe has three steps. The model predicts a probability $p_{t,k}$ for each frame $t$ and class $k$. A pooling function turns the frame probabilities into a clip probability $p_k$, which the weak label can supervise. At test time you read the frame probabilities. The pooling function decides which frames get the gradient, and a study of five of them (Wang, Li and Metze 2019) showed that this choice decides how well events are located.

• Max pooling, $p_k = \max_t p_{t,k}$, sends all the gradient to one frame. The model learns to find the single loudest bark and under-detects long events.
• Mean pooling, $p_k = \frac{1}{T}\sum_t p_{t,k}$, sends equal gradient to every frame. A positive label pushes up the silent frames too, and the frame outputs smear across the clip.
• Linear softmax pooling, $p_k = \sum_t p_{t,k}^2/\sum_t p_{t,k}$, weights each frame by its own probability. Its derivative is $\partial p_k/\partial p_{t,k} = (2p_{t,k} - p_k)/\sum_{t'} p_{t',k}$. So for a positive clip, the frames above half the clip probability are pushed up and the rest are pushed down, which sharpens the boundaries. With two frames at 0.9 and 0.1, for example, $p_k = (0.81 + 0.01)/1.0 = 0.82$, and the derivatives are $+0.98$ for the loud frame and $-0.62$ for the quiet one. It located events best of the five.
• Attention pooling, $p_k = \sum_t w_{t,k}p_{t,k}$ with learned softmax weights $w_{t,k}$, is the attentive pooling idea from the start of this section.

Detection is scored in one of two ways, as in the yearly DCASE challenges (DCASE). Event-level scoring counts an event as found if its onset falls within a tolerance window, often 200 ms, of the reference onset, and reports an F1 score over events. Segment-level scoring cuts the audio into fixed one-second segments and reports precision and recall over them.

Music tagging is multi-label tagging with a different vocabulary, such as genre, instrument, mood, and era. The standard sets are MagnaTagATune (usually its 50 most frequent tags) and MTG-Jamendo, and results are reported as the area under the ROC curve and the area under the precision-recall curve, averaged over tags. The second is AP again. Music needs things speech models throw away, pitch and harmony at fine frequency resolution and rhythm over many seconds, so music encoders add their own pretraining targets.

Speaker verification

The task and its two errors

Speaker identification picks one speaker from a fixed list, an ordinary classification. Speaker verification answers a yes-or-no question about two recordings. Someone first enrolls by saying a few sentences, which become a reference embedding. Later a test recording arrives with a claimed identity. The system computes a score $r$ comparing the two and accepts the claim when $r \ge \eta$ for a threshold $\eta$.

The speakers at test time were never seen in training, which is called open-set. So the embedding cannot just separate the training speakers. It has to put any two recordings of one voice close together and recordings of different voices far apart.

A comparison of two recordings is called a trial. A trial whose recordings come from the same speaker is a target trial, and otherwise it is a non-target (impostor) trial. There are two ways to be wrong. Accepting an impostor is a false accept. Rejecting the real speaker is a false reject, also called a miss. Moving $\eta$ trades one for the other, and that trade is what verification metrics measure. We make it precise after building the embeddings.

Speaker embeddings, from x-vectors to ECAPA-TDNN

The x-vector network has three parts. First come five frame-level layers, 1-D convolutions over time on 24-dimensional filterbank features. Speech papers call such layers time-delay layers, and a network of them a time-delay neural network (TDNN). Their contexts are $[t-2, t+2]$, then $\{t-2, t, t+2\}$, then $\{t-3, t, t+3\}$, then two layers that look only at frame $t$. In the language of §0.8 these are kernels 5, 3, and 3 with dilations 1, 2, and 3. The receptive-field recursion $r_l = r_{l-1} + (k_l - 1)d_l$ for kernel $k_l$ and dilation $d_l$ at stride 1 gives $1 + 4 = 5$, $5 + 2\cdot2 = 9$, and $9 + 2\cdot3 = 15$ frames.

Second, statistics pooling concatenates the mean and standard deviation of the last frame layer's 1,500 channels over the whole utterance, a 3,000-dimensional vector. Third, two segment-level layers of 512 units follow, then a softmax over the training speakers. The embedding is the output of the first segment layer, before its nonlinearity. The softmax layer is thrown away after training, since its only job was to force the segment layers to separate speakers.

ECAPA-TDNN (Desplanques, Thienpondt and Demuynck 2020) keeps that shape and improves every part.

• Its frame layers are 1-D convolutions with dilations that widen the context. Inside each block the channels are split into groups processed at different scales (the Res2Net design). A squeeze-and-excitation step then takes every channel's mean over the whole utterance, passes that vector through a small network, and rescales each channel by the resulting gate between 0 and 1. That way the frame features know about the whole recording.
• The outputs of all three blocks are concatenated before pooling, because shallower features also carry speaker information.
• Pooling is the channel-dependent attentive statistics pooling defined at the start of this section, and the embedding has 192 dimensions.
• The inputs are 80-dimensional features from 25 ms windows with a 10 ms shift (the paper calls them MFCCs). Training uses random 2-second crops, subtracts each crop's mean feature vector, and applies light SpecAugment masking (§35.3).

Training for open-set verification with AAM-softmax

A plain softmax over the $K$ training speakers separates them, but only as far as classification needs. A new speaker who sits between two training speakers can land near either. Verification wants each speaker's embeddings packed into a tight cone with a wide empty gap around it, so that the cosine between two embeddings is a reliable score even for unseen voices. The additive angular margin softmax (AAM-softmax) builds that gap into the loss. It was introduced for faces as ArcFace (Deng et al. 2019) and is the standard loss for speaker embeddings.

In plain words, it is cross-entropy over cosine logits multiplied by $s$, except that the correct class is scored as if its angle were $m$ larger than it is. Normalizing removes the embedding's length from the decision, so the only thing the network can improve is the angle. The scale $s$ sets how confident a given cosine gap is. Cosines lie in $[-1, 1]$, so without a scale the softmax could never become confident, the same temperature argument as in §7.2.

The example is one case of a general rule, which says exactly what the margin demands.

So training does not stop when an embedding is merely closer to its own speaker than to the others. It keeps pushing until the gap is at least $m$ radians, 0.2 rad or 11.5 degrees for ECAPA, and that slack is what keeps unseen speakers apart at test time.

The main alternative needs no class weights at all. The generalized end-to-end (GE2E) loss (Wan et al. 2018) builds each batch from $n_{\text{utt}}$ utterances of each of $n_{\text{spk}}$ speakers. Each utterance is asked to pick its own speaker's average embedding, the centroid, out of all the centroids in the batch.

In plain words, every utterance plays the test recording of a trial against every speaker in the batch, with that speaker's other utterances as the enrollment. Two details carry the method.

The utterance is left out of its own centroid. The full centroid is $\mathbf{c}_j = \frac{1}{n_{\text{utt}}}\mathbf{e}_{ji} + \frac{n_{\text{utt}} - 1}{n_{\text{utt}}}\mathbf{c}_j^{(-i)}$, so with 4 utterances the dot product with the full centroid is $\frac14 + \frac34\mathbf{e}_{ji}^\top\mathbf{c}_j^{(-i)}$. A quarter of the score would be the utterance matching itself. The authors found that leaving it out stabilizes training and helps avoid trivial solutions.

The scale $w$ is kept positive, since a negative scale would reward being far from your own speaker. The paper starts from $(w, b) = (10, -5)$. GE2E trains directly for what verification does at test time, comparing an utterance with an enrollment average.

import numpy as np

def attentive_stats_pool(H, W, b, v, k, lengths):
    """H (B, T, C), W (R, C), b (R,), v (C, R), k (C,), lengths (B,). Returns (B, 2C)."""
    raise NotImplementedError

def aam_softmax_loss(emb, W_cls, labels, s=30.0, m=0.2):
    """emb (B, D), W_cls (K, D), labels (B,) ints. Returns the mean loss."""

import numpy as np

def ge2e_loss(E, w, b):
    """E (n_spk, n_utt, D). Returns (mean loss, S of shape (n_spk, n_utt, n_spk))."""
    raise NotImplementedError

EER, DET curves, and minDCF

Now the metrics. Sweep the threshold $\eta$ over all values. At each one, the false accept rate $P_{\text{fa}}(\eta)$ is the fraction of non-target trials with $r \ge \eta$. That is the false positive rate of the ROC curve in §0.11. The false reject rate $P_{\text{miss}}(\eta)$ is the fraction of target trials with $r < \eta$, one minus the true positive rate. As $\eta$ rises, $P_{\text{fa}}$ falls and $P_{\text{miss}}$ rises. Here is the trade in eight trials, before any definitions.

The definition states both numbers for any set of trials, including how to find the EER when no threshold makes the two rates exactly equal.

In plain words, the EER is the error rate at the one threshold where both kinds of mistake are equally common. MinDCF is the expected cost of the best threshold for a stated mix of trials and costs, measured against the cheaper of the two systems that ignore the audio (accept everyone or reject everyone). With the VoxCeleb convention, $C_{\text{default}} = 0.01$ and the normalized cost is $P_{\text{miss}} + 99P_{\text{fa}}$. A 1 percent false accept rate alone already costs 0.99, almost as much as rejecting everybody. That is why minDCF lives where false accepts are rare, and why a deployed unlock system cares about it far more than about the EER.

Fact (ii) is why papers report EER and minDCF and not accuracy. Both judge the ranking of the scores, whatever their scale. The plot that shows every operating point is the DET curve, $P_{\text{miss}}$ against $P_{\text{fa}}$ with both axes in normal deviates ($\Phi^{-1}$ of the rate). Fact (iii) turns Gaussian-like systems into nearly straight lines on it, which are easy to compare.

A deployed system has a harder job, because it has to fix its threshold in advance. The cost it actually pays there, the actual DCF, also depends on calibration, whether a score means the same thing on new data (§0.11). The deeper box shows where the ideal threshold sits when the scores are calibrated.

The cell reproduces the eight-trial table and then turns an EER into a product number, which the real-scale example below uses.

The widget draws the two kinds of score as Gaussians. Different-speaker scores are centered on 0 with a spread of 0.10, and same-speaker scores on 0.45. Drag the threshold and watch the two shaded error areas trade against each other while the operating point moves along the DET line beside them.

import numpy as np

def eer_and_mindcf(scores, is_target, p_target=0.01, c_miss=1.0, c_fa=1.0):
    """Returns (eer, threshold, min_dcf). Accept a trial when its score >= threshold."""
    raise NotImplementedError

Scoring, and what the channel does to it

The simplest way to score a trial is the cosine between the enrollment and test embeddings. With embeddings trained by AAM-softmax it is the usual choice and hard to beat. The older alternative is PLDA, the back end of the i-vector and x-vector era, which helps most when the embeddings were not trained for cosine scoring.

Real trials are messier than the training data. The enrollment may come from a phone and the test from a laptop. A microphone and its audio compression shift every embedding recorded through them by a roughly constant offset. This is the embedding-space version of the channel offset that cepstral mean normalization removes from features (§35.3). Two remedies are standard, and both use a cohort, a set of speakers who appear in no trial.

Condition mean normalization estimates each recording condition's mean embedding from the cohort, subtracts it, and normalizes to unit length again. Score normalization rescales each trial's score by how its two sides score against the cohort. The symmetric form, s-norm, is

where $\mu_{\text{e}}$ and $\sigma_{\text{e}}$ are the mean and standard deviation of the enrollment embedding's cosines with the cohort, and $\mu_{\text{t}}$ and $\sigma_{\text{t}}$ the same for the test embedding. In plain words, each side asks how unusual this score is compared with the scores it gets against strangers, and the two answers are averaged. Adaptive s-norm uses only the $k$ most similar cohort members on each side. ECAPA-TDNN scores with cosine plus adaptive s-norm. Its cohort holds one averaged embedding per training speaker, and the paper sets the cohort size to 1,000.

The cell builds a hard case on purpose. Every target trial crosses devices, and every non-target trial shares one. The device offset makes impostors on the same phone look alike and real speakers on different devices look different.

Raw cosine scores give an EER of 31 percent, since the shared device lifts the impostor scores almost to the level of the real speakers. Subtracting each device's cohort mean drops it to 3.5 percent. The cohort never contains a trial speaker. That matters, because a mean computed from the trial speakers themselves would quietly use test information. Exercise 35.6f shows the other half of the lesson, that score normalization alone cannot do this. An interviewer who asks "your verifier works in the lab and fails in the field, what do you check?" wants to hear about this offset first.

import numpy as np

def device_mean_normalize(E, dev, cohort_idx):
    """Subtract each device's mean cohort embedding, then L2-normalize. Returns (n, d)."""
    raise NotImplementedError

def adaptive_snorm(r, S_enroll, S_test, top_k=None):
    """r (M,) raw scores, S_enroll and S_test (M, n_cohort) cohort scores. Returns (M,) normalized scores."""

What the numbers look like at real scale

Error rates this low look like a solved problem. The example shows what one of them means for a product that must almost never let an impostor in.

Two things threaten a deployed system beyond its EER. Domain shift, meaning new languages, devices, audio compression, or distant microphones, degrades embeddings in ways the training data did not cover, as the channel cell showed. And spoofing, replaying a recording or synthesizing the voice, attacks the system directly. The ASVspoof challenges (ASVspoof) evaluate countermeasures, and the voice cloning that makes spoofing easy is the subject of the safety discussion at the end of speech generation. The same embeddings also power diarization, labeling who spoke when in a long recording, which §36.6 builds.

Keyword spotting and wake words

Before a phone can check who is speaking, it has to notice that anyone said "Hey X" at all. Keyword spotting detects one or a few fixed phrases in an audio stream that never ends. A wake word is a keyword whose detection switches on the rest of the system.

It looks like the easiest task in this section. Speech Commands (the fact box above) is ordinary classification of one-second clips, and good small models get about 95 percent on its standard 12-class version (ten command words, silence, and "unknown"). The difficulty is all in deployment. The detector listens all day, making a decision every 10 to 100 milliseconds on audio that almost never contains the keyword. It runs on a processor with a tiny power budget. And its errors are counted per hour, not per clip.

A streaming detector and its operating metric

A deployed detector slides a window of about one second of features along the stream and outputs a keyword probability, its posterior, at every step. The raw posteriors flicker, because a single frame of a near-miss word or a door slam can look like the keyword. So they are averaged over a short window before the threshold is applied.

An early and influential detector showed that a small feed-forward network beats the older hidden Markov model detectors at this task (Chen, Parada and Heigold 2014). It made a decision every 10 ms and smoothed over 30 decisions. For a phrase of several words, its confidence was the geometric mean, over the words, of each word's largest smoothed posterior in the last 100 decisions.

A classification benchmark reports an error rate per window, and a product needs a count per hour. The bridge between them is short.

In plain words, a detector deciding every 10 ms makes 360,000 decisions an hour, so a per-window false positive rate that sounds negligible is not. If false positives come in runs of $r$ decisions on average, then $\mathrm{FA/h} = 3600q/(r\,t_{\text{hop}})$. Smoothing turns isolated spikes into rarer but longer runs, which makes the bound loose.

The cell simulates a detector with a 10 ms hop on two hours of keyword-free audio, with 30 near-miss words an hour, plus 400 two-second clips that contain the keyword. The posteriors are correlated noise around a low logit, with a bump wherever a near miss or the keyword occurs. It counts triggers at every threshold at once, since a run starts at decision $t$ exactly when $\bar p_{t-1} < \eta \le \bar p_t$, and it picks the lowest threshold that gives at most 0.5 false accepts an hour.

At the same 0.5 false accepts an hour, smoothing over 30 decisions (0.3 s) cuts the false reject rate from about 23 percent to about 5.5 percent. The raw detector's short noise spikes force its threshold up to 0.988, which also rejects every keyword whose posterior is not very confident. Smoothing averages the spikes away, since a real keyword lasts 0.6 s and a spike a few frames, so the threshold can come down to 0.62.

The third column checks the theorem. For raw posteriors the bound is met, because the false positives above that threshold are isolated decisions. For smoothed ones it overstates the true rate 15 times, because each false trigger is a run of about 15 decisions.

Small footprints and cascades

The always-listening part has to fit a microcontroller or a small signal processor with a tiny power budget, and its size is easy to count. Chen, Parada and Heigold's network reads 30 past frames, the current one, and 10 future frames of 40 log filterbank energies. That is 41 frames of 10 ms, so it waits 100 ms for the future frames before deciding. It has 3 hidden layers of 128 ReLU units and one output per word of the phrase plus one for everything else.

That two-stage design, a cascade, is the general answer to the budget. A cheap first stage runs at a high false accept rate, perhaps a few per hour. A much larger stage runs only after a trigger and brings the rate down. Apple's second pass has 36 times the hidden-to-hidden weights of its first, since $192^2/32^2 = 36$. A check on the server can follow if needed.

The cascade's false accept rate is the first stage's rate times the probability that the later stages also accept one of its false triggers. Its false reject rate is $1 - (1 - \mathrm{FRR}_1)(1 - \mathrm{FRR}_2)$, about the sum of the two. A personalized wake word adds one more stage, a speaker verifier from earlier in this section that accepts the phrase only from the enrolled voice. A voice activity detector, which §36.6 builds, can sit in front of the whole cascade so that silence costs nothing.

Audio and text in one space with CLAP

A tagger can only name the 527 classes it was trained on. To search a sound library for "rain on a tin roof", you need embeddings that put audio and text in one space. A clip and a caption that describe the same thing should end up close together. CLIP did this for images (§30.1), and CLAP, contrastive language-audio pretraining, is the same recipe for sound. Recall the recipe in two sentences. Two encoders map a batch of matching pairs to unit vectors. A softmax loss then asks each item to pick its own partner out of the batch, in both directions.

The loss, its gradient, and what the learned temperature does are derived in §7.2. The training and zero-shot recipes for images are in §30.2 and §30.3. What is specific to audio is the data. CLIP trained on 400 million image-text pairs, and audio has nothing close.

Retrieval is scored with recall at $k$ (§7.5), the fraction of queries whose correct item is among the top $k$ results, in both directions. Ties count against the model. A query is a hit at $k$ when fewer than $k$ other items score at least as high as the correct one. So a model that gives every item the same score has recall 0 at every $k$ below the batch size, not the credit an argmax would hand to whichever item happens to come first.

The wording of the prompt matters more than people expect. The original CLAP paper reports ESC-50 zero-shot accuracy from 78.6 percent with "i can hear [class label]" to 82.6 percent with "this is a sound of [class label]", and 81.2 percent for the bare label. Averaging several prompts, as in the definition box, removes some of that variation. The cell simulates it with synthetic embeddings in which each prompt template adds its own shared offset and each (class, prompt) pair its own noise.

One prompt gives 73 percent on average and an ensemble of 16 gives 91.5 percent, with most of the gain in the first four prompts. The gain stops because what remains is noise in the audio embeddings, which no prompt can average away. Real prompt noise is smaller than in this simulation, so real gains are smaller (a few points for CLIP), but the shape is the same.

import numpy as np

def clap_loss_and_retrieval(A, T, log_tau, ks=(1, 5, 10)):
    """A, T (B, d) paired rows. Returns (loss, recall_t2a (len(ks),), recall_a2t (len(ks),))."""
    raise NotImplementedError

def zero_shot_classify(audio_emb, text_emb_per_prompt):
    """audio_emb (N, d), text_emb_per_prompt (K, n_p, d). Returns (predictions (N,), cosines (N, K))."""

Zero-shot numbers need the same scrutiny as any benchmark (§22.3), for three reasons.

• ESC-50's clips come from Freesound, and so do FSD50K and Clotho, which audio-text training sets include. The LAION-CLAP authors found overlaps between their training data and the zero-shot test sets, removed the overlapping clips before reporting, and published the counts.
• The audio encoder is usually pretrained on AudioSet labels, whose classes include most of ESC-50's. So "zero-shot" means "no ESC-50 labels", not "never trained on these sounds".
• Contrastive audio-text models, like their image counterparts (§30.6), behave much like bags of words, so "a dog barks, then a car passes" and "a car passes, then a dog barks" get nearly the same embedding. On test pairs built to differ only in event order or in which sound has which attribute, CLAP-style models score barely above chance (Ghosh et al. 2023).

In PyTorch and in practice

The block below builds a tiny AST, with the patch embedding as a strided convolution, a CLS token, learned positions, and a sigmoid head trained by binary cross-entropy for one step on random spectrograms. It then builds an AAM-softmax module and checks it against a NumPy version of the formula.

In practice nobody trains these from scratch. The pretrained AST, LAION-CLAP, and ECAPA-TDNN models are a few lines away in Hugging Face Transformers and SpeechBrain. The block is not run here, since it downloads weights.

Audio encoders inside audio language models

The last step of the chapter is the one that matters most at frontier labs today. An audio language model is an LLM that can listen. It has four parts. An audio encoder turns the waveform into frames. A downsampling step merges neighboring frames. A projector maps each merged frame into the LLM's embedding space. And the LLM reads these audio tokens interleaved with text. This is the same connector pattern that §31.1 builds for images.

The encoder is most often Whisper's, trained with supervision on transcripts rather than self-supervised. §35.5 noted this when it placed today's speech encoders, and the newest systems train their own. The number that shapes the whole system is the frame rate, because every audio token costs context and compute in the LLM. In the table, a Conformer is a Transformer block with a convolution module added, common in speech recognition.

Read the table by its last two columns. The encoders come from three places. Many are Whisper's, trained with supervision on transcripts. Some descend from self-supervised speech models, as Gemma 3n's descends from the Universal Speech Model (Zhang et al. 2023), which was pretrained with the BEST-RQ objective of §35.5 on 12 million hours. And the newest are trained from scratch by the lab that builds the LLM, on 10 to 20 million hours. The rates fall over time, from 50 tokens a second to 12.5 and then 6.25, because the rate sets the cost of everything downstream.

The frontier has also moved in how these models are judged and trained. Benchmarks moved from probing frozen encoders to asking audio LLMs questions about clips (the box below lists them), and the largest recent gains came from post-training rather than from new encoders.

A multiple-choice audio question has an answer a program can check, which is the setting of §18.3. GRPO (§18.4) samples several answers to each question and pushes the model toward the ones that score above the group's average. It applies to audio unchanged, with a rule that checks the chosen option as the reward. Applied to Qwen2-Audio-7B-Instruct with only 38,000 training questions from another dataset (AVQA, audio-visual question answering), GRPO beat supervised fine-tuning on the same data and reached 64.5 percent on MMAU test-mini (Li et al. 2025). An explicit chain of thought did not help noticeably. Step-Audio 2 ends its post-training with two PPO stages and then GRPO, and reports 78.0 on the revised MMAU test-mini.

Scale is the other lever. MiMo-Audio pretrained on more than 100 million hours of audio. It reports that the model learns new audio tasks from a few examples in the prompt, the in-context learning that GPT-3 showed for text (§6.8). This is the frontier as of this writing, and it moves fast, so for any new model check the tokens-per-second column first.

Speech recognition proper, where Whisper itself is taught, starts in §36.1, and models that listen and also speak back are in §37.5.

Read more

• Gong, Chung & Glass (2021). AST: Audio Spectrogram Transformer. Short and complete. Section 2 has the ImageNet adaptation, and Tables 2, 4, and 5 the pretraining, position, and overlap ablations quoted here. Read with Chen et al. (2022), BEATs: Audio Pre-Training with Acoustic Tokenizers, the HuBERT idea for general audio that overtook it.
• Gemmeke et al. (2017). Audio Set: An ontology and human-labeled dataset for audio events. How the dataset and its 527-class labels were built, read with the AudioSet download page for the splits and the rerating notes.
• Desplanques, Thienpondt & Demuynck (2020). ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. The speaker embedding model of record, with attentive statistics pooling, the training recipe, and the VoxCeleb evaluation protocol.
• Deng et al. (2019). ArcFace: Additive Angular Margin Loss for Deep Face Recognition, and Wan et al. (2018), Generalized End-to-End Loss for Speaker Verification. The two losses behind modern speaker embeddings, the angular margin and the leave-one-out centroid softmax.
• Snyder et al. (2018). X-Vectors: Robust DNN Embeddings for Speaker Recognition. The architecture table and the augmentation study that started neural speaker embeddings.
• Chen, Parada & Heigold (2014). Small-Footprint Keyword Spotting using Deep Neural Networks, and Zhang et al. (2017), Hello Edge: Keyword Spotting on Microcontrollers. Posterior smoothing and the per-hour operating point, then the memory and compute budgets of always-on hardware.
• Elizalde et al. (2022). CLAP: Learning Audio Concepts From Natural Language Supervision, and Wu et al. (2023), Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. The two CLAP papers, including the prompt study and the overlap table.
• Chu et al. (2024). Qwen2-Audio Technical Report, and Liu et al. (2025), Voxtral. Two clear descriptions of how an audio encoder, a downsampler, and an LLM are put together, with the frame rates used above. Read with Li et al. (2025), Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering, for GRPO on an audio LLM.

Watch next

▶

▶

▶

▶