A log-mel spectrogram is the picture of sound that almost every speech and audio model reads. It is a grid of numbers with one row for every 10 milliseconds of audio and one column for each frequency band. Each entry is the logarithm of the energy in one band during one 10 ms step. The bands are narrow at low frequencies and wide at high ones, roughly the way the ear tells pitches apart.

Whisper, OpenAI's speech recognizer, reads this grid. So do most sound classifiers, speaker-recognition networks, and the audio encoders inside chat models that listen. Train a model on one version of the grid, serve it with a slightly different one, and nothing crashes. Accuracy just drops. So a speech engineer needs to know every step exactly, why it is there, and how the common libraries differ.

The grid starts from the power spectrogram of §35.2, which works like this. You cut the waveform into short overlapping slices called frames. You multiply each frame by a smooth window that fades its ends to zero. Then you take its discrete Fourier transform (DFT), which measures how much of each frequency the frame contains. The squared size of each Fourier coefficient, $\lvert X[t,k]\rvert^2$, is the energy of frame $t$ near frequency bin $k$.

Here are the usual speech settings in numbers. The audio has $f_s = 16{,}000$ samples per second (16 kHz). A frame is $N = 400$ samples (25 ms), and a new frame starts every $R = 160$ samples (10 ms), a step called the hop. Thirty seconds of audio then becomes a matrix with $T = 3{,}000$ rows, one per frame. It has $N/2 + 1 = 201$ columns, one per bin, spaced $f_s/N = 40$ Hz apart from 0 to 8 kHz.

Almost no model reads that matrix as it is. Models first apply three steps. They pool the 201 bins into 80 or 128 bands spaced the way the ear spaces pitch, they take a logarithm, and they normalize. §1.8 sketched this front end in one paragraph. This section builds each step, says why it is there, and matches it line by line to the code real models were trained with. It then covers three things every speech engineer is expected to know. These are the classical features that came before (the cepstrum and MFCCs), the standard augmentation for these inputs (SpecAugment, which hides random bands and stretches during training), and front ends that learn their own filters.

What is wrong with the raw spectrogram

We need a test signal, so the first cell builds a short utterance from the source-filter picture of §35.1. Here is that picture in brief. In a vowel, the vocal folds open and close at a steady rate, the pitch $F_0$. That buzz is made of harmonics, pure tones at every whole multiple of $F_0$. The throat and mouth then boost the harmonics near a few resonant frequencies, called formants. The formants identify the vowel.

Our utterance has four parts. It starts with the vowel of "hod" at a pitch of 120 Hz. Then comes a short pause, then a hiss like the /s/ in "see", and last the vowel of "heed" at 140 Hz.

The next cell computes the power spectrogram with Whisper's framing and measures it.

The spectrogram has three problems. The first two are in the printed numbers, and the third is in every voiced row.

The range is enormous. The largest entry is $10^{12.5}$ times the smallest. A network that reads these numbers sees a handful of huge values and a sea of values that look like zero.

The bins are spent in the wrong place. Half the columns, 101 of 201, describe 4 to 8 kHz. Over the whole clip only 1.9 percent of the energy lands there. The /s/ frames put 99.9 percent of their energy there, but /s/ is noise. A hundred separate measurements of noise, 40 Hz apart, say little more than a few wide bands would. Meanwhile everything below 1 kHz, where the first formant and the strongest harmonics live, gets only 25 columns.

Pitch is tangled with the vowel. Every voiced row carries the comb of harmonics, one peak every $F_0$ hertz. The same vowel spoken at a different pitch gives a different row, although a listener hears the same vowel.

The rest of the section fixes these in turn. The mel scale fixes the spacing, the logarithm fixes the range, and the cepstrum separates the pitch from the vowel.

How the ear spaces frequency

The ear does not measure frequency on an even ruler. A 100 Hz step is a large change at 200 Hz and a small one at 6 kHz. The reason is the cochlea, the coiled organ of the inner ear.

A membrane runs along the cochlea. Each place on it vibrates most for one frequency, high frequencies near its base and low ones near its far end. So each place acts as a band-pass filter, a filter that passes only frequencies near its own. These filters are narrow at low frequencies and wide at high ones.

So resolution is nearly constant in hertz at low frequencies and nearly proportional to frequency at high ones. Stevens, Volkmann, and Newman (1937) measured the same effect as perceived pitch. Listeners adjusted one tone until it sounded half as high as a reference tone. The results were fitted to a scale whose unit is the mel, from melody. Equal steps in mels sound like equal steps in pitch. The scale is anchored so that a 1,000 Hz tone, played 40 dB above the quietest level a listener can hear, has a pitch of 1,000 mels. Many formulas have been fitted to such data since, and speech software settled on two of them.

In plain words, the HTK curve is nearly a straight line well below 700 Hz and nearly a logarithm well above it, and it bends smoothly in between. Slaney's curve has a sharp corner at 1 kHz. Below the corner each mel is $200/3$ Hz. Above it, every factor of 6.4 in frequency adds 27 mels.

The two units differ in size. 1 kHz is 1,000 HTK mels but only 15 Slaney mels. That never matters, because a filterbank uses only equal steps along the scale, so only the shape of the curve counts.

A few values make the shape concrete. In HTK mels, $\operatorname{mel}(1000) = 1000.0$, $\operatorname{mel}(4000) = 2146.1$, and $\operatorname{mel}(8000) = 2840.0$. So the band from 0 to 1 kHz takes 35 percent of the mel range up to 8 kHz (33 percent in Slaney units), although it is only 12.5 percent of the hertz range. The octave from 4 to 8 kHz is half the hertz range but only 24 percent of the mel range. A filterbank spaced evenly in mels therefore spends about a third of its bands below 1 kHz, where speech carries its fine structure.

The mel scale is an empirical fit to listening experiments, not a law of nature. What models rely on is only its shape, fine resolution at low frequencies and coarse resolution above. That shape is solid. An interviewer who asks "is the mel scale a law?" wants to hear exactly this distinction.

Triangles pool the spectrum into bands

A mel filterbank turns the 201 bins of a frame into a few dozen band energies. Each band is a triangle of weights over the bins. The triangles are evenly spaced on the mel scale, so they are narrow and close together at low frequencies and wide and far apart at high ones. A band's energy is the weighted sum of the bin powers under its triangle. Here is a tiny version by hand.

The general recipe has four steps.

1. Choose a frequency range $[f_{\min}, f_{\max}]$, usually 0 Hz to $f_s/2$, and a number of bands $n_{\text{mels}}$. Whisper uses 80 or 128. The Audio Spectrogram Transformer, a sound classifier, uses 128, and classical recognizers use 23 to 40.
2. Place $n_{\text{mels}} + 2$ points equally spaced on the mel axis from $\operatorname{mel}(f_{\min})$ to $\operatorname{mel}(f_{\max})$, and convert them back to hertz. Call these edges $f_0 < f_1 < \dots < f_{n_{\text{mels}}+1}$.
3. Filter $j$, counting from 0 as code does, is a triangle that rises from 0 at $f_j$ to 1 at $f_{j+1}$ and falls back to 0 at $f_{j+2}$. So each triangle starts at its left neighbor's peak and ends at its right neighbor's peak.
4. Evaluate each triangle at the bin frequencies $kf_s/N$. Each triangle becomes one row of a matrix.

In plain words, each band's energy is a weighted sum of the power in nearby bins. The weights peak at the band's center and fade to zero at its neighbors' centers. The features are the logs of those sums. For a whole spectrogram it is one matrix product, $(T, 201)\times(201, 80)\to(T, 80)$ for Whisper.

The two normalizations behave differently, and the difference is exact enough to box.

So peak-normalized filters split each bin's power among the bands without creating or losing any, and neighboring triangles cross at half height. Area normalization gives that up. In return, the wide high bands get the same total weight as the narrow low ones, so a flat (white) spectrum produces roughly equal energy in every band. Either choice works, but the two give different numbers, which matters when you switch libraries.

import numpy as np

def mel_filterbank(sr, n_fft, n_mels, fmin=0.0, fmax=None, scale="slaney", norm="slaney"):
    """Return F_mel of shape (n_mels, n_fft // 2 + 1)."""
    raise NotImplementedError

The next cell holds the filterbank used for the rest of the section, the exercise's solution in compact form. It checks the mel values quoted above, then looks for one trap, filters that come out empty.

The empty filters are a real trap. With HTK spacing, 128 bands up to 8 kHz put the lowest edges only about 14 Hz apart, while the FFT bins at $N = 400$ are 40 Hz apart. A triangle narrower than a bin can fall entirely between two bin frequencies and get no weight at all. Here filters 0, 3, 6, and 13 are all zero. Their log-mel values sit at the floor in every frame, so they are four dead features that carry nothing.

Both libraries warn about this case. librosa reports empty filters, and torchaudio reports a filterbank with all-zero values. Slaney spacing is linear below 1 kHz, with edges 23 Hz apart for 128 bands, and at $N = 400$ it leaves no filter empty. So Whisper large-v3's 128 bands, built with librosa's Slaney defaults on the same frames, have no dead features. The last line of the output checks area normalization. Each triangle has an area close to 1, off only by the error of sampling it every 40 Hz.

In the widget, frequency runs along the mel scale, so every triangle has the same width. The FFT bins, evenly spaced in hertz, thin out toward the bottom of that axis. Each triangle is drawn as the matrix row it really is, sampled at those bins, and a triangle that catches no bin is drawn dashed. Below, one frame of the "hod" vowel at 120 Hz is pooled into band energies, each the weighted average of the power under its triangle. Change the number of bands, the mel scale, and the FFT size, and watch where the filters go.

The logarithm fixes the range and turns a channel into an offset

After pooling, the mel energies $e_{tj}$ still span many orders of magnitude. The standard fix is a logarithm. There are three separate reasons for it, and an interviewer will want all three.

Range. Band energies within speech span 60 dB or more. A loud vowel is far stronger than a quiet consonant, and a vowel's high bands are far weaker than its low ones. A recording adds silence near the noise floor. A log turns ratios into differences. A band 1,000 times weaker than another is 3 units lower in $\log_{10}$, not invisible. This is the decibel scale of §35.1, since $10\log_{10}e$ is the band's level in decibels relative to a power of 1.

Perception. Perceived loudness grows roughly with the logarithm of intensity over much of the hearing range. That is the old Weber-Fechner rule. It is an approximation, since loudness follows a power of intensity with an exponent near 0.3 more closely (Stevens 1957). Either way, equal steps in log energy are closer to equal steps in what a listener hears than equal steps in energy are.

A fixed channel becomes an offset. This is the reason that mattered most for recognizers. A microphone, a telephone line, or a vocal tract acts on sound as a convolution. Each output sample is a weighted sum of recent input samples, and the weights form a short impulse response. In the spectrum that convolution is a multiplication, and the log turns the multiplication into an addition.

In plain words, a fixed microphone or channel multiplies every frame's spectrum by the same curve. After the log, it adds the same number to each band in every frame. Subtracting each band's average over the utterance removes that number. This step is called cepstral mean normalization (CMN). The name comes from the cepstral features later in this section, where it was first used. Those features are a linear map of the log-mel vector, so subtracting the mean before or after the map gives the same result.

The identity is only approximate for real audio, because real frames are not circular convolutions. It holds well when the impulse response is much shorter than the 25 ms window. That is true for microphones and phone lines and false for long room echoes. It also fails at onsets, where a frame holds the start of a sound but not the channel's whole response to it.

The cell tests the theorem with a strong fixed channel, $y[n] = x[n] - 0.9\,x[n-1]$. This filter cuts low frequencies by up to 20 dB and boosts high ones by 5.6 dB. Classical front ends apply the same kind of filter on purpose, with 0.97 in place of 0.9, and call it pre-emphasis. The cell measures how constant the resulting offset is.

Logs of nearly empty bands are unstable, which is why every front end floors the energy before the log. Some, such as Kaldi, also add a tiny random noise to the waveform, called dither, for the same reason. In practice this theorem is why mean normalization is a standard step, and why a model trained on one microphone survives another.

Whisper's front end, line by line

Now the real thing. The Whisper paper (Radford et al. 2022) describes its input as an 80-channel log-magnitude mel spectrogram on 25 ms windows with a 10 ms stride, scaled to lie between $-1$ and $1$ with roughly zero mean. The code in whisper/audio.py is more precise than the prose. When the two disagree, the code is what the checkpoints were trained on. It squares the magnitudes, so it uses power, not magnitude. Its scaling is three fixed operations, not statistics measured on a dataset.

The clamp in step 5 does two jobs. It discards detail more than 80 dB below the peak, which is noise for any realistic recording. It also turns silence and zero padding into one constant value, so a mostly empty 30 s window does not flood the encoder with arbitrary numbers.

The maximum in step 5 depends on what counts as the input. For a long recording, Whisper's transcribe computes the features once for the whole file, with 30 s of zeros appended, and then cuts them into 3,000-frame windows. So the clamp uses the loudest moment of the whole recording, not of each window. A reimplementation that clamps each 30 s window on its own gets slightly different numbers in quiet windows. Try writing the front end yourself before reading our version.

import numpy as np

def whisper_log_mel(audio, mel_fb=MEL_80, n_fft=400, hop=160):
    """audio: (n_samples,) at 16 kHz. Returns Whisper's log-mel features, shape (T, n_mels)."""
    raise NotImplementedError

Here is our version, run on our utterance. The cell pads it to 30 s, as Whisper's pad_or_trim does, and prints the shape and the range of values after every step.

The same shapes, counted in bytes and in encoder positions, are the numbers worth having ready for any model that reads Whisper's features.

So the log-mel is not much of a compression in size. What it buys is structure. It is a smooth image with a small range, whose rows change slowly and whose columns mean the same thing for every speaker. This grid is the "log-mel frames" stage of the speech-to-text walkthrough in §36.1, and §36.4 follows it into Whisper's encoder and decoder.

The same front end in PyTorch, and in the libraries

The block below reproduces log_mel_spectrogram from whisper/audio.py with torch.stft and checks it against the NumPy version. It builds its own copy of librosa's filterbank, which matches the matrix Whisper ships to within about $2\times10^{-9}$. It ends with a batched, loop-free way to draw an independent frequency mask for each example, which SpecAugment, later in this section, needs.

In practice you call a library, and the libraries disagree in their defaults. These are the calls that reproduce Whisper's mel energies, before its log and scaling, in librosa and torchaudio. Every argument that differs from the library's default is one that has broken a deployed model somewhere.

The table lists the defaults side by side. Kaldi's compute-mfcc-feats program is the reference for older recipes. Its default Povey window is a Hann window raised to the power 0.85. It also zero-pads each 400-sample frame to a 512-point FFT, so its spectrum has 257 bins rather than 201, and its filters start at 20 Hz.

The conventions matter just as much in the other direction. Text-to-speech models predict exactly these log-mel matrices (§37.1), and a neural vocoder turns them back into a waveform (§37.2). An acoustic model and a vocoder built with different filterbanks or log bases do not fit together.

The cepstrum separates the vowel from the pitch

Mel pooling and the log fixed the spacing and the range. One problem remains. Each log-mel frame still mixes two kinds of structure, the vowel and the pitch, and a classical recognizer wants only the vowel.

Return to the source-filter picture. The vocal folds produce a buzz, a comb of harmonics at multiples of the pitch $F_0$. The vocal tract shapes that buzz with its formants. In the frequency domain the tract multiplies the source spectrum, $\lvert X(f)\rvert^2 = \lvert S(f)\rvert^2\lvert H(f)\rvert^2$, where $S$ is the source and $H$ the tract. By the theorem above, the log spectrum is then a sum, $\log\lvert S\rvert^2 + \log\lvert H\rvert^2$.

Plotted against frequency, the two terms look very different. $\log\lvert H\rvert^2$ is a slow, smooth curve with a few broad peaks, and it carries the vowel. $\log\lvert S\rvert^2$ is a fast ripple with one peak every $F_0$ hertz, and it carries the pitch. A Fourier transform is exactly the tool that separates a slow pattern from a fast one added on top of it.

The odd names are anagrams of spectrum, frequency, and filtering. Bogert, Healy, and Tukey coined them in 1963 while looking for echoes in seismic records, as told in Oppenheim and Schafer's history of the cepstrum.

In plain words, the cepstrum is the spectrum of the log spectrum. A ripple that repeats every $F_0$ hertz along the frequency axis shows up as a peak at quefrency $1/F_0$ seconds, which is $f_s/F_0$ samples. A smooth envelope lives at small quefrencies. Keep only the small ones, transform back, and you get the envelope without the pitch. The cell does this to one frame of the "hod" vowel.

MFCCs are the cepstrum of the log-mel vector

Mel-frequency cepstral coefficients (MFCCs) apply the same idea to the log-mel vector instead of the full log spectrum, with one change. The log-mel vector is real, short, and not periodic, so MFCCs use the discrete cosine transform (DCT) instead of an inverse DFT. The DCT is the DFT of the vector mirrored onto itself, so it sees no jump at the ends, and it turns real numbers into real numbers.

Davis and Mermelstein (1980) compared several feature sets for recognizing spoken words and found mel-frequency cepstra the best. For the next thirty years they were the standard input to speech recognizers.

In plain words, row $i$ of $\mathbf{D}$ is a cosine that completes $i/2$ cycles across the bands. So coefficient 1 measures the overall tilt of the log spectrum, coefficient 2 its curvature, and higher coefficients ever finer wiggles. Truncating at 13 keeps the slow shape, which is the vowel, and drops the fast ripple, which is the pitch.

Toolkits add small changes on top of this definition, and they matter when you match features. Kaldi, for example, replaces coefficient 0 with the frame's log energy by default. It also multiplies coefficient $i$ by $1 + 11\sin(\pi i/22)$, a fixed weighting called a lifter that boosts the higher coefficients.

Now test the claim that the low cepstral coefficients describe the vowel and not the pitch. The cell builds steady vowels and changes either the pitch or the vowel. It measures how far each change moves the low coefficients $c_1$ to $c_{12}$ and the high ones, $c_{13}$ and up. It does this once with 23 bands, the classical MFCC setting, and once with 80. It leaves out $c_0$, which only tracks the overall level.

The DCT nearly decorrelates the bands

There is a second, statistical reason for the DCT. Neighboring log-mel bands are strongly correlated, because the envelope is smooth and adjacent triangles overlap. Classical recognizers modeled each frame with a Gaussian mixture model (GMM), a weighted sum of a few Gaussian densities. To keep it cheap, each Gaussian had a diagonal covariance matrix, which treats the features as uncorrelated. That is a good model only of features that really are nearly uncorrelated.

The transform that exactly decorrelates a set of vectors is principal component analysis (PCA), which projects onto the eigenvectors of their covariance (§0.10). But PCA depends on the data. The DCT is fixed, and for smooth data it does nearly the same job.

The cell measures it for 40 dimensions and $\rho = 0.95$, and then for realistic log-mel frames.

For the Markov model, the first six DCT rows match the PCA axes with $\lvert\cos\rvert \ge 0.99$. The off-diagonal share of the covariance drops from 0.93 to 0.02. For 400 random vowels at random pitches and levels, the average correlation between bands is 0.63, and between DCT coefficients it is 0.16. The DCT is not the PCA of this data, since real data never has exactly Markov covariance. But it removes most of the correlation without ever looking at the data.

Deltas and the 39-dimensional MFCC vector

A frame describes 25 ms. Speech sounds are defined as much by how the spectrum moves as by where it is. A formant slides into a stop consonant, or a hiss starts suddenly. So classical front ends appended estimates of the time derivative of each coefficient, called deltas, and of the second derivative, called delta-deltas. The usual formula looks arbitrary until you see where it comes from.

A small example shows the formula at work, and why it is less noisy than a plain difference of neighbors.

At the first and last frames the window runs off the sequence, and HTK and Kaldi repeat the edge frame. Delta-deltas apply the same formula to the deltas. Thirteen coefficients plus their deltas and delta-deltas give the famous 39-dimensional vector. HuBERT clustered exactly this vector to make its first round of targets (§35.5).

Beware that librosa.feature.delta uses different defaults. It is a Savitzky-Golay filter, which fits a low-order polynomial to a sliding window of frames and returns its derivative. For first deltas it fits a straight line over 9 frames, so away from the edges it equals the HTK formula with $N_\delta = 4$, not 2. It treats the edges differently too, by fitting the polynomial to the first and last 9 frames instead of repeating the edge frame. Its delta-deltas fit a parabola and take its second derivative, rather than applying the delta formula twice. So its numbers differ from HTK's.

import numpy as np

def mfcc(log_mel, n_ceps=13):
    """log_mel: (T, n_mels). Orthonormal DCT-II over the mel axis, first n_ceps coefficients. Returns (T, n_ceps)."""
    raise NotImplementedError

def deltas(C, N=2):
    """C: (T, d). HTK regression deltas over time with edge replication. Returns (T, d)."""

Normalizing the features

Normalization removes the parts of the features that come from the recording rather than the speech. It comes in two kinds.

Per-utterance cepstral mean and variance normalization (CMVN) subtracts each band's mean over the utterance and divides by its standard deviation. It removes the channel and the speaker's average level. But it needs the whole utterance, which is a problem for streaming, and it is noisy for clips of a second or two. Global normalization uses one mean and one variance per band, computed over the training set. It keeps absolute levels and works frame by frame.

Real models use both kinds. Whisper applies a fixed global map, $(x + 4)/4$. AST normalizes with dataset statistics to mean 0 and standard deviation 0.5. ECAPA-TDNN subtracts each utterance's mean. CMVN is the audio version of normalizing each feature, the same question §5.2 asks about BatchNorm and LayerNorm.

CMVN has one twist that causes real bugs. A batch of utterances of different lengths is padded with zeros to a common length. The padded frames are not speech, so they must be left out of the statistics.

import numpy as np

def cmvn(X, lengths, eps=1e-5):
    """X: (B, T, n_mels) padded batch, lengths: (B,) valid frames per utterance.
    Per utterance and band, subtract the mean and divide by sqrt(var + eps) over valid frames; padding becomes 0."""
    return (X - X.mean(axis=1, keepdims=True)) / np.sqrt(X.var(axis=1, keepdims=True) + eps)   # uses the padding

Why deep models went back to log-mels

Every argument for MFCCs was an argument about the model that read them. Truncating to 13 coefficients made the features small and nearly free of pitch for Gaussian mixtures. Decorrelating them made diagonal covariances adequate. Neural networks need neither. A network can learn to ignore pitch if pitch does not help. Its first layer can learn any linear map of its input, so decorrelating in advance buys nothing.

What the DCT costs is locality, the property that nearby numbers in the input describe nearby frequencies. Every MFCC mixes all the bands. A 2-D convolution or a patch Transformer looks at a small time-frequency neighborhood, such as a formant, a harmonic, or a burst at 5 kHz. In MFCCs that neighborhood is no longer in one place. Log-mels keep it, which is why spectrogram CNNs and the Audio Spectrogram Transformer (§35.6) treat the log-mel as an image. Truncation also throws away information a network could use, including the pitch that tonal languages, emotion, and speaker identity depend on.

The evidence is empirical. By 2013, groups building neural recognizers reported that log-mel features worked better than MFCCs (Deng et al. 2013). Every model in the box on the log-mel inputs of real models, above, reads log-mels except two, and the two exceptions make the point. ECAPA-TDNN keeps 80 coefficients rather than the classical 13, so little or nothing is truncated. An untruncated DCT is an invertible linear map of the log-mel vector, so it loses nothing. Its first layer is a convolution over time that treats the 80 values as channels, so there was no frequency locality to lose. HuBERT used MFCCs only to make its first clustering targets, never as the input to its Transformer.

The table puts the options side by side. GMM-HMM systems are the classical recognizers that modeled each frame with a Gaussian mixture and the sequence of sounds with a hidden Markov model (§36.1). PCEN, SincNet, and LEAF are the learnable front ends at the end of this section.

SpecAugment hides bands and stretches of the input

Once the input is a log-mel image, it is natural to augment it the way images are augmented. SpecAugment (Park et al. 2019) hides random blocks of the log-mel spectrogram during training, whole bands of frequency and whole stretches of time. It is the audio cousin of cutout and random erasing, the augmentations that blank out patches of training images for vision models (§27.4). It runs on the features on the fly, costs almost nothing, and needs no extra data. In 2019 it set new state-of-the-art results on both standard English benchmarks.

The paper's four hand-designed policies are LibriSpeech basic (LB) and double (LD), for audiobook speech, and Switchboard mild (SM) and strong (SS), for telephone conversations.

The widget draws a synthetic log-mel spectrogram of speech with the masks of one of the paper's four policies, without time warping. Each mask is outlined and filled with the mean, the flat value the model sees. Pick a policy and a clip length, and press Play to draw new masks, as every training epoch does, and see how much each draw hides.

import numpy as np

def spec_augment(X, F, m_F, T_max, m_T, p, rng, mask_value=0.0):
    """X: (T, n_mels) log-mel features. Returns a masked copy; X itself is not changed."""
    return X.copy()                       # placeholder, no masks at all

With the exercise's function in hand, the cell measures what each policy hides on a 3 s and a 12 s clip.

Between a fifth and a half of every training example disappears, and the model gets better. To read the results, three terms help. The models were LAS recognizers (Chan et al. 2015), attention encoder-decoders built from LSTMs (§3.3), whose name stands for "listen, attend and spell". Accuracy is measured by the word error rate (WER, lower is better), the number of word substitutions, deletions, and insertions needed to turn the output into the reference transcript, divided by the number of reference words. Shallow fusion adds a separately trained language model's score during decoding.

The paper's ablation shows which part matters. It used the smaller LAS-4-1024 model, policy LB, and test-other with no language model. The full policy gave 10.0 percent WER, against 13.4 percent without any augmentation. Removing frequency masking raised it to 11.0, removing time masking to 10.9, and removing time warping only to 10.1. The authors conclude that time warping, the most expensive and least useful of the three, is the first to drop, and many later recipes do drop it.

Why does hiding data help this much? The paper's own observation is the key. With SpecAugment the network underfits even its training set. So the authors made it wider and deeper and trained it longer, which an overfitting model could not use. The usual explanation is that the masks force redundancy. A model that cannot rely on any single band or any 100 ms stretch must recognize a speech sound from several cues at once, its formants, its neighbors, and its transitions. That is the robustness that noise, band-limited channels, and dropouts demand at test time.

Front ends that learn their own filters

Every step above was designed by hand. The filter shapes, the spacing, the log, and the normalization all encode guesses about hearing. A middle path keeps the structure, a bank of band-pass filters followed by pooling and compression, and learns its few parameters together with the rest of the model.

The most widely used piece is a learned replacement for the log. Per-channel energy normalization (PCEN), from Wang et al. (2017), divides each band's energy by a running average of that band's own recent energy. A band that has been loud for the last fraction of a second is turned down, and a quiet one is turned up. Then a root, rather than a log, compresses the result.

In symbols, let $e_{tj}$ be the mel energy of band $j$ at frame $t$, and let $M_{tj} = (1 - s)M_{t-1,j} + s\,e_{tj}$ be its running average, a first-order smoother with smoothing coefficient $s$. Then

where $\epsilon$ is a small constant, $\alpha$ sets the strength of the gain control, and $\delta$ and $r$ set the root compression.

So PCEN is mean normalization done adaptively, per band, over the last fraction of a second. It follows a talker who walks away from the microphone. It was introduced for keyword spotting with distant microphones, where it improved robustness, and its parameters can be trained.

These front ends help most where hand-designed hearing is the wrong prior, such as animal sounds, machine sounds, and speech from distant microphones. They help least where the mel prior is right and data is plentiful. They also raise an obvious question. If a few hundred parameters can learn better filters, why not learn everything from raw audio, the filters, the pooling, and the representation itself, and from unlabeled audio at that? That is wav2vec 2.0, a stack of seven convolutions that turns the waveform into one frame every 20 ms, trained with no transcripts at all (§35.4).

Read more

• Davis and Mermelstein (1980). Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Sentences. The paper that made mel-frequency cepstral coefficients the standard speech feature for three decades.
• Park et al. (2019). SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. The three operations, the four policies, and the ablation, in a short and readable paper.
• Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. Whisper's paper. Read its input description next to the code in the openai/whisper repository, whose audio.py is the precise definition of the front end.
• Slaney (1998). Auditory Toolbox. The technical report behind the Slaney mel scale and area-normalized filters that librosa, and through it Whisper, still use.
• Wang et al. (2017). Trainable Frontend For Robust and Far-Field Keyword Spotting. Per-channel energy normalization, the adaptive replacement for the log.
• Zeghidour et al. (2021). LEAF: A Learnable Frontend for Audio Classification. Gabor filters, learned pooling, and learned PCEN as a drop-in replacement for mel filterbanks.
• Ravanelli and Bengio (2018). Speaker Recognition from Raw Waveform with SincNet. Band-pass filters with two learned cutoffs each, the simplest learnable filterbank.
• Jurafsky and Martin. Speech and Language Processing, 3rd edition. Chapter 15 covers the mel scale, log-mel features, and MFCCs from the speech side, and Chapter 16 places them in the history of recognizers.
• Oppenheim and Schafer (2004). From Frequency to Quefrency: A History of the Cepstrum. How the cepstrum went from echo detection in seismic data to the core of speech features.

Watch next

▶

▶

▶