From air pressure to the frame embeddings every speech and audio model reads. Sampling, the Fourier transform, mel spectrograms, and the self-supervised encoders (wav2vec 2.0, HuBERT, WavLM) behind modern speech systems, built from first principles to the audio encoders of 2026.

This chapter begins the book's part on audio and speech. It does for sound what earlier parts did for text and images. Text became a sequence of token vectors (Chapter 1 and Chapter 2), and an image became a grid of patches, each turned into a vector (§27.1). This chapter turns a recording into the sequence of frame vectors that every speech and audio model reads. Chapter 36 then turns those vectors into text, and Chapter 37 turns text back into sound with the generative tools of Chapter 33 and the codes of §32.5. Frontier chat models now listen and speak, so these representations sit at the front of many products.

This section is about the list of numbers a computer stores when it records sound, and what that list keeps and what it loses. A microphone turns air pressure into a voltage. A converter measures that voltage at evenly spaced instants, 16,000 times per second for a speech model, and rounds each measurement to one of a fixed set of levels. Every audio model, from a speech recognizer to an audio language model, reads that list or something computed from it. So the two choices behind it, how often to measure and how finely to round, set a hard limit on what any model can hear.

§1.8 sketched what happens next. A front end, the first stage of an audio model, turns the list into a sequence of frame vectors, one for every short slice of time. One kind of front end is a log-mel spectrogram, a grid that records how much energy each band of frequencies holds in a short window taken every 10 ms, so 100 frames per second. The other is a stack of learned convolutions, which gives 50 frames per second. This chapter builds both. This section starts one step earlier, with the numbers themselves.

Two bugs show why the details matter, and practitioners say most audio teams ship each of them at least once. In the first, music recorded at 44,100 samples per second has to be fed to a speech model that expects 16,000. The quick fix keeps, for each output time, the nearest input sample. The result sounds roughly right, except that a cymbal shimmering near 12,000 cycles per second now carries a steady whine at 4,000 that nobody played. In the second, a corpus recorded at 8,000 samples per second is read as if it were 16,000. Every voice comes out an octave higher and twice as fast, and the model's accuracy collapses with no error message anywhere.

The first bug comes from one fact. Samples taken at some rate can only represent frequencies below half that rate. Anything above it does not vanish. It comes back at a predictable wrong frequency. The second bug comes from another fact. The samples do not record their own rate, so a wrong rate in a file header stretches or squeezes everything in time and frequency, and nothing complains. This section makes the first fact exact, and its last part shows how to guard against the second. You will compute the wrong frequency in your head, prove the theorem that says when sampling loses nothing, and resample without the bug. Then you will derive how much noise rounding adds (about 6 decibels less for every extra bit), see why adding a little noise on purpose makes rounding better, mix speech and noise at an exact signal-to-noise ratio, and do the storage arithmetic for real datasets and models.

Sound is a pressure wave

Sound is a small, fast change in air pressure that travels. A vibrating object, such as a guitar string, a loudspeaker cone, or your vocal folds, pushes on the air next to it. That air pushes on its neighbors, and a pattern of squeezed and stretched air moves outward at the speed of sound, about 343 m/s in dry air at 20 °C. The air itself does not travel with the wave. Each small parcel only jiggles around its resting place.

The changes are tiny. Atmospheric pressure is about 101,325 pascals (Pa). Ordinary conversation, heard from a meter away, moves the pressure by about 0.02 Pa, two parts in ten million. Even a sound loud enough to hurt changes it by only tens of pascals. A microphone turns this pressure deviation into a voltage proportional to it. Call the deviation at the microphone $x(t)$, a real number for every time $t$ in seconds. That function is the waveform, and everything in this chapter is a way of describing it.

Pure tones

The simplest waveform is a pure tone, a cosine.

Two numbers anchor the range. Young adults hear from about 20 Hz to about 20 kHz, and the upper limit falls with age. At 343 m/s those are wavelengths from 17 m down to 1.7 cm, and a 1 kHz tone has a wavelength of 34.3 cm. A higher frequency is heard as a higher pitch. Doubling the frequency raises the pitch by one octave, and an octave splits into 12 semitones, the steps of a piano keyboard.

Real sounds are rarely pure tones, but many are sums of them. Take a sound that repeats itself every $1/F_0$ seconds, such as a sung vowel or a bowed string. It can be written as a sum of cosines at whole-number multiples of $F_0$,

A general periodic signal also has a constant term $A_0$. For sound it is zero, because pressure deviations average to zero over a period. The component at $F_0$ is the fundamental. The components at $kF_0$ for $k \ge 2$ are its harmonics, and $F_0$ is what we hear as the pitch. The amplitudes $A_k$ shape the timbre, the color of the sound, which is one reason a violin and a flute playing the same note sound different. This sum is the Fourier series of a periodic signal. It is the first appearance of the idea that drives §35.2, that a waveform which looks complicated in time can be simple when described by its frequencies.

Voices as a source and a filter

Speech adds one more layer, and the mel features of §35.3 are built on it. In a voiced sound such as a vowel, the vocal folds open and close periodically and release puffs of air. That buzz is the source. It is periodic, so it is a fundamental plus a dense comb of harmonics. A typical adult male speaking voice has $F_0$ near 100 to 120 Hz and a typical adult female voice near 200 Hz, with wide variation between people and across a sentence.

The throat, mouth, and lips then act as a filter, something that boosts some frequencies and weakens others. Their shape has resonances that boost the harmonics near certain frequencies. Those resonant frequencies are the formants, labeled F1, F2, F3 from the lowest up. Moving the tongue and lips moves the formants, and the first two largely identify the vowel. In classic measurements of American English vowels (Peterson and Barney 1952), the average F1 and F2 of men were about 270 and 2,290 Hz for the vowel of "heed", 730 and 1,090 Hz for the vowel of "hod", and 300 and 870 Hz for the vowel of "who'd".

Sounds such as /s/ and /f/ have no periodic source at all. They are turbulent noise from air forced through a narrow gap, with much of their energy above 4 kHz. That fact will matter when we decide how fast to sample.

In plain words, pitch lives in the spacing of the harmonics, and the vowel lives in the shape of their envelope. A recognizer for a language such as English must read the words whoever speaks them, so it reads the envelope and largely ignores the spacing. In a tonal language such as Mandarin the pitch contour changes the word, so the spacing matters too. The cepstrum of §35.3 separates the two approximately. A speaker model needs both, the speaker's pitch range and the fine shape of the envelope that one person's vocal tract gives it. An interviewer may well ask which of them a given feature keeps.

The cell builds 30 ms of a synthetic "hod" vowel this way. The source is a 120 Hz buzz whose harmonic amplitudes fall off as $1/k$. It passes through three resonances, at the F1 and F2 of "hod" above (730 and 1,090 Hz) and at its F3, 2,440 Hz, from the same measurements. The $1/k$ is a common rough model. Levels here are in decibels (dB), $20\log_{10}$ of a ratio of amplitudes, which this section defines properly later. The glottal buzz falls about 12 dB per octave, and radiation from the lips adds back about 6. The net fall of 6 dB per octave halves the amplitude each time $k$ doubles, which is exactly $1/k$.

The waveform repeats every $1/120$ s $\approx 8.3$ ms, the period of its 120 Hz pitch. The spectrum's local peaks sit at the harmonics closest to the formants, 720, 1,080, and 2,400 Hz, which is the vowel. Change F0 to 200 and the comb spreads out while the envelope stays where it was. That is the same vowel spoken at a higher pitch.

From a waveform to a list of numbers

A computer cannot store $x(t)$ for every real $t$. An analog-to-digital converter (ADC) measures it at evenly spaced instants and rounds each measurement to one of finitely many levels. The first step is sampling and the second is quantization. We take them in turn, and for each we ask exactly what is lost.

For a speech model, $f_s = 16{,}000$ Hz, so $T_s = 62.5\ \mu$s and the Nyquist frequency is 8 kHz. For a CD, $f_s = 44{,}100$ Hz and the Nyquist frequency is 22.05 kHz, just above the edge of hearing. That is not a coincidence, and the next two parts explain it.

Two tones that give the same samples

Take a slow sampling rate so the numbers stay small, $f_s = 8$ Hz. Sample two tones at $t = n/8$ for $n = 0, \dots, 7$. One is $\cos(2\pi\cdot 1\cdot t)$, a 1 Hz tone. The other is $\cos(2\pi\cdot 7\cdot t)$, seven times faster.

Once the samples are taken, no algorithm can tell which tone produced them, because the two lists are identical. This is aliasing, one frequency showing up as another. The cell checks it, and adds a 9 Hz tone that hides in the same eight numbers.

The folding formula

The example generalizes in two lines. Let $j$ be any integer. Because $f_sT_s = 1$, adding $jf_s$ to the frequency adds $2\pi jn$, a whole number of turns, to the angle at every sample,

Replacing $f$ by $jf_s - f$, its mirror image about $jf_s/2$, changes nothing except the sign of the phase,

In plain words, after sampling, two frequencies that differ by a multiple of $f_s$ look the same. So do two frequencies that mirror each other about a multiple of $f_s/2$, such as 1 Hz and 7 Hz about 4 Hz when $f_s = 8$ Hz. So every frequency has a stand-in between 0 and $f_s/2$, and that stand-in is what the samples report.

Plot $f_{\text{alias}}$ against $f$ and you get a zigzag. It rises from 0 to $f_s/2$, falls back to 0 at $f_s$, rises again, and so on. It is as if the frequency axis were a strip of paper folded like a fan at every multiple of $f_s/2$. That is where the name comes from, and why $f_s/2$ is also called the folding frequency. The stretches between folds are the Nyquist zones.

In practice, this is the first thing to compute when a spectrogram, the grid of energy by frequency and time, shows a tone nobody played. Interviewers like it for the same reason. They give you a frequency and a rate and ask where it lands.

The widget samples one tone and draws both sides of the formula. The top panel shows the tone, its samples, and the slowest sine through those samples, which is what the samples say. The bottom panel is the zigzag, with the reported frequency plotted against the tone's frequency and the zones where the phase flips shaded.

import numpy as np

def alias_frequency(f, fs):
    f = np.asarray(f, dtype=float)
    return f % fs          # wrong: wraps but does not fold

The sampling theorem

Aliasing says what goes wrong. The sampling theorem says when nothing goes wrong. It is a theorem in the strict sense, with a short proof. To state it we need a way to talk about the frequency content of signals that are not single tones.

The Fourier transform in one paragraph

Recall Euler's formula from §5.5, $e^{\mathrm{i}\theta} = \cos\theta + \mathrm{i}\sin\theta$. Here $\mathrm{i}$ is the imaginary unit, and $e^{\mathrm{i}\theta}$ is the point on the unit circle at angle $\theta$. So $e^{2\pi\mathrm{i}ft}$ is a point going around the circle $f$ times per second, the rotating point of the strobe picture. A real cosine is the average of two such points turning in opposite directions,

That is all "negative frequency" means. It is the mirror half of every real cosine.

In plain words, $X(f)$ is a complex number. Its magnitude says how much rotation at frequency $f$ the signal contains, and its angle says where that rotation starts. The inverse rebuilds the signal by adding all the rotations back with those weights. Here we take the pair as given, for signals of finite energy ($\int x(t)^2\,dt < \infty$). §35.2 derives its discrete cousin, the discrete Fourier transform (DFT), from scratch as a change of basis, and proves its inverse. For a real signal $X(-f) = \overline{X(f)}$, the complex conjugate, so the positive frequencies carry all the information.

What sampling does to the spectrum

One fact does all the work. It says exactly what sampling does to the frequency content, and both the theorem and aliasing follow from it.

The theorem

Picture the copies. The original spectrum is a bump between $-f_{\max}$ and $f_{\max}$. Sampling places a copy of the bump at every multiple of $f_s$. If $f_s$ is more than twice $f_{\max}$, the copies sit apart with gaps between them. Then the copy in the middle is untouched, and keeping it while deleting the rest gives back the original signal. If $f_s$ is too small, neighboring copies overlap, and nothing can pull them apart again.

In plain words, put a sinc bump at every sample, scale it by the sample's value, and add the bumps up. Each bump equals 1 at its own sample time and 0 at every other sample time, because the sinc is zero at every nonzero integer. So the sum passes through every sample. Between samples it traces the one curve with no content above $f_s/2$ that does so. In the frequency picture, summing sincs is an ideal low-pass filter, one that keeps the middle copy of the spectrum and deletes all the others.

The same picture explains aliasing. If $f_s \le 2f_{\max}$, neighboring copies overlap (at equality they touch). Energy from the copy centered at $f_s$ then lands inside $[-f_s/2, f_s/2]$, and no filter can separate it from the original. For a single tone at $f$, the spectrum is two spikes at $\pm f$. The copies put spikes at $\pm f + jf_s$, and the one that lands between 0 and $f_s/2$ sits at exactly the $f_{\text{alias}}$ of the folding formula.

The theorem has three practical consequences.

• The ideal interpolator cannot be built. The sinc decays only like $1/u$ and extends forever in both directions, so exact reconstruction needs every sample, past and future. Real converters use finite filters. Those need a guard band, a gap between the highest frequency they keep and the Nyquist frequency. CD audio keeps up to 20 kHz and puts the Nyquist frequency at 22.05 kHz, which leaves 2.05 kHz for the filters.
• Content above $f_s/2$ must be removed before sampling. Once it reaches the converter, it folds into the band for good. So every ADC has an anti-alias filter in front of it, an analog low-pass filter. Modern converters first sample many times faster than needed, where a gentle analog filter is enough. They then filter sharply in the digital domain and reduce the rate, which is the decimation we build below.
• Nothing below the Nyquist frequency is lost. A sampled waveform is not a staircase approximation of the original, despite the common picture. Up to the guard band and the rounding noise of the next parts, it is the original.

An interviewer who asks you to "state the sampling theorem" wants all three pieces, the band limit, the strict inequality and why practice needs it, and the sinc reconstruction, plus the guard band that practice adds.

The cell rebuilds a band-limited signal from its samples with the sinc sum. Then it feeds the same machinery a tone above the Nyquist frequency.

The first error is about $6\times10^{-5}$. It is not zero only because we kept 2,001 samples instead of infinitely many, and the sinc tails we dropped decay like $1/u$. The second line shows the other side of the theorem. The samples of a 70 Hz tone are rebuilt, to within the same truncation error ($2\times10^{-4}$), as the 30 Hz tone with negated phase that the folding formula predicts, because that is the only band-limited signal those samples describe.

Anti-aliasing and resampling

Back to the opening bug. A model trained at 16 kHz must be fed 16 kHz audio, so everything recorded at another rate has to be converted. That conversion is resampling, and it is where most of the aliasing in ML pipelines is born, long after the microphone's own filter did its job. Resampling is sampling again, so the theorem applies again. Before the rate goes down, everything above the new Nyquist frequency has to go.

Filters in plain words

Removing frequencies is the job of a filter, and a few terms carry the rest of this section. The filters here replace each sample by a weighted sum of its neighbors, with the same weights at every position.

§35.2 makes all of this formal with the frequency response and the convolution theorem. For now one fact matters, and it needs only trigonometry.

In plain words, a symmetric filter weighs the past and the future equally, so it cannot push a tone later or earlier. Where $G(\theta)$ is negative the tone is flipped in sign. A low-pass filter does that only in its stopband, where the gain is tiny anyway.

Two practical consequences follow. A real-time filter cannot look at future samples. So it stores the same taps at indices $0$ to $n_{\text{taps}} - 1$, and its output is $y[n - K]$, the same tone delayed by $K = (n_{\text{taps}} - 1)/2$ samples. An offline pipeline can use the centered form directly, and np.convolve(x, h, mode="same") with an odd $n_{\text{taps}}$ does exactly that, as long as x is at least as long as h (otherwise it returns len(h) values). An even $n_{\text{taps}}$ has no center tap, so its center falls between two taps, half a sample off.

Changing the rate

There are three cases, and each follows from the theorem.

Downsampling by an integer factor $q$, called decimation. Low-pass filter at the new Nyquist frequency, $f_s/(2q)$, then keep every $q$-th sample. Skipping the filter is the strobe light again. Content between $f_s/(2q)$ and $f_s/2$ folds into the new band.

Upsampling by an integer factor $L$. Insert $L - 1$ zeros between samples. That raises the rate but leaves copies of the spectrum, called images, at multiples of the old rate. So low-pass filter at the old Nyquist frequency to remove them, and multiply by $L$ to restore the level the zeros diluted.

Rational factors $L/M$. Upsample by $L$, filter once at the smaller of the two Nyquist frequencies, and downsample by $M$. Measure frequencies here in cycles per sample, a frequency divided by the sampling rate, so the Nyquist frequency is always 1/2. At the upsampled rate $Lf_s$, the old Nyquist frequency is then $1/(2L)$ and the new one is $1/(2M)$. So the one filter has cutoff $\min\big(1/(2L), 1/(2M)\big)$ cycles per upsampled sample. Going from 44.1 kHz to 16 kHz is $L/M = 160/441$. Done literally, this upsamples to 7.056 MHz, filters 7 million values for every second of audio, and then throws away 440 of every 441.

The polyphase form avoids that waste by computing only the outputs it keeps. Each output then costs about $n_{\text{taps}}/L$ multiplications, no inserted zero is ever stored or multiplied, and the long upsampled signal never exists. scipy.signal.resample_poly(x, 160, 441) does exactly this. The box below shows how, and Exercise 35.1f builds it.

The low-pass filter is a finite sinc

The low-pass filter is the sinc of the theorem made finite. Measure frequency in cycles per sample again, so the Nyquist frequency is $1/2$. The ideal low-pass filter with cutoff $f_c$ has gain 1 for $\lvert f\rvert < f_c$ and gain 0 above it. Its taps are

which the derivation below proves. These taps go on forever, decaying like $1/m$. Real filters keep a few dozen of them and multiply by a window, a smooth taper that falls to zero at both ends. Common windows are the Hann window, a raised cosine, and the Kaiser window, whose shape parameter $\beta$ trades a narrower transition band against higher stopband ripple. With a Hann window the result is a filter whose transition band is about $3.2/n_{\text{taps}}$ cycles per sample wide, with stopband ripples about 44 dB down. A Kaiser window can go lower for the same length. Why the window matters, and how its shape trades transition width against ripple height, is the spectral leakage story of §35.2.

The opening bug, measured

The cell runs the opening bug for real. It builds 0.2 s of a 1 kHz tone plus a 12 kHz "cymbal" at 44.1 kHz and converts it to 16 kHz three ways. Then it reads the level of each frequency from one value of the result's fast Fourier transform (FFT), an algorithm that measures how much of each frequency a block of samples contains. The FFT is a black box until §35.2 derives it. All we use is this. With $N$ samples at rate $f_s$, a tone of amplitude $A$ at frequency $kf_s/N$ (a whole number of cycles in the block) puts magnitude $AN/2$ in output $k$, which is called bin $k$.

Nearest-sample resampling keeps the 1 kHz tone and produces the 4 kHz alias at almost full strength, $-1.1$ dB. Linear interpolation is no better. Both are weak low-pass filters in disguise. Taking the nearest sample is the same as convolving the samples with a box one input sample wide, and that box's gain at frequency $f$ is $\operatorname{sinc}(f/f_s)$. At 12 kHz that is $0.88$, the $-1.1$ dB above. Linear interpolation draws straight lines between samples, which is the same as convolving them with a triangle two input samples wide. The triangle's gain is $\operatorname{sinc}^2(f/f_s)$, which at 12 kHz is $0.78$, or $-2.2$ dB, exactly the alias level printed.

The 100 Hz junk has the same cause. The 44.1 kHz samples also carry an image of the 12 kHz tone at $44.1 - 12 = 32.1$ kHz. These weak filters pass part of it, $\operatorname{sinc}(32.1/44.1)$ is $-9.6$ dB and its square is $-19.3$ dB, and at 16 kHz it folds to $\lvert 32.1 - 32\rvert = 0.1$ kHz. Only the real low-pass filter pushes the alias and the junk down, to about $-68$ and $-92$ dB.

The table compares the ways you will see resampling done.

In practice the rule is short. Never change a rate by slicing or interpolating. Call a resampler with a real low-pass filter, and check its output rate. In an interview, "a colleague used x[::3], what broke?" is a standard question, and the answer is the 48 kHz example above.

import numpy as np

def decimate(x, q):
    x = np.asarray(x, dtype=float)
    return x[::q]          # no anti-alias filter yet

import numpy as np

def resample_rational(x, up, down):
    x = np.asarray(x, dtype=float)
    n_out = -(-len(x) * up // down)
    t = np.arange(n_out) * down / up                  # output times, in input samples
    return np.interp(t, np.arange(len(x)), x)         # linear interpolation, no anti-alias filter

Measuring size in decibels

Before quantization, we need the unit every audio number is quoted in. Sound spans an enormous range. The quietest audible sound and the loudest bearable one differ in pressure amplitude by a factor of about a million. So audio works with logarithms of ratios, which turn that factor into a number near 120.

For orientation in dB SPL, a quiet bedroom at night is around 30, conversation at a meter around 60 (the 0.02 Pa from earlier), heavy traffic around 85, and the threshold of pain around 120 to 130. The cell checks the rules of the box.

The uncorrelated sum comes out near $+3.01$ dB rather than exactly, because two finite noise samples are never perfectly uncorrelated. An interviewer asks "why is doubling the amplitude 6 dB but doubling the power 3 dB?" to see whether you know that decibels always measure power.

Quantization and bit depth

Sampling made time discrete. The converter must also make the values discrete, because each sample is stored in a fixed number of bits. It does this by rounding each value to the nearest of a set of evenly spaced levels.

The standard format for uncompressed audio is pulse-code modulation (PCM), which stores each sample as a plain integer. 16-bit PCM, the format of CDs and of most WAV files, is exactly this quantizer, with integer codes from $-32{,}768$ to $32{,}767$ and $\Delta = 1/32{,}768$. Whisper's loader reads those integers and divides by 32,768.0, which maps them onto exactly these levels in $[-1, 1)$. Note the asymmetry. The most negative level is $-1$ but the most positive is $1 - \Delta$, so a float sample of exactly $+1.0$ cannot be stored without clipping.

The noise model and the 6 dB per bit rule

For a busy signal such as music or speech, which crosses many levels and is not locked to the step grid, the rounding error behaves like random noise. The model says the error in each sample is spread evenly over $[-\Delta/2, \Delta/2]$, unrelated to the signal and to the errors in other samples. This is a model, not a theorem, and we will see exactly where it fails. Under it, the error has mean 0 and, by the variance of a uniform random variable (§0.4),

The derivation takes three lines. A sine of amplitude $A$ has power $A^2/2$, because $\cos^2\theta = (1 + \cos 2\theta)/2$ averages to $1/2$ over a cycle. With $\Delta = 2^{1 - n_{\text{bits}}}$, the ratio is

Taking $10\log_{10}$ term by term gives $10\log_{10}1.5 + 2n_{\text{bits}}\cdot 10\log_{10}2 + 20\log_{10}A = 1.76 + 6.02\,n_{\text{bits}} + 20\log_{10}A$.

In plain words, every bit halves the step, which halves the error amplitude and removes 6.02 dB of noise. The constant $1.76 = 10\log_{10}(3/2)$ dB is what remains of the ratio once the powers of 2 are pulled out. And a quieter signal loses SQNR decibel for decibel, because the noise floor stays put while the signal drops toward it.

Sixteen bits span about the distance from a quiet room to the threshold of pain. That is why 16-bit storage is enough for every speech model, and why 24-bit recording is about headroom while recording rather than about what anyone hears. The same $\Delta^2/12$ is the starting point for rounding model weights to 8 or 4 bits (§20.2), where it measures the error one rounding step adds to each weight.

import numpy as np

def quantize(x, n_bits):
    raise NotImplementedError

def sqnr_db(x, xq):
    raise NotImplementedError

Where the model fails, and what dither does

The uniform-error model fails when the signal is simple and small compared to $\Delta$. Then the error is a fixed function of the signal. For a pure tone the error repeats with the tone, so its power gathers at the tone's harmonics. Quantization stops sounding like a faint hiss and starts sounding like distortion, buzzy and tied to the music. At the extreme, a tone smaller than $\Delta/2$ rounds to silence. This happens in quiet fades, in reverb tails, and whenever audio is reduced to fewer bits.

Dither is the fix, and it sounds backward at first. You add a little random noise before rounding. The noise jitters each sample across neighboring levels. So on average the output tracks the input even between levels, and the error becomes a steady hiss whose level no longer depends on the signal.

TPDF is the standard choice, and one uniform is not enough, as the next box shows.

The price of TPDF dither is the ratio $(\Delta^2/4)/(\Delta^2/12) = 3$, which is $10\log_{10}3 = 4.77$ dB more noise. At 16 bits that turns 98.1 dB of SQNR into 93.3 dB, still far more than any listener or model needs. In exchange, the error is a steady hiss whose mean and power carry no trace of the signal. The theorem fixes only those two moments, not the whole distribution of the error, and the optional box below says what is left.

Two refinements are worth knowing by name. Mastering engineers use noise shaping, which feeds the error back through a filter so that its spectrum moves toward frequencies where the ear is least sensitive. Another way to spend few bits well is to space the levels unevenly, finely near zero and coarsely for loud values. That is companding, used by G.711 telephony at 8 bits per sample, with the mu-law in North America and Japan and the similar A-law elsewhere. The mu-law is also what WaveNet used to turn audio into 256 classes (§37.2).

The cell measures all of this. The first part sweeps the bit depth for a full-scale sine and a $-40$ dBFS sine, with and without TPDF dither, against the theory line. The second part checks the conditional error power of the theorem box, in units of one step, at eleven positions between two levels.

Read the plot from right to left. The undithered full-scale curve sits on the dotted theory line (computed for the same amplitude) from 16 bits down to about 4, and the dithered one runs about 4.8 dB below it. The undithered $-40$ dBFS curve wobbles within about 2 dB of its line from 7 bits up, because a quiet tone spans few steps and the error is no longer uniform noise. At 6 bits and below it is exactly 0 dB. There the whole sine is smaller than half a step and rounds to silence, and an error equal to minus the signal is an SQNR of 0 dB. The dithered curve keeps falling smoothly below 0 dB, because the signal is still in there, under the noise. The printed rows confirm the theorem box, $r(1-r)$ for RPDF, rising from 0 to 0.25 and back, and a flat 0.25 for TPDF.

The widget looks at the same rounding error in frequency, where distortion and noise look different. Distortion is a few tall spikes at the harmonics of the tone. Noise is a flat floor spread over every frequency.

import numpy as np

def tpdf_dither(n, delta, rng):
    return (rng.random(n) - 0.5) * delta          # one uniform: this is RPDF, not TPDF

def dither_report(x, n_bits, fs, f0, rng=None):
    raise NotImplementedError

Loudness, noise, and mixing at a target SNR

Decibels have two everyday jobs in an audio pipeline. One is setting how loud the added noise is during training, and the other is comparing how loud recordings are. The second needs more than a plain decibel, because the ear does not hear all frequencies equally.

Mixing speech and noise at an exact SNR

The most common audio augmentation is adding background noise at a chosen SNR. Speech recognizers, speaker models, and enhancement models are routinely trained on clean speech mixed with recorded noise, with the SNR drawn at random from a range such as 0 to 20 dB, so that they hold up in kitchens and cars.

The recipe follows from the definition. Keep the speech $s$ as it is, and scale the noise $v$ by an amplitude gain $g$. We write $v$ rather than $n$ because $n$ is the sample index. Choose $g$ so that

where $P_s$ and $P_v$ are the mean squares of the speech and of the noise segment actually added. The mix is $s + gv$.

In plain words, first make the noise as loud as the speech, then turn it down by the target SNR in decibels. The exponent is $\mathrm{SNR}/20$ because $g$ is an amplitude gain.

Three details separate a correct augmentation pipeline from a subtly wrong one.

• Measure the noise power on the segment you add, after looping or cropping it to the speech length. Noise recordings change over time, so the power of the whole file is not the power of the slice.
• Use means, not sums, for both powers. Sums silently depend on the two lengths.
• Decide what "speech power" means. Averaging over an utterance with long pauses underestimates the level of the speech itself. That is why telephony standards measure an active speech level that ignores the silences (ITU-T P.56). Many training pipelines use the plain mean square anyway, which is fine as long as it is used consistently.

import numpy as np

def mix_at_snr(speech, noise, snr_db, rng):
    speech = np.asarray(speech, dtype=float)
    seg = np.resize(np.asarray(noise, dtype=float), len(speech))
    g = 10 ** (-snr_db / 20)                          # wrong: ignores the two powers
    return speech + g * seg, g * seg

Decibels are not loudness

A decibel meter weighs every frequency equally, and the ear does not. Hearing is most sensitive between roughly 2 and 5 kHz and much less sensitive at low frequencies, especially at low levels. So a 50 Hz hum and a 3 kHz beep at the same dB SPL do not sound equally loud. The curves of equal perceived loudness across frequency are standardized as the equal-loudness contours (ISO 226). The old fix was A-weighting, a fixed filter that roughly follows one of those contours, and dB(A) is still what noise regulations quote.

For recorded programs, such as broadcasts, podcasts, and music, the modern measure is loudness in LUFS (loudness units relative to full scale), defined in ITU-R BS.1770. It weights frequencies roughly the way the ear does and ignores silence. One loudness unit (LU) is one decibel, so raising the level by 1 dB raises the reading by 1 LU. European broadcasters normalize programs to $-23.0$ LUFS (EBU R 128), and music streaming services normalize to louder targets, often around $-14$ LUFS.

Loudness matters to models in a quieter way. Suppose one data source is systematically louder than another, and the sources correlate with the labels. Then a classifier can learn "loud means class A". Normalizing levels, or randomizing the gain during training, removes that shortcut.

Rates, sizes, and what models read

Here are the sampling rates you will meet. For each, the table gives the Nyquist frequency, how many samples fill Whisper's 30 s input window, and where the rate is used.

Why 16 kHz for speech? An 8 kHz Nyquist frequency keeps nearly everything that tells speech sounds apart, including much of the hiss of /s/ and /f/. Telephony at 8 kHz keeps only about 300 to 3,400 Hz. Losing the band above 4 kHz is a large part of why "s" and "f" are easy to confuse on the phone. It is also why automatic speech recognition (ASR) is harder on phone calls. Music needs the full 20 kHz of hearing, hence 44.1 or 48 kHz. The cost of a higher rate is sequence length. A model reading raw samples at 48 kHz has three times as many inputs per second as at 16 kHz. A model reading frames has three times as many samples per frame to summarize.

To compare representations, we need two more terms. An encoder output is the sequence of vectors a trained network such as Whisper's encoder produces, one per 20 ms here, stored as bfloat16 (a 2-byte floating-point format). A neural audio codec is a network trained to compress audio into a short stream of integers and decode it back. Codec tokens are those integers (§37.3).

The surprise in that comparison is the point. A learned representation is not compression in bytes. It is compression in sequence length, 480,000 samples down to 1,500 positions for Whisper. Attention costs grow with the square of the length (§4.7), so that cut makes attention over the clip about $10^5$ times cheaper. It is also a change in what is kept, the content that predicts words and speakers rather than every detail of the waveform. Codec tokens are short and small at the same time, which is why speech generation models them the way language models treat text (§37.4). The cell reproduces every number above.

Where audio pipelines break

Most audio accuracy bugs are not in the model. They are in the few lines that load, convert, and mix the samples, and the list below is the one to check first.

The cell shows three of these bugs in numbers.

The 2nd and 4th harmonics of the clipped sine come out at $-240$ dB. That is the $10^{-12}$ floor inside level_db, so they are exactly zero. The 3rd and 5th are large, $-17.5$ and $-30.7$ dB. Clipping that treats positive and negative values alike makes only odd harmonics, and the box below says why.

The same tools in PyTorch

In a training pipeline these operations run on batches, often on the GPU. Here is a self-contained batched version of the two workhorses of this section, noise mixing at a different SNR per example and anti-aliased decimation. In practice you would call torchaudio.functional.resample, which uses the same windowed-sinc idea. Writing it once shows that resampling is a convolution followed by a stride, like the convolution layers of §0.8.

Read more

• Shannon (1949). Communication in the Presence of Noise. The sampling theorem stated and proved in a few paragraphs, by the founder of information theory, with the same Fourier-series argument used here.
• Smith. Mathematics of the Discrete Fourier Transform (DFT) with Audio Applications, a free online book whose appendix on the sampling theorem proves the continuous-time aliasing theorem and Whittaker-Shannon interpolation carefully, and its companion Digital Audio Resampling Home Page, on windowed-sinc filters and how high-quality resamplers are actually built.
• Lipshitz, Wannamaker, and Vanderkooy (1992). Quantization and Dither: A Theoretical Survey. The survey behind the dither theorem, with RPDF and TPDF dither compared exactly.
• Wannamaker, Lipshitz, Vanderkooy, and Wright (2000). A theory of nonsubtractive dither. The general result behind the dither box, that $k$ uniforms make the first $k$ moments of the error independent of the input, and why TPDF is the practical choice.
• Peterson and Barney (1952). Control Methods Used in a Study of the Vowels. The classic formant measurements of American English vowels quoted in the source-filter part.
• Jurafsky & Martin. Speech and Language Processing, 3rd edition draft. Chapter 15, "Phonetics and Speech Feature Extraction", covers waveforms, sampling and quantization for speech, pitch, and formants. For the hands-on side of loading audio for models, unit 1 of the Hugging Face audio course, Introduction to audio data, is a short companion.
• ITU-R. BS.1770: Algorithms to measure audio programme loudness and true-peak audio level. The definition of K-weighting, gating, and LUFS, with the $-3.01$ LUFS reference sine.
• EBU. R 128, Loudness normalisation and permitted maximum level of audio signals. The $-23$ LUFS broadcast target and how loudness normalization is used in practice.

Watch next

▶

▶

▶

▶