Chapter 37 · Section 37.2 · 100 min

Neural vocoders

A vocoder turns a spectrogram back into sound. The acoustic models of §37.1 stop one step short of audio. They output a log-mel spectrogram, a grid with one column of 80 numbers for every 11.6 ms of speech, where each number is the logarithm of how much energy the sound has in one band of frequencies. A loudspeaker needs a waveform instead, 22,050 numbers per second, each one the air pressure at one instant. The vocoder is the model that goes from the grid to the waveform.

This last stage decides much of how natural a voice sounds and much of what speaking costs. Tacotron 2 (Shen et al. 2018) trained one acoustic model to predict spectrograms and turned them into sound in two ways, first with a classical algorithm and then with a neural vocoder. Listeners rated each clip's naturalness from 1 to 5, and the average rating, the mean opinion score (MOS), rose from 3.944 to 4.510. The original recordings scored 4.582. Speed varies even more. On one GPU, the slowest neural vocoder in a standard benchmark takes 52 minutes to produce ten seconds of speech, and the one that sounds best takes 0.06 seconds.

This section explains both numbers. First we say exactly what a vocoder must invent, and why a network trained to predict the waveform with a squared error cannot invent it. Then we build the two designs that work. WaveNet writes the waveform one sample at a time, the way a language model writes text, which sounds excellent and is slow. HiFi-GAN writes every sample at once, and a set of critic networks that listen at several periods and time scales teaches it what real speech sounds like. We follow the sentence Dr. Smith paid $5. from §37.1 through the generator with every shape, count the arithmetic behind real-time speed, and find where the audible artifacts come from. The neural audio codecs of §37.3 reuse almost all of it.

What a vocoder has to invent

Start with how the vocoder's input was made (§35.2 and §35.3 build it in full). A waveform $x[n]$ is a list of samples taken $f_s$ times a second, each a number in $[-1, 1]$ . The analysis cuts it into overlapping windows of $N$ samples and starts a new window every $R$ samples, a step called the hop. A Fourier transform turns each window, called a frame, into $N/2 + 1$ complex numbers $X[t, k]$ , one for each frequency bin $k$ of frame $t$ .

A complex number has a size and an angle. The size, the magnitude $\lvert X[t, k]\rvert$ , says how strong that frequency is in the frame. The angle, the phase, says where that frequency's wave is in its cycle at the start of the frame. The analysis keeps only the magnitudes. It then pools the $N/2 + 1$ magnitudes of a frame into $n_{\text{mels}}$ overlapping triangular bands, narrow at low frequencies and wide at high ones. That spacing, the mel scale, matches how finely human hearing resolves pitch. Last, it takes a logarithm. With $N = 1024$ and $n_{\text{mels}} = 80$ , each frame goes from 513 complex numbers, which is 1,026 real ones, down to 80.

✏️ The sentence of §37.1, through the analysis

§37.1 followed Dr. Smith paid $5. to a log-mel spectrogram of 156 frames, 1.81 s of speech at 22,050 Hz. Its recipe is HiFi-GAN's for LJSpeech, with a window of 1,024 samples, a hop of 256, and 80 bands up to 8 kHz. The cell below analyzes a voiced stand-in of exactly that length, 39,936 samples with a falling pitch and about seven syllables. There are 156 frames. The complex STFT holds $156\times 513$ complex numbers, 160,056 real ones, and the log-mel spectrogram keeps $156\times 80 = 12{,}480$ of them. The lowest mel band pools 2 frequency bins and the highest pools 24. The vocoder reads those 12,480 numbers and must write $156\times 256 = 39{,}936$ samples, 3.2 samples for every number it is given. Later in the section we follow the same 156 frames through HiFi-GAN's generator.

Python · cell 1

📐 Vocoder

Let $\phi$ be a fixed analysis map from a waveform $\mathbf{x} \in [-1, 1]^{N_s}$ of $N_s$ samples to a log-mel spectrogram $\mathbf{X}_{\text{mel}} = \phi(\mathbf{x}) \in \R^{T_{\text{mel}}\times n_{\text{mels}}}$ with $T_{\text{mel}}$ frames and hop $R$ , so $N_s \approx T_{\text{mel}}R$ . A vocoder is a model that maps $\mathbf{X}_{\text{mel}}$ back to a waveform of $T_{\text{mel}}R$ samples. It is either a conditional distribution $p_\theta(\mathbf{x}\mid\mathbf{X}_{\text{mel}})$ that can be sampled (WaveNet, diffusion vocoders) or a deterministic map $G_\theta(\mathbf{X}_{\text{mel}})$ (GAN vocoders). Either way it is trained so that its outputs sound like real speech whose analysis is $\mathbf{X}_{\text{mel}}$ .

In plain words, a vocoder undoes an analysis that threw information away. Many waveforms have the same mel spectrogram, and the vocoder must pick one that a listener accepts.

The phase is the larger loss. The classical way to recover it is the Griffin-Lim algorithm, which §35.2 derived. It starts from any phase and repeats two steps. It builds the waveform whose STFT is closest to the current complex array, then keeps that waveform's phases and puts back the target magnitudes. No round makes the mismatch worse, but the algorithm stalls at a phase that sounds metallic. Tacotron 2's score of 3.944 was Griffin-Lim working from full linear-frequency spectrograms, 1,025 magnitudes per frame. From a mel spectrogram it would first have to guess those magnitudes from 80 numbers.

Why a regression cannot do it

A natural idea is to train a network to predict the waveform from the mel with a squared error. It fails because of a fact about averages. Take a 200 Hz tone, $\sin(2\pi\cdot 200\,t)$ , and the same tone shifted by half a cycle, $-\sin(2\pi\cdot 200\,t)$ . They have identical magnitudes in every STFT bin, so identical mel spectrograms, and they sound the same. If a training set held both equally often, the best squared-error prediction would be their average, which is zero. The network would answer this mel spectrogram with silence.

Speech is the same story with many harmonics. A sustained vowel is a sum of harmonics, sinusoids at whole multiples of a fundamental frequency $F_0$ . Changing the phases of the harmonics leaves every STFT magnitude, and so the mel spectrogram, essentially unchanged, and it barely changes how the vowel sounds. But it changes the waveform completely. The cell draws 200 vowels with the same harmonics and random phases. It checks that their magnitude spectrograms agree, that the waveforms themselves do not, and how loud their average is.

Python · cell 2

The 200 magnitude spectrograms agree to 0.1 percent of their peak. The waveforms' correlations range from about $-0.8$ to $+0.8$ around a mean of zero. Their average has 22 dB less power than any one of them, so a squared-error vocoder would answer this vowel with something close to silence. The same averaging, one level up, is why acoustic models trained with squared error predict oversmoothed spectrograms (§37.1), which Tacotron 2's authors point out.

So the waveform given the mel is a distribution with many separate peaks, and the vocoder must be a generative model, one that samples or commits to one plausible waveform rather than averaging them. Two answers work. One models the distribution explicitly, one sample at a time (WaveNet). The other trains a second network, a critic, to decide what counts as a real waveform (GANs). Both reappear inside every neural audio codec of §37.3. An interviewer who asks why a vocoder cannot be trained with an L2 loss on samples wants exactly this argument.

✏️ Ten seconds of speech, a factor of 50,000

Kong, Kim & Bae (2020) timed vocoders on one V100 GPU in 32-bit floats. A WaveNet whose output is a mixture of logistic distributions (a 16-bit output described later in this section) generated 0.07 kHz, that is, 70 samples per second. Ten seconds at 22.05 kHz is 220,500 samples, so it takes $220{,}500/70 = 3{,}150$ s, about 52 minutes. HiFi-GAN V1, the best-sounding model in that table, generated 3,701 kHz and takes $220{,}500/3{,}701{,}000 = 0.060$ s. The ratio is $3{,}701/0.07 \approx 53{,}000$ . (HiFi-GAN's smallest version, V3, is seven times faster still, at a lower quality.)

A vocoder that runs at $x$ times real time produces $x$ seconds of audio per second of computation. The WaveNet runs at 0.003 times real time and HiFi-GAN at 168 times. In that paper's listening test the WaveNet also sounded worse, with a MOS of 4.02 against 4.36. The rest of the section explains that gap.

WaveNet writes one sample at a time

The first answer treats a waveform the way a language model treats text. A language model writes a sentence one token at a time, each drawn from a probability distribution given the tokens before it. WaveNet (van den Oord et al. 2016) writes a waveform $\mathbf{x} = (x[1], \dots, x[N_s])$ one sample at a time in the same way. The chain rule of probability makes this exact,

p(\mathbf{x}\mid\mathbf{c}) = \prod_{n=1}^{N_s} p\big(x[n] \mid x[1], \dots, x[n-1],\ \mathbf{c}\big),

where $\mathbf{c}$ is the conditioning, the extra input the waveform depends on. For a vocoder it is the mel frames and possibly a speaker.

In plain words, the probability of a whole waveform is the probability of the first sample, times that of the second given the first, and so on. Training maximizes the log-likelihood with teacher forcing. Every sample is predicted from the true samples before it, so all positions are trained at once, in parallel. Generation cannot be parallel. It samples $x[1]$ , feeds it back, samples $x[2]$ , and so on, $N_s$ times. Two design questions remain. What is the distribution over one sample, and what network sees enough of the past to predict it?

One sample is one of 256 classes

A sample is a real number, so the obvious output is a continuous density such as a Gaussian. WaveNet's authors chose a softmax over discrete values instead, as in their earlier image models, which generate a picture one pixel at a time (van den Oord et al. 2016). A categorical distribution assumes nothing about the shape of the next sample's distribution, which can be skewed, heavy-tailed, or have several peaks, and which changes from one moment of speech to the next. But a 16-bit sample takes 65,536 values, and a 65,536-way softmax at every sample is expensive. So WaveNet first compresses each sample to 8 bits, 256 classes, with a quantizer borrowed from telephony.

The idea is to put more levels near zero, where speech spends most of its time. With 256 evenly spaced levels the step is $2/256 = 0.0078$ . A quiet sample of 0.01 is rounded to 0.0078, a 22 percent error, while a loud sample of 0.5 is represented exactly. Mu-law companding (compressing, then expanding) moves the levels so that both errors are under 2 percent. It bends each sample with a logarithmic curve, quantizes the result evenly, and undoes the curve when decoding.

📐 Mu-law companding

For $x \in [-1, 1]$ and a constant $\mu > 0$ (WaveNet and North American telephony use $\mu = 255$ ; in this section $\mu$ is never a mean), the mu-law compressor is

F_\mu(x) = \operatorname{sign}(x)\,\frac{\ln(1 + \mu\lvert x\rvert)}{\ln(1 + \mu)} \in [-1, 1],

and its exact inverse, the expander, is $F_\mu^{-1}(y) = \operatorname{sign}(y)\,\big((1+\mu)^{\lvert y\rvert} - 1\big)/\mu$ . An $n_{\text{bits}}$ -bit mu-law quantizer applies $F_\mu$ and splits $[-1, 1]$ into $2^{n_{\text{bits}}}$ equal cells of width $\Delta_y = 2/2^{n_{\text{bits}}}$ . It outputs the class $q = \min\big(\lfloor (F_\mu(x) + 1)/\Delta_y\rfloor,\ 2^{n_{\text{bits}}} - 1\big)$ and decodes a class to the expanded cell center, $\hat x = F_\mu^{-1}\big((q + \tfrac12)\Delta_y - 1\big)$ . WaveNet uses $n_{\text{bits}} = 8$ , so 256 classes.

The inverse follows in two lines. If $y = F_\mu(x)$ , then $\lvert y\rvert\ln(1+\mu) = \ln(1 + \mu\lvert x\rvert)$ , so $(1+\mu)^{\lvert y\rvert} = 1 + \mu\lvert x\rvert$ , and $x$ has the sign of $y$ . The minimum in the class formula matters only at $x = 1$ , which would otherwise land in a class 256 that does not exist.

✏️ Two samples through the quantizer, by hand

With $\mu = 255$ , $\ln 256 = 5.545$ . A quiet sample $x = 0.01$ gives $F_\mu(x) = \ln(3.55)/5.545 = 0.2285$ and class $\lfloor 1.2285\times 128\rfloor = 157$ . It decodes to $F_\mu^{-1}(157.5/128 - 1) = F_\mu^{-1}(0.2305) = 0.01015$ , an error of 1.5 percent. A loud sample $x = 0.5$ gives $F_\mu(x) = \ln(128.5)/5.545 = 0.8757$ , class 240, and decodes to $0.5090$ , an error of 1.8 percent. An 8-bit linear quantizer (step $0.0078$ ) decodes $0.01$ to $0.0078$ , an error of 22 percent, and represents $0.5$ exactly. Mu-law spends its levels near zero, where speech spends its time, and keeps the error a roughly fixed fraction of the sample.

That last sentence can be made exact. Recall from §35.1 that a uniform quantizer with step $\Delta$ adds an error that behaves like uniform noise of power $\Delta^2/12$ . The signal-to-quantization-noise ratio (SQNR) is the signal's power divided by that noise power, in decibels. For a full-scale sine it is $6.02\,n_{\text{bits}} + 1.76$ dB. Because the noise stays the same size, the SQNR falls one decibel for every decibel the signal gets quieter. Mu-law makes the noise shrink with the signal.

∴ Mu-law is a logarithmic quantizer

Assume the quantization error in the companded domain, $e_y = \hat y - F_\mu(x)$ , is uniform on $[-\Delta_y/2, \Delta_y/2]$ and independent of $x$ , that the step is small enough for a first-order Taylor expansion of $F_\mu^{-1}$ across one cell, and that $\lvert x\rvert \le 1$ . Then the step in the signal domain at amplitude $x$ is

\Delta_x(x) \approx \frac{\Delta_y}{F_\mu'(x)} = \Delta_y\,\frac{(1 + \mu\lvert x\rvert)\ln(1+\mu)}{\mu}.

It is $\Delta_y\ln(1+\mu)/\mu$ near zero, and for $\lvert x\rvert \gg 1/\mu$ it approaches a constant fraction of the amplitude, $\Delta_x/\lvert x\rvert \to \Delta_y\ln(1+\mu)$ . The SQNR is

\mathrm{SQNR} = \frac{12}{\big(\Delta_y\ln(1+\mu)\big)^2}\cdot\frac{\mathbb{E}[x^2]}{\mathbb{E}\big[(\lvert x\rvert + 1/\mu)^2\big]} .

Its first factor does not depend on the signal level. Its second factor tends to 1 when the signal is mostly far above $1/\mu$ . At 8 bits with $\mu = 255$ the first factor is $10\log_{10}\big(12/(0.0078125\times 5.545)^2\big) = 38.06$ dB. The proof is in the derivation below.

In plain words, mu-law is uniform quantization of the logarithm of the amplitude. Away from zero its cells have a fixed relative width, 4.3 percent of the amplitude at 8 bits, so the error is at most about 2.2 percent of the sample. The price is paid at the top. There mu-law's cells are about 5.6 times wider than linear ones ( $\Delta_x(1)/\Delta_y = 256\ln 256/255 = 5.57$ ). That is why a full-scale signal gets 38 dB from mu-law against 50 dB from linear coding.

∂ Proof that mu-law's SQNR does not depend on the level

The decoded sample is $\hat x = F_\mu^{-1}(F_\mu(x) + e_y)$ . To first order, $\hat x - x \approx (F_\mu^{-1})'(F_\mu(x))\,e_y = e_y/F_\mu'(x)$ , by the inverse function rule. Differentiating the definition for $x \ne 0$ gives $F_\mu'(x) = \mu/\big((1 + \mu\lvert x\rvert)\ln(1+\mu)\big)$ . A cell of width $\Delta_y$ therefore maps to a cell of width $\Delta_y/F_\mu'(x)$ , which proves the step formula.

Given $x$ , the error power is $\mathbb{E}[e_y^2]/F_\mu'(x)^2 = (\Delta_y^2/12)\,(1 + \mu\lvert x\rvert)^2\ln^2(1+\mu)/\mu^2 = (\Delta_y^2/12)\ln^2(1+\mu)\,(\lvert x\rvert + 1/\mu)^2$ . Averaging over $x$ and dividing $\mathbb{E}[x^2]$ by the result gives the SQNR formula. ∎

For a sine of amplitude $A$ , $\mathbb{E}[x^2] = A^2/2$ and $\mathbb{E}\lvert x\rvert = 2A/\pi$ , so the second factor is $(A^2/2)/(A^2/2 + 4A/(\pi\mu) + 1/\mu^2)$ . It is $-0.04$ dB at $A = 1$ , $-0.4$ dB at $A = 0.1$ , and $-3.6$ dB at $A = 0.01$ .

The cell checks the theorem with a simulation. It quantizes sines with mu-law and with 8-bit linear coding, at levels from full scale down to $-50$ dB. Each level is the sine's peak in dBFS, decibels relative to full scale, $20\log_{10}A$ .

Python · cell 3

Mu-law holds between 34 and 39 dB from full scale down to $-40$ dBFS, within a decibel of the theory. Eight-bit linear coding falls from 49 dB to 11 dB, and at $-50$ dBFS it rounds the whole sine to zero (0 dB, the error is the signal). The two cross between $-10$ and $-20$ dBFS. Speech has a wide dynamic range. Its quiet syllables, consonants, and decays sit tens of decibels below its peaks, exactly where mu-law wins by 8 to 30 dB.

The widget draws one stretch of a sine through both 8-bit coders, with each coder's output levels as ticks at the edges, beside the SQNR curves of the cell. Lower the peak level and count the linear levels left inside the sine.

Eight bits still limit quality, since 38 dB of SQNR can be heard as faint hiss in quiet passages. The successors kept a discrete output at 16 bits without paying for a 65,536-way softmax.

Warm-up

Mu-law in and out

Implement the mu-law quantizer of the definition box with $\mu + 1$ classes. The compressor is $F_\mu(x) = \operatorname{sign}(x)\ln(1+\mu\lvert x\rvert)/\ln(1+\mu)$ , which maps $[-1, 1]$ onto itself. Split $[-1, 1]$ into $\mu + 1$ equal cells, numbered 0 to $\mu$ from the left. The class of $x$ is the cell that contains $F_\mu(x)$ , and $x = 1$ belongs to the top cell, $\mu$ .

mulaw_encode(x, mu=255) maps an array of samples in $[-1, 1]$ to an integer array of classes in $\{0, \dots, \mu\}$ .
mulaw_decode(q, mu=255) maps classes back to samples. It takes the center of each class's cell and applies the exact inverse of $F_\mu$ .

The logarithm's base cancels in the ratio $\ln(1+\mu\lvert x\rvert)/\ln(1+\mu)$ , so np.log10 works as well as np.log.

import numpy as np

def mulaw_encode(x, mu=255):
    raise NotImplementedError

def mulaw_decode(q, mu=255):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Dilated causal convolutions see far

The network must read the past samples and output a distribution for the next one. During training it must do so for all $N_s$ positions at once. A causal convolution does both. It is a convolution whose output at time $n$ reads only inputs at time $n$ and earlier, so it never sees the future and still runs over the whole sequence in one pass. Giving it a dilation $d$ spaces its taps $d$ steps apart, so the same few weights reach further back.

Here is the payoff on a tiny example. Three causal layers with kernel 2 and no dilation let one output see 4 input samples. With dilations 1, 2, and 4 the same three layers see 8, and each further layer with a doubled dilation doubles the view again. The number of input steps that one output can depend on is the receptive field, which §0.8 defined for any stack of convolutions. Here is the version we need.

📐 Dilated causal convolution

For an input $\mathbf{H} \in \R^{C\times N_s}$ (channels by time), weights $\mathbf{W} \in \R^{C'\times C\times k}$ with kernel size $k$ , and dilation $d$ , the dilated causal convolution outputs $\mathbf{Y} = \mathbf{W}\circledast_d\mathbf{H} \in \R^{C'\times N_s}$ with

Y_{c', n} = \sum_{c=1}^{C}\sum_{u=0}^{k-1} W_{c', c, u}\; H_{c,\ n - d(k-1-u)} ,

where $H_{c, m} = 0$ for $m \lt 1$ . Equivalently, pad $d(k-1)$ zeros on the left only and apply an ordinary stride-1 convolution. The output has the same length $N_s$ as the input, and output $n$ reads inputs $n, n - d, \dots, n - d(k-1)$ , never anything after $n$ .

In plain words, each layer adds $(k-1)$ times its dilation to how far back the output can see. Doubling the dilations makes the view grow exponentially with depth, while the cost grows only linearly.

∂ Proof of the receptive field

By induction on the number of layers. Before any layer, output $n$ of the identity depends on input $n$ only, a window of $1 = 1 + (k-1)\cdot 0$ . Suppose after $l - 1$ layers every unit at time $m$ depends on inputs in $[m - D_{l-1}, m]$ with $D_{l-1} = (k-1)\sum_{j \lt l} d_j$ . Layer $l$ combines the units at times $n, n - d_l, \dots, n - (k-1)d_l$ , whose windows together cover $[n - (k-1)d_l - D_{l-1}, n]$ , so $D_l = D_{l-1} + (k-1)d_l$ . A residual connection adds the unit at time $n$ of the layer below, whose window is already inside, and pointwise nonlinearities change no windows. This is the recursion $r_l = r_{l-1} + (k_l - 1)d_l\,j_{l-1}$ of §0.8 with every jump $j = 1$ , since nothing is strided. The geometric sum $\sum_{l=0}^{L-1}2^l = 2^L - 1$ gives the doubling case. ∎

✏️ Receptive fields, by hand and at real scale

Kernel 2 with dilations 1, 2, 4 sees $1 + 1 + 2 + 4 = 8$ samples, where three undilated layers see 4. The WaveNet paper's example stack has kernel 2 and dilations $1, 2, 4, \dots, 512$ repeated three times. One cycle sees $1 + 1023 = 1{,}024$ samples and three cycles see $1 + 3\times 1023 = 3{,}070$ , which is 192 ms at 16 kHz. Without dilation the same view would take 3,069 layers. (The paper quotes receptive fields of 240 ms for its text-to-speech models and about 300 ms for its multi-speaker model, which use other stacks.)

Tacotron 2's WaveNet vocoder has 30 layers in 3 cycles of dilations 1 to 512 at 24 kHz. Its Table 4 lists a receptive field of 6,139 samples (255.8 ms) for that stack. That equals $1 + 2\times 3\times 1023$ , the theorem with kernel size 3.

The same table holds a surprise. A 12-layer WaveNet with two cycles of dilations 1 to 32 sees only 253 samples (10.5 ms) and still scores 4.481, against 4.526 for the 30-layer baseline. Thirty undilated layers (61 samples, 2.5 ms) fall to 3.930. A vocoder conditioned on mel frames does not need a long memory, because the frames already carry the slow structure, such as the pitch contour, the phonemes, and the loudness. It needs context at the scale of a few pitch periods, enough to make each period consistent with the last. This is why the fast vocoders later in the section get away with modest receptive fields. It is also the answer when an interviewer asks how much context a vocoder needs.

The cell checks causality with an impulse, a single 1 in a row of zeros. Then it measures receptive fields by pushing an impulse through random positive weights, so that nothing cancels, and compares them with the theorem.

Python · cell 4

An impulse at $n = 10$ reaches outputs 10 and 14 (the current tap, and the tap 4 back as seen from 14) and nothing earlier. The measured fields match the formula, including WaveNet's 3,070.

Not every schedule fills its window. Kernel 2 with dilations 1, 3, 9 spans 14 samples, but a path from input to output takes one tap per layer, so it can only step back by sums of 0 or 1, 0 or 3, and 0 or 9. It reaches 8 of the 14 offsets, and the other inputs can never influence the output. These gaps are the one-dimensional version of the gridding that dilated image networks suffer. The common schedules avoid them. Doubling dilations with kernel 2 fill every offset, and so does MelGAN's schedule 1, 3, 9 with kernel 3 (Kumar et al. 2019), which sees all 27.

The widget builds a stack one layer at a time. Each new layer reads $k$ copies of the tree of taps below it, the copies shifted back by multiples of its dilation. Press Stack layers with dilations 1, 2, 4 and the copies land end to end, so the view doubles per layer. Switch to 1, 3, 9 with kernel 2 and they overshoot, leaving pink inputs that no path reaches. The box after it gives the exact rule.

🔬 When the window has holes

The offsets $n - m$ of the inputs $m$ that can influence output $n$ are the sums $\sum_l u_ld_l$ with each $u_l \in \{0, \dots, k-1\}$ . This set does not depend on the order of the layers, so sort the dilations in increasing order. Then the sums fill the whole window, with no holes, if and only if $d_l \le 1 + (k-1)\sum_{j \lt l}d_j$ for every $l$ (for $l = 1$ this says $d_1 = 1$ ). Doubling dilations with $k = 2$ meet it with equality at every layer, since $2^l = 1 + \sum_{j\lt l}2^j$ , and so do the powers of 3 with $k = 3$ . Dilations 1, 3, 9 with $k = 2$ fail at the second layer, since $3 \gt 1 + 1$ , and reach only the offsets 0, 1, 3, 4, 9, 10, 12, 13.

Proof. Follow one path of influence from input $m$ to output $n$ . At each layer it passes through one tap, moving back by $u_ld_l$ for some $u_l \in \{0, \dots, k-1\}$ . A residual connection moves back by 0, which is the tap $u_l = 0$ . So the reachable offsets are the sums $\sum_lu_ld_l$ . After the first $l - 1$ sorted layers, suppose the reachable offsets are all of $\{0, \dots, D_{l-1}\}$ . Layer $l$ adds the shifts $0, d_l, \dots, (k-1)d_l$ . Consecutive shifted copies of the interval overlap or touch exactly when $d_l \le D_{l-1} + 1$ , which keeps the set an interval up to $D_l = D_{l-1} + (k-1)d_l$ . If $d_l \gt D_{l-1} + 1$ , the offset $D_{l-1} + 1$ is not reachable, since the shifts are either 0 (too small) or at least $d_l$ (too large), and later layers with larger dilations cannot fill it either. ∎

The gated residual block

WaveNet's layers are not bare convolutions. Each one wraps a dilated convolution in a small block with a gate and two paths out, borrowed from the gated PixelCNN (van den Oord et al. 2016). A gate multiplies a feature by a number between 0 and 1 that the network computes, so the network decides at every time step and channel how much of the feature to pass on.

📐 WaveNet residual block

Layer $l$ reads $\mathbf{H}^{(l-1)} \in \R^{C\times N_s}$ (residual channels by time; $\mathbf{H}^{(0)}$ is the embedded input) and a conditioning sequence $\mathbf{Y}_c \in \R^{C_c\times N_s}$ at the sample rate. It computes

\mathbf{Z}^{(l)} = \tanh\!\big(\mathbf{W}_f^{(l)}\circledast_{d_l}\mathbf{H}^{(l-1)} + \mathbf{V}_f^{(l)}\mathbf{Y}_c\big)\odot\sigma\!\big(\mathbf{W}_g^{(l)}\circledast_{d_l}\mathbf{H}^{(l-1)} + \mathbf{V}_g^{(l)}\mathbf{Y}_c\big) \in \R^{C\times N_s},

\mathbf{H}^{(l)} = \mathbf{H}^{(l-1)} + \mathbf{W}_r^{(l)}\mathbf{Z}^{(l)}, \qquad \mathbf{S}^{(l)} = \mathbf{W}_s^{(l)}\mathbf{Z}^{(l)} \in \R^{C_s\times N_s},

with filter and gate weights $\mathbf{W}_f^{(l)}, \mathbf{W}_g^{(l)} \in \R^{C\times C\times k}$ , conditioning projections $\mathbf{V}_f^{(l)}, \mathbf{V}_g^{(l)} \in \R^{C\times C_c}$ , and $1\times1$ convolutions $\mathbf{W}_r^{(l)} \in \R^{C\times C}$ (residual) and $\mathbf{W}_s^{(l)} \in \R^{C_s\times C}$ (skip), each applied at every time step. Here $\odot$ is the elementwise product and $\sigma$ the sigmoid. The network sums the skip outputs and applies $\operatorname{ReLU}$ , a $1\times 1$ convolution, $\operatorname{ReLU}$ , and a $1\times1$ convolution to 256 logits per time step. Column $n$ of the softmax is $p(x[n]\mid x[1..n-1], \mathbf{c})$ when the input is the waveform shifted right by one step.

In plain words, the tanh half proposes a feature and the sigmoid half decides how much of it to let through. The residual path carries the block's input forward unchanged plus a correction, which keeps 30 or more layers trainable (the argument of §5.1). The skip path lets every layer, short-range and long-range, speak directly to the output. The input is shifted right by one step so that output $n$ never sees sample $n$ , which makes the network a next-sample predictor rather than a copy machine.

Conditioning comes in two kinds. Global conditioning is one vector for the whole utterance, such as a speaker embedding $\mathbf{e}_{\text{spk}}$ , added through $\mathbf{V}\mathbf{e}_{\text{spk}}$ at every time step. Local conditioning is a slower sequence, here the mel frames at $f_s/R$ frames per second, which must be brought up to the sample rate first. WaveNet learned this upsampling with a stack of transposed convolutions and found that repeating each frame $R$ times worked slightly worse. Tacotron 2's vocoder upsamples its 12.5 ms frames with two upsampling layers in its conditioning stack.

Interview

A WaveNet stack and its receptive field

The setup provides causal_conv1d(h, w, dilation), the dilated causal convolution of the definition box for h of shape (C, T) and w of shape (C_out, C, k). Each layer is a dictionary with the dilation under "d", the filter and gate weights "Wf" and "Wg" (shape (C, C, k)), the residual weights "Wr" (shape (C, C)), and the skip weights "Ws" (shape (S, C)). There is no conditioning.

Implement wavenet_stack(h, layers). It runs the gated residual blocks in order on h of shape (C, T) and returns the summed skip outputs, shape (S, T).
Implement measured_receptive_field(layers, C, T=64), which finds the receptive field by experiment. Perturb the input one time step at a time (all channels, by a random vector) and count how many input time steps change the skip output at the last time step T - 1. Use only wavenet_stack, not the formula, because the count is smaller than $1 + (k-1)\sum d$ when the dilations leave holes.

import numpy as np

def wavenet_stack(h, layers):
    raise NotImplementedError

def measured_receptive_field(layers, C, T=64):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Generating is sequential, and caching makes each step cheap

Training is parallel, and generation is not. To sample $x[n]$ the network needs $x[n-1]$ , which it has only just produced. So a WaveNet vocoder runs its whole network once per sample, one run after another, 24,000 times for one second of audio at 24 kHz.

A naive implementation also makes each run expensive, because it recomputes every layer over the whole receptive field. Most of that work was already done at the previous step. With kernel 2, layer $l$ at time $n$ reads its input at times $n$ and $n - d_l$ . So it is enough to keep, for each layer, a queue of its last $d_l$ inputs. Each step pushes one new column through the stack and reads one old column per layer from the queues. The work per sample is then proportional to the number of layers $L$ , instead of growing with the receptive field. Paine et al. (2016) call this Fast WaveNet.

The total cache is $C\sum_l d_l(k-1)$ numbers. It plays the role that the KV cache of §6.5 plays for a Transformer, the stored results of past steps that let each new step skip recomputing them. Unlike the KV cache, it never grows with the length of the utterance, because nothing older than the receptive field can matter.

Interview

Fast WaveNet generation with per-layer queues

The setup provides a complete greedy WaveNet over 16 classes. wavenet_skip(H, layers) runs the gated residual stack of the definition box, without conditioning, on an embedded sequence H of shape (C, T). Each layer is a dictionary with the dilation "d", filter and gate weights "Wf" and "Wg" of shape (C, C, k), and $1\times1$ weights "Wr" of shape (C, C) and "Ws" of shape (S, C). Layer by layer it computes $\mathbf{z} = \tanh(\mathbf{W}_f\circledast_d\mathbf{H})\odot\sigma(\mathbf{W}_g\circledast_d\mathbf{H})$ with the dilated causal convolution $\circledast_d$ , adds Ws @ z to the skip sum, and replaces H by H + Wr @ z. It returns the skip sum, shape (S, T). embed is a (16, C) table of class embeddings and Wout is a (16, S) readout. naive_generate(first, n, layers, embed, Wout) produces a sequence of n + 1 classes. At every step it recomputes the whole stack over the whole history and takes the argmax of Wout @ relu(skip[:, -1]). Implement the fast version in two functions.

wavenet_step(h_t, layers, queues) takes one embedded column h_t of shape (C,) and a list queues. Entry l of the list has shape (C, d_l * (k - 1)) and holds the last inputs of layer l, oldest first. The function returns (skip_t, new_queues), where skip_t has shape (S,) and equals the column of the full stack's skip output at this time step.
fast_generate(first, n, layers, embed, Wout) starts from all-zero queues. It must return exactly the sequence that naive_generate returns while storing only the queues.

import numpy as np

def wavenet_step(h_t, layers, queues):
    raise NotImplementedError

def fast_generate(first, n, layers, embed, Wout):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Four ways around one sample at a time

Caching makes each step cheap, but it leaves 24,000 steps per second of audio in a strict sequence. On a GPU such a sequence is limited by memory traffic, not by arithmetic. Each step reads every weight once to produce one number, the regime §20.1 calls memory-bound decoding. Kalchbrenner et al. (2018) put numbers on it. A WaveNet with a 16-bit mixture output needs about 60 matrix-vector products in sequence per sample (30 residual blocks of two layers each). Even a small recurrent network of 3 million parameters, if it reloads its weights for each of 24,000 samples, moves $3\times10^6\times 24\times10^3\times 4$ bytes, 288 GB per second of audio. That is more than a third of a P100 GPU's memory bandwidth before any computation.

Four families of answers followed. One makes each step tiny, with a single small recurrent layer instead of a deep stack (WaveRNN, LPCNet). One distills a trained WaveNet, training a fast student network to imitate it and generate all samples in one parallel pass (Parallel WaveNet). One uses a normalizing flow, an invertible network from noise to waveforms trained by exact likelihood (WaveGlow). And one uses diffusion, which turns noise into a waveform in a few parallel denoising passes (DiffWave, WaveGrad). The box gives each family's mechanism and numbers, and the table after it compares them.

🔬 The four families in detail

Make each step tiny. WaveRNN replaces the stack by a single recurrent layer with the coarse and fine softmaxes, about 5 matrix-vector products in sequence per sample. With custom GPU kernels it generates 96,000 16-bit samples per second on a P100, 4 times real time at 24 kHz, where the authors' best WaveNet kernel managed about 0.3 times. Pruning most of its weights to zero then makes it fit a phone's CPU. Subscale generation folds the waveform into 16 interleaved subsequences and produces 16 samples per step. LPCNet (Valin & Skoglund 2019) leans on classical signal processing. A linear predictor estimates each sample from the previous ones, and a small recurrent network predicts only what the predictor misses. It needs under 3 GFLOP (billion floating-point operations) per second of speech, light enough for phones and embedded devices.

Distill into a parallel student. Parallel WaveNet (van den Oord et al. 2018) trains an inverse autoregressive flow, a network that turns independent random noise into a waveform in one parallel pass, to match a trained WaveNet teacher. It minimizes $D_{\mathrm{KL}}(P_S\Vert P_T) = H(P_S, P_T) - H(P_S)$ on the student's own samples, plus three auxiliary losses. One matches the power in each frequency band. One matches statistics of the features of a phone classifier, which penalizes bad pronunciation. A contrastive one penalizes waveforms the teacher finds likely whatever the conditioning. The entropy term $-H(P_S)$ stops the student from collapsing onto the teacher's most likely waveform (§0.5 defines both terms). Student and teacher both scored 4.41. The cached teacher generated 172 samples per second on a P100 and the student over 500,000, and the student shipped in Google Assistant.

Use a flow trained by likelihood. A normalizing flow is trained by maximizing the exact likelihood of the data, which an invertible map makes computable (§33.6). WaveGlow (Prenger, Valle & Catanzaro 2019) is one, trained with no teacher, and it generates over 500 kHz on a V100. It is large (87.7 M parameters in HiFi-GAN's table) and slow on a CPU (0.21 times real time).

Use diffusion. A diffusion model learns to remove noise a little at a time, starting from pure noise (§33.2). DiffWave (Kong et al. 2021) and WaveGrad (Chen et al. 2021) apply it to waveforms conditioned on mel frames. DiffWave matched a strong WaveNet (MOS 4.44 against 4.43), and WaveGrad reached high fidelity with as few as six refinement steps, the kind of few-step sampling that §33.4 develops.

A fifth answer, the GAN, won in practice. One forward pass of a convolutional generator, parallel over all samples, turned out to be enough to be both small and good. The rest of the section builds it.

Family	Examples	Sequential steps per second of audio	Quality, as published	Training	On a CPU
Sample-level autoregression	WaveNet, WaveRNN, LPCNet	$f_s$ (WaveRNN subscale $f_s/16$ )	WaveNet 4.41 at 24 kHz, 16 bits	likelihood, stable, simple	WaveNet no; WaveRNN and LPCNet yes, with sparsity or linear prediction
Distilled flow	Parallel WaveNet	one pass per flow	4.41, equal to its teacher	needs a trained teacher and extra losses	not its target
Likelihood flow	WaveGlow	one pass	3.81 in HiFi-GAN's test	exact likelihood, large model	0.21 times real time
Diffusion	DiffWave, WaveGrad	one pass per denoising step (6 or more)	DiffWave 4.44 against WaveNet 4.43	regression loss, stable	slow with many steps
Time-domain GAN	MelGAN, Parallel WaveGAN, HiFi-GAN, BigVGAN	one pass	HiFi-GAN V1 4.36 against 4.45 for recordings	adversarial plus reconstruction losses	HiFi-GAN V3 13.4 times real time on a laptop
GAN with a Fourier head	iSTFTNet, Vocos	one pass at the frame rate, then an inverse STFT	Vocos not significantly different from BigVGAN	adversarial plus reconstruction losses	Vocos 170 times real time

The MOS values come from different listening tests, so compare them only within one row's source. The column that matters is the third. A GAN vocoder's work is not a chain of $f_s$ dependent steps, so a GPU can apply it to all samples at once, and its speed is limited only by the arithmetic and memory of one pass.

GAN vocoders

A generative adversarial network (GAN) trains two networks against each other. A discriminator, which this section also calls a critic, looks at a waveform and outputs a score for how real it looks. A generator turns mel frames into a waveform and is trained to raise the critic's scores of its outputs. §32.6 built this game with a critic trained by cross-entropy. For a fixed generator, the best such critic outputs $p_{\text{data}}/(p_{\text{data}} + p_G)$ , where $p_{\text{data}}$ is the density of real examples and $p_G$ that of generated ones. Against that critic the generator minimizes the Jensen-Shannon divergence, a symmetric measure of how different two distributions are that is zero only when they are equal. That section also showed that the original generator loss stops giving gradient once the critic is confident. It replaced it by the non-saturating loss and by the hinge loss, which stops pushing on an example once its score clears a margin.

The least-squares game

HiFi-GAN and Parallel WaveGAN use a third loss, least squares. (MelGAN, Vocos, and the codecs of §37.3 use the hinge loss.) The critic outputs an unbounded number and is trained as a regression, toward 1 on real audio and toward 0 on generated audio.

📐 Least-squares GAN

Let the discriminator $\operatorname{Disc}(\mathbf{x}) \in \R$ output an unbounded score (no sigmoid). The least-squares GAN losses (Mao et al. 2017), with the targets used by HiFi-GAN (0 for generated examples, 1 for real ones, and 1 for the generator), are

\mathcal{L}_D = \mathbb{E}_{\mathbf{x}\sim p_{\text{data}}}\big[(\operatorname{Disc}(\mathbf{x}) - 1)^2\big] + \mathbb{E}_{\mathbf{x}\sim p_G}\big[\operatorname{Disc}(\mathbf{x})^2\big], \qquad \mathcal{L}_{\text{adv}}(G) = \mathbb{E}_{\mathbf{x}\sim p_G}\big[(\operatorname{Disc}(\mathbf{x}) - 1)^2\big],

where $p_{\text{data}}$ is the distribution of real waveforms and $p_G$ the distribution of generated ones $G(\mathbf{X}_{\text{mel}})$ . The discriminator regresses real examples to 1 and generated ones to 0, and the generator pushes the scores of its outputs toward 1.

Is this a sensible game? If the critic were perfect, would the generator's best move be to produce real speech? The answer is yes, and the proof is short.

∴ The least-squares game has the right equilibrium

For densities $p_{\text{data}}$ and $p_G$ , the discriminator minimizing $\mathcal{L}_D$ over all functions is $\operatorname{Disc}^\star = p_{\text{data}}/(p_{\text{data}} + p_G)$ , the same function as in the cross-entropy game. Against it, if $p_{\text{data}} \gt 0$ everywhere, the generator's loss is

\mathcal{L}_{\text{adv}}(G) = \int p_{\text{data}}(\mathbf{x})\,\varphi\!\Big(\frac{p_G(\mathbf{x})}{p_{\text{data}}(\mathbf{x})}\Big)\,d\mathbf{x}, \qquad \varphi(r) = \frac{r^3}{(1+r)^2},

which is at least $\varphi(1) = \tfrac14$ , with equality exactly when $p_G = p_{\text{data}}$ . The proof is in the derivations below.

In plain words, against a perfect critic, nothing scores better than producing exactly the distribution of real speech. So the least-squares game is a legitimate way to fit a distribution, not only a heuristic.

∂ Proof of the optimal critic and the bound of one quarter

$\mathcal{L}_D = \int \big[p_{\text{data}}(D - 1)^2 + p_G D^2\big]d\mathbf{x}$ can be minimized pointwise. At each $\mathbf{x}$ the integrand is a parabola in $D$ with derivative $2p_{\text{data}}(D - 1) + 2p_GD$ and second derivative $2(p_{\text{data}} + p_G) \gt 0$ , so its minimum is at $D = p_{\text{data}}/(p_{\text{data}} + p_G)$ . Then $D^\star - 1 = -p_G/(p_{\text{data}} + p_G)$ , and

\mathbb{E}_{p_G}\big[(D^\star - 1)^2\big] = \int \frac{p_G^3}{(p_{\text{data}} + p_G)^2}\,d\mathbf{x} = \int p_{\text{data}}\,\frac{(p_G/p_{\text{data}})^3}{(1 + p_G/p_{\text{data}})^2}\,d\mathbf{x},

which is the formula with $r = p_G/p_{\text{data}}$ . Differentiating twice, $\varphi'(r) = r^2(r+3)/(1+r)^3$ and $\varphi''(r) = 6r/(1+r)^4 \gt 0$ for $r \gt 0$ , so $\varphi$ is strictly convex. Because $p_{\text{data}}$ is a probability density, Jensen's inequality gives $\int p_{\text{data}}\,\varphi(p_G/p_{\text{data}}) \ge \varphi\big(\int p_{\text{data}}\cdot p_G/p_{\text{data}}\big) = \varphi(1) = \tfrac14$ . Equality holds only if $p_G/p_{\text{data}}$ is constant, and a constant ratio of two densities is 1. ∎

Mao et al. derive their theory for other targets, $-1$ for generated examples, 1 for real ones, and 0 for the generator, which make the generator's objective a Pearson $\chi^2$ divergence. In their experiments they train with 0, 1, 1, the targets HiFi-GAN uses, which give the convex divergence above, with the same fixed point.

∂ When the generator puts mass where the data has none

Drop the assumption $p_{\text{data}} \gt 0$ . Let $m$ be the mass $p_G$ puts where $p_{\text{data}} = 0$ . There $D^\star = 0$ , so each unit of that mass costs the generator $(0 - 1)^2 = 1$ , contributing $m$ . On the support of $p_{\text{data}}$ , Jensen as above gives at least $\varphi(1 - m) = (1-m)^3/(2-m)^2$ , since $\int p_G = 1 - m$ there. So the loss is at least $g(m) = m + (1-m)^3/(2-m)^2$ . Its derivative is $g'(m) = 1 - (1-m)^2(4-m)/(2-m)^3$ . Writing $u = 1 - m \in [0, 1]$ , $(2-m)^3 - (1-m)^2(4-m) = (1+u)^3 - u^2(3+u) = 1 + 3u \gt 0$ , so $g' \gt 0$ . The bound is smallest at $m = 0$ , where it is $\tfrac14$ . Mass outside the data never helps, and the unique minimizer is still $p_G = p_{\text{data}}$ .

What makes least squares useful in practice is its gradient. Take one generated waveform with score $s = \operatorname{Disc}(G(\mathbf{X}_{\text{mel}}))$ . The generator's loss $(s - 1)^2$ has derivative $2(s - 1)$ , which is $-2$ for a sample the critic is sure is fake ( $s = 0$ ). It shrinks only as the score approaches the real target, so a confident critic still teaches the generator.

The cell checks the optimal discriminator on a grid of candidate outputs. Then it checks the bound $\tfrac14$ for a few one-dimensional Gaussian generators against a standard normal data distribution.

Python · cell 5

The grid minimizer agrees with $p_{\text{data}}/(p_{\text{data}} + p_G)$ to within the grid step. The generator's loss is exactly 0.25 when it matches the data, larger for every mismatch in mean or width, and close to 1 when the two distributions barely overlap.

Critics that listen at several periods and scales

The theorem is about ideal critics. Real critics are networks, and what they can see decides what the generator learns. The HiFi-GAN paper (Kong, Kim & Bae 2020) started from an observation about speech. Voiced sounds are close to periodic, built from sinusoids with many different periods. A vocoder's typical failures, such as a buzz, a smeared pitch, or a phase jump from one pitch period to the next, are failures of that periodic structure. So the authors built a critic that looks at the waveform through several periods at once.

The trick is a reshape. Write the samples 0 to 8 into rows of 3, so the rows are (0, 1, 2), (3, 4, 5), and (6, 7, 8). Column 0 now holds samples 0, 3, 6, every third sample. A 2-D convolution that slides only down the columns judges each such strand as a signal of its own.

Four terms appear in the definitions below. A leaky ReLU is a ReLU that keeps a small slope (0.1 here) for negative inputs instead of zeroing them. A grouped convolution splits the channels into groups and lets each output channel read only its own group, which cuts the cost. Weight normalization writes each weight vector as a learned length times a unit direction, which makes training faster and steadier. Spectral normalization divides each weight matrix by its largest singular value, so the layer cannot stretch its input by more than a factor of 1. That keeps a critic from becoming too sharp to learn from.

📐 Multi-period discriminator

For a period $p$ (in this subsection $p$ is always a period, never a probability), reflect-pad a waveform $\mathbf{x} \in \R^{N_s}$ on the right to length $p\lceil N_s/p\rceil$ and reshape it row by row into $\mathbf{X}^{(p)} \in \R^{\lceil N_s/p\rceil\times p}$ , so that $X^{(p)}_{m, j} = x[mp + j]$ (counting from 0). Column $j$ is the subsequence $x[j], x[j+p], x[j+2p], \dots$ . A period sub-discriminator $\operatorname{Disc}_p$ applies 2-D convolutions whose kernels have width 1 across the period axis, so every column is processed separately with shared weights, and it outputs a grid of scores. HiFi-GAN uses five $5\times1$ convolutions with 32, 128, 512, 1024, and 1024 channels, the first four with stride 3 along the column, then a $3\times1$ convolution to one channel, with leaky ReLU and weight normalization. The multi-period discriminator (MPD) is the set of sub-discriminators with $p \in \{2, 3, 5, 7, 11\}$ .

In plain words, each sub-discriminator deals the waveform out into $p$ interleaved strands, every $p$ -th sample, and judges each strand as a signal at rate $f_s/p$ . Every input sample lands in exactly one cell of the grid, so the critic's gradient reaches every sample of the generated waveform and nothing is averaged away.

The widget deals a short waveform into $p$ columns. For a pulse train that repeats every $P$ samples, every pulse lands in the same column exactly when $p$ divides $P$ , and each column then repeats every $P/p$ rows. A real vowel repeats only every hundred samples or more, longer than any of the periods, so each column is the same wave read at one sample in $p$ .

Why prime periods? Compare periods 2 and 4. Column 0 of period 4 holds samples 0, 4, 8, ..., all of which are already in column 0 of period 2 (0, 2, 4, 6, 8, ...). So a period-4 critic looks only at strands that the period-2 critic already sees in full. Now compare periods 2 and 3. Column 0 of period 3 holds 0, 3, 6, 9, 12, ..., and only 0, 6, 12, every other one, are in column 0 of period 2. In general, if $p \lt p'$ share no factor, the Chinese remainder theorem says that a remainder on division by $p$ and a remainder on division by $p'$ together fix the remainder on division by $pp'$ . So two such columns share one sample in every $pp'$ , only a fraction $1/p$ of the period- $p'$ column.

Periods 2, 3, 5, 7, 11 share no factors in pairs, which is what the paper means by choosing them "to avoid overlaps as much as possible". Its ablation agrees, with a MOS of 3.90 for periods 2, 4, 8, 16, 32 against 4.10 for the primes.

Both are window-based critics. Each output score sees one receptive field of the waveform, and the loss averages over all of them. This is the audio version of the PatchGAN of §32.6, a critic that scores many local patches of an image instead of the whole image.

The two critics complement each other. The MSD looks at consecutive samples, but its pooling is a low-pass filter, so its coarser scales barely hear high frequencies. The MPD keeps every sample but judges strands of non-consecutive ones. The cell shows the reshape, counts the overlaps between periods, and measures what HiFi-GAN's pooling (kernel 4, stride 2, padding 2) keeps of a pure tone.

Python · cell 6

A 2 kHz tone keeps 65 percent of its power after one pooling and 8 percent after two, and a 5 kHz tone keeps about 1 percent after one. A 9 kHz tone keeps 6 percent after one pooling, and what is left is folded down to 2,025 Hz, because the pooled signal has half the sample rate. So high-frequency artifacts, such as the hiss of aliasing later in the section, are nearly invisible to the coarse MSD scales, and the little that gets through shows up at the wrong frequency. The MPD sees them in full.

The ablations make the point at scale. Removing the MPD dropped HiFi-GAN V3 from a MOS of 4.10 to 2.28, and adding an MPD to MelGAN raised it from 2.88 to 3.35. An interviewer who asks what HiFi-GAN contributed expects the MPD, with these numbers.

Reconstruction losses that ignore phase

A GAN alone learns slowly and can produce plausible speech that does not match the input mel. So every GAN vocoder adds a reconstruction loss, a direct comparison of its output with the real waveform. That loss must not repeat the mistake of the regression vocoder, so it compares magnitudes, never raw samples. HiFi-GAN uses the mel spectrogram itself,

\mathcal{L}_{\text{mel}}(G) = \mathbb{E}\,\big\lVert\phi(\mathbf{x}) - \phi\big(G(\phi(\mathbf{x}))\big)\big\rVert_1 ,

the L1 distance between the log-mel of the real waveform and that of the generated one. Parallel WaveGAN (Yamamoto, Song & Kim 2020) introduced a loss at several STFT resolutions, which neural codecs reuse.

📐 Multi-resolution STFT loss

For a resolution $m$ given by an FFT size, a window length, and a hop, let $\mathbf{S}_m(\mathbf{x})$ be the STFT magnitudes of $\mathbf{x}$ , with $N_m$ entries. For a real waveform $\mathbf{x}$ and a generated one $\hat{\mathbf{x}}$ ,

\mathcal{L}_{\text{sc}}^{(m)} = \frac{\lVert\mathbf{S}_m(\mathbf{x}) - \mathbf{S}_m(\hat{\mathbf{x}})\rVert_F}{\lVert\mathbf{S}_m(\mathbf{x})\rVert_F}, \qquad \mathcal{L}_{\text{mag}}^{(m)} = \frac{1}{N_m}\big\lVert\log\mathbf{S}_m(\mathbf{x}) - \log\mathbf{S}_m(\hat{\mathbf{x}})\big\rVert_1, \qquad \mathcal{L}_{\text{mr}} = \frac{1}{M}\sum_{m=1}^{M}\big(\mathcal{L}_{\text{sc}}^{(m)} + \mathcal{L}_{\text{mag}}^{(m)}\big),

the spectral convergence and log-magnitude terms, averaged over $M$ resolutions. Implementations floor the magnitudes at a small $\epsilon$ before the logarithm. Parallel WaveGAN uses $M = 3$ at 24 kHz, with FFT sizes 1024, 2048, 512, Hann windows of 600, 1200, 240 samples, and hops 120, 240, 50.

In plain words, the spectral convergence term is a relative error dominated by the loud bins. The log term weighs every bin equally, loud or quiet. Neither looks at phase. Several resolutions are needed because of the time-frequency trade-off of §35.2. A 50 ms window resolves harmonics but smears a plosive's onset over 50 ms, and a 10 ms window catches the onset but blurs the harmonics. A generator fit at one resolution can hide its errors in that resolution's blind spot.

The cell compares a reference voice (8 harmonics of 200 Hz over a faint noise floor) with four alternatives, under both spectral terms and under a plain waveform L1 loss.

Python · cell 7

Read the output row by row. New harmonic phases, which a listener barely notices, cost more under the waveform L1 than replacing the voice by white noise, while the spectral terms barely move. A missing harmonic, which a listener does notice, raises the spectral convergence sharply. A fresh noise floor 60 dB down is inaudible, yet it costs the log term more than the missing harmonic does, because the log term counts the thousands of quiet bins between harmonics as much as the loud ones. No generator can reproduce one particular noise realization from a mel spectrogram, so in quiet regions the log term has the same averaging problem as the waveform loss. That is why spectral losses sit next to an adversarial loss and never replace it.

Core

Multi-resolution STFT loss

The setup provides stft_mag(x, n_fft, win, hop), which returns the magnitude array of a 1-D signal, frames as rows, with a Hann window of length win centered in n_fft samples. Implement mr_stft_loss(x, y, configs, eps=1e-7) for a real signal x, a generated signal y, and a list of (n_fft, win, hop) tuples. For each resolution, compute the two terms of the definition box and add them.

The spectral convergence $\lVert\mathbf{S} - \hat{\mathbf{S}}\rVert_F/(\lVert\mathbf{S}\rVert_F + \epsilon)$ .
The log-magnitude term, the mean over all entries of $\lvert\log\max(\mathbf{S}, \epsilon) - \log\max(\hat{\mathbf{S}}, \epsilon)\rvert$ , with the natural logarithm.

Return the average of these sums over the resolutions, as a float.

import numpy as np

def mr_stft_loss(x, y, configs, eps=1e-7):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

HiFi-GAN's full objective

One more term completes the objective. The feature matching loss (Larsen et al. 2016, used for audio by MelGAN) compares the intermediate activations of each sub-discriminator on the real waveform and on the generated one. It turns the critic into a learned measure of how different two sounds are. It also gives the generator a dense, stable signal even when the critic's final scores say little.

📐 HiFi-GAN objective

Let $\operatorname{Disc}_1, \dots, \operatorname{Disc}_8$ be the five MPD and three MSD sub-discriminators, $\operatorname{Disc}_k^{(i)}$ the activations of layer $i$ of $\operatorname{Disc}_k$ with $N_i$ entries, $\mathbf{x}$ a real waveform, and $\hat{\mathbf{x}} = G(\phi(\mathbf{x}))$ the generator's output from its mel spectrogram. The losses are

\mathcal{L}_{\text{fm}}(G; \operatorname{Disc}_k) = \mathbb{E}\sum_{i}\frac{1}{N_i}\big\lVert\operatorname{Disc}_k^{(i)}(\mathbf{x}) - \operatorname{Disc}_k^{(i)}(\hat{\mathbf{x}})\big\rVert_1,

\mathcal{L}_G = \sum_{k=1}^{8}\Big[\mathcal{L}_{\text{adv}}(G; \operatorname{Disc}_k) + \lambda_{\text{fm}}\,\mathcal{L}_{\text{fm}}(G; \operatorname{Disc}_k)\Big] + \lambda_{\text{mel}}\,\mathcal{L}_{\text{mel}}(G), \qquad \mathcal{L}_D = \sum_{k=1}^{8}\mathcal{L}_D(\operatorname{Disc}_k),

with the least-squares $\mathcal{L}_{\text{adv}}$ and $\mathcal{L}_D$ of the definition above, each averaged over its sub-discriminator's grid of scores, and $\lambda_{\text{fm}} = 2$ , $\lambda_{\text{mel}} = 45$ . The two players take alternating AdamW steps ( $\beta_1 = 0.8$ , $\beta_2 = 0.99$ , weight decay 0.01, learning rate $2\times10^{-4}$ decayed by 0.999 per epoch) on random 8,192-sample segments.

In plain words, the generator is told three things at once. Sound real to eight critics. Produce the same internal features in those critics that real audio does. Match the input mel. The weight 45 balances terms of very different kinds, a per-element mean of log-mel errors against squared critic scores. The mel term ties the output to its input from the first step, so the adversarial terms only have to add realism. HiFi-GAN's ablation measured each piece on V3, one of its two small generators. Without the mel loss the MOS fell from 4.10 to 3.25, without the MPD to 2.28, and without the MSD to 3.74.

Interview

HiFi-GAN's discriminator plumbing and losses

Implement the pieces that sit between the networks and the optimizer. Discriminator outputs and features arrive as NumPy arrays.

mpd_reshape(x, p) takes x of shape (B, 1, T) and pads the time axis on the right to a multiple of p by reflection, with no padding if T already is one. Reflection mirrors the signal about its last sample without repeating it, as NumPy's and PyTorch's "reflect" modes do, so samples $a, b, c$ padded by two become $a, b, c, b, a$ . It returns shape (B, 1, T_pad // p, p), with element [b, 0, m, j] equal to the padded x[b, 0, m * p + j].
lsgan_d_loss(real_outs, fake_outs) takes two lists with one array of scores per sub-discriminator and returns $\sum_k[\operatorname{mean}((1 - r_k)^2) + \operatorname{mean}(f_k^2)]$ .
lsgan_g_loss(fake_outs) returns $\sum_k\operatorname{mean}((1 - f_k)^2)$ .
feature_matching(real_feats, fake_feats) takes two lists with one entry per sub-discriminator, each a list of arrays, one per layer. It returns the sum over sub-discriminators and layers of the mean absolute difference within each layer.

import numpy as np

def mpd_reshape(x, p):
    raise NotImplementedError

def lsgan_d_loss(real_outs, fake_outs):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The generator upsamples by the hop

HiFi-GAN's generator must turn 86 mel frames per second into 22,050 samples per second, a factor of 256, the hop. It does this in four stages, each of which stretches time by a fixed factor and then refines the result. The generator is fully convolutional, made only of convolutions, so it works on any number of frames.

Two building blocks need a sentence each. A transposed convolution with stride $s$ and kernel $k$ lets each input step stamp a learned pattern of $k$ values onto the output, each stamp shifted $s$ steps from the previous one, which multiplies the length by $s$ (§0.8). A multi-receptive-field fusion (MRF) module runs three residual blocks in parallel on the same input, with kernel sizes 3, 7, and 11, and averages their outputs, so short and long patterns are modeled side by side. Each of those blocks has three pairs of convolutions. In each pair the first convolution has dilation 1, 3, or 5, and the second has dilation 1. (The paper says the MRF sums the blocks, and the official code divides the sum by three.)

The V1 generator is then this chain, with a leaky ReLU (slope 0.1) in front of each upsampler and each convolution inside the MRF blocks.

A 7-tap convolution lifts the 80 mel channels to 512.
Four stages, each a transposed convolution that upsamples time and halves the channels, followed by an MRF. The upsampling rates are 8, 8, 2, 2, with kernels 16, 16, 4, 4 and padding $(k - s)/2$ , so the channels go 512, 256, 128, 64, 32.
A final 7-tap convolution to one channel and a tanh produce the waveform in $[-1, 1]$ .

Proof. The output length of a transposed convolution, from §0.8, is $(n - 1)s - 2p + k$ for padding $p$ , with no dilation or output padding. With $p = (k - s)/2$ this is $ns - s - (k - s) + k = ns$ . Length-preserving layers change nothing, and composing the stages multiplies the factors. ∎

✏️ Factoring a hop

HiFi-GAN's hop of 256 at 22.05 kHz factors as $8\cdot8\cdot2\cdot2$ , and its V3 uses $8\cdot8\cdot4$ . A 12.5 ms hop at 24 kHz is 300 samples, and $300 = 2^2\cdot3\cdot5^2$ , so one choice is $5\cdot5\cdot3\cdot2\cdot2$ and another $10\cdot5\cdot3\cdot2$ . A 10 ms hop at 24 kHz is $240 = 8\cdot5\cdot3\cdot2$ . EnCodec's 320 samples per frame (§37.3) is $8\cdot5\cdot4\cdot2$ . A hop with a large prime factor forces one huge stride, which is why front ends choose hops with small factors. An interviewer who hands you a sample rate and a hop expects this factorization in a few seconds.

Now follow the sentence Dr. Smith paid $5. from the start of the section through HiFi-GAN V1. Its 156 mel frames enter as an $80\times156$ array at 86 frames per second. Each stage multiplies the time axis by its stride and halves the channels, and the cell prints the shape after each one.

Python · cell 8

✏️ The sentence of §37.1 through HiFi-GAN V1

The $80\times156$ log-mel array of Dr. Smith paid $5. becomes $512\times156$ after the input convolution. The four stages turn it into $256\times1{,}248$ at 689 steps per second, $128\times9{,}984$ at 5,512, $64\times19{,}968$ at 11,025, and $32\times39{,}936$ at the full 22,050. The output convolution and tanh leave $1\times39{,}936$ , 1.811 s of samples in $[-1, 1]$ , exactly 256 per frame. The last three stages each hold 1.28 million numbers, 32 times the size of the waveform they produce. Within each layer all of those numbers are computed at once, with no step waiting for the previous sample, which is the whole difference from WaveNet.

Checkerboards and tonal artifacts

§0.8 warned that a transposed convolution overlaps its stamps unevenly when the kernel size is not a multiple of the stride. Images then show a faint checkerboard (Odena, Dumoulin & Olah 2016). Here is the smallest case. With kernel 3 and stride 2, each input stamps 3 outputs and moves 2 to the right, so the outputs receive 2, 1, 2, 1, ... stamps in turn. In audio a pattern that repeats every few samples is not a picture but a tone, so it is worth knowing exactly when it happens.

∴ Overlap of a transposed convolution

Let a 1-D transposed convolution with stride $s$ and kernel $k$ (no padding) stamp each of $n$ inputs onto the output, with input $i$ covering outputs $is, \dots, is + k - 1$ . Write $k = as + b$ with $0 \le b \lt s$ . Every interior output position $o$ (with $k - 1 \le o \le (n-1)s$ ) receives either $a$ or $a + 1$ stamps, and a fraction $b/s$ of positions receive $a + 1$ . All interior positions receive the same number, $k/s$ , if and only if $s$ divides $k$ .

More generally, output $o$ uses only the kernel taps $u \equiv o \pmod s$ , the taps whose index leaves the same remainder as $o$ on division by $s$ . So for a constant input the interior output is periodic with period $s$ , with value $g_r = \sum_{u \equiv r \ (\mathrm{mod}\ s)}w_u$ at positions $o \equiv r$ . The proof is in the derivation below.

∂ Proof of the overlap count

Input $i$ reaches output $o$ through tap $u = o - is$ , which must lie in $\{0, \dots, k-1\}$ . So the stamps on $o$ are the multiples $is$ of $s$ in the window of $k$ consecutive integers $o - k + 1, \dots, o$ , and for interior $o$ every such $i$ is a valid input index. Split the window into $a$ blocks of $s$ consecutive integers, each containing exactly one multiple of $s$ , and $b$ leftover integers. The leftovers contain one multiple of $s$ for exactly $b$ of the $s$ residues of $o$ modulo $s$ , and none for the others. So the count is $a$ or $a + 1$ , the latter at a fraction $b/s$ of positions, and it is constant if and only if $b = 0$ . Finally $u = o - is \equiv o \pmod s$ , so with all inputs equal to 1, output $o$ is the sum of the taps $u \equiv o$ , which depends only on $o \bmod s$ . ∎

In plain words, a stride- $s$ transposed convolution is $s$ separate filters, its polyphase components, whose outputs are interleaved. With an all-ones kernel the components differ only in how many taps they have, and that difference is the checkerboard. Choosing $k$ a multiple of $s$ fixes it, as HiFi-GAN ( $k = 2s$ ) and MelGAN do.

But a trained kernel's components can have different gains even when $k = 2s$ . Then a steady input comes out modulated with a period of $s$ samples. Its spectrum has lines at multiples of $f_{\text{out}}/s$ , which is the input rate of that layer. So the first upsampler of a 22.05 kHz HiFi-GAN can leave a faint tone at multiples of 86 Hz, the frame rate. Pons et al. (2021) call these tonal artifacts. They show that nearest-neighbor upsampling followed by a convolution avoids them, since every output phase then uses the same filter, at the price of a mild low-pass effect.

The cell counts the stamps for four kernel and stride pairs, then runs a random kernel with $k = 2s$ on a constant input.

Python · cell 9

With $k = 3$ , $s = 2$ the counts alternate 2, 1, and with $k = 4$ or $16$ against strides 2 and 8 they are flat. The random kernel with $k = 2s$ still turns a constant input into a pattern of period 8 equal to its polyphase gains. That is the tonal artifact, which the counting argument alone cannot see. In a design review, "is $k$ a multiple of $s$ ?" is the first question about an upsampler, and "are the polyphase gains equal?" is the second.

Core

Where checkerboards and tones come from

Implement three functions for a 1-D transposed convolution with stride s, kernel size k, and no padding. Input i adds x[i] * w to the outputs i * s through i * s + k - 1, so n_in inputs give (n_in - 1) * s + k outputs.

transposed_overlap(k, s, n_in) returns an integer array with the number of kernel taps that land on each output position when the kernel and the input are all ones.
ripple(k, s) uses n_in = 2 * k + 2 and returns the maximum minus the minimum of the counts over the interior positions, k - 1 to (n_in - 1) * s inclusive.
polyphase_gains(w, s) returns an array of s values for a constant input of ones and the kernel w. Entry r is the output value at the interior positions o with o % s == r.

import numpy as np

def transposed_overlap(k, s, n_in):
    raise NotImplementedError

def ripple(k, s):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Aliasing, and periodic activations

A second artifact comes from the activation functions. A pointwise nonlinearity applied at the sample rate turns a tone at frequency $f$ into harmonics at $2f, 3f, \dots$ . Recall the folding formula of §35.1. A sampled signal cannot represent frequencies above the Nyquist frequency $f_s/2$ , and a harmonic at $f'$ above it folds back to $\lvert f' - mf_s\rvert$ for the nearest whole number $m$ . The folded harmonics land at frequencies that are not harmonics of anything in the signal, and the ear hears them as roughness or hiss.

The fix, taken from image generators (Karras et al. 2021), is to apply the nonlinearity at a higher rate. Upsample by 2 with a low-pass filter, apply the nonlinearity, low-pass at the original Nyquist frequency, and downsample by 2. The cell applies HiFi-GAN's leaky ReLU to a 3 kHz tone at 22.05 kHz, once directly and once at twice the rate, and measures the power that lands away from the tone's harmonics.

Python · cell 10

A leaky ReLU with slope 0.1 equals $0.55x + 0.45\lvert x\rvert$ . The absolute value of a sine repeats every half period, so it adds only even harmonics, at 6, 12, 18, 24 kHz, and so on. At 22.05 kHz every harmonic above the Nyquist frequency of 11.025 kHz folds back. The one at 12 kHz lands at 10,050 Hz, 18 kHz at 4,050 Hz, 24 kHz at 1,950 Hz, and 30 kHz at 7,950 Hz. Together the aliases sit 25.5 dB below the signal. Running the same activation at twice the rate pushes them down to $-42.7$ dB. (Harmonics above 22.05 kHz still fold even at the doubled rate, which is why the improvement is finite.)

BigVGAN (Lee et al. 2023) builds this into every residual layer, and it also changes the activation itself.

📐 Snake activation

The Snake activation (Ziyin, Hartwig & Ueda 2020) with a frequency parameter $\alpha \gt 0$ (in this section $\alpha$ is only this parameter) is

f_\alpha(x) = x + \frac{1}{\alpha}\sin^2(\alpha x).

Its derivative is $f_\alpha'(x) = 1 + \sin(2\alpha x) \in [0, 2]$ , so it is non-decreasing, its periodic part repeats every $\pi/\alpha$ , and $f_\alpha(x) \to x$ as $\alpha \to 0$ because $\sin^2(\alpha x)/\alpha \approx \alpha x^2$ . BigVGAN learns one $\alpha$ per channel and applies Snake inside an anti-aliased multi-periodicity composition (AMP). That is 2 times upsampling with a low-pass filter (a Kaiser-windowed sinc), Snake, another low-pass, and 2 times downsampling.

In plain words, Snake is a straight line with a learnable ripple. It gives each channel a built-in oscillation at a frequency of its choosing, which suits a network whose output is a sum of oscillations, and the AMP keeps the ripple's harmonics from folding. Trained only on LibriTTS speech at 24 kHz, BigVGAN generalizes to unseen speakers, recording conditions, singing, and instrumental music without fine-tuning, which is what a universal vocoder means.

Skipping the upsampler with an inverse STFT

A time-domain generator spends most of its layers upsampling. It re-creates at 22,050 samples per second a structure that a Fourier transform could produce exactly. The alternative is to predict an STFT and let a fixed inverse STFT make the samples. iSTFTNet (Kaneko et al. 2022) replaced the last upsampling stages of HiFi-GAN by an inverse STFT, and Vocos (Siuzdak 2024) went all the way.

Vocos's backbone is a stack of ConvNeXt blocks, convolutional blocks borrowed from image classification (Liu et al. 2022), and it runs only at the frame rate, never changing the time resolution. For each frame its head outputs two vectors $\mathbf{m}, \mathbf{p} \in \R^{N/2+1}$ . The complex STFT coefficient of bin $k$ is $\exp(m_k)(\cos p_k + i\sin p_k)$ , and a fixed inverse STFT with the analysis hop turns the frames into a waveform. The exponential keeps magnitudes positive. Passing $p_k$ through cosine and sine makes every real number a valid phase, so the network never has to model the jump at $\pm\pi$ that makes raw phase hard to regress.

Vocos trains with an MPD, a multi-resolution discriminator, and the hinge loss of §32.6. Its listening-test quality was not significantly different from BigVGAN's, and on a CPU it ran 170 times real time against 5.8 for HiFi-GAN in the same benchmark. Skipping the high-rate layers is where that factor of 30 comes from.

What it costs at real scale

A GAN vocoder is fast because all of its work is arithmetic that runs in parallel, and that arithmetic can be counted. A convolution from $C_{\text{in}}$ to $C_{\text{out}}$ channels with kernel $k$ costs $2C_{\text{in}}C_{\text{out}}k$ floating-point operations (FLOPs) per output time step, one multiply and one add per weight. A transposed convolution costs the same per input step. Each layer runs at its own rate, so its cost per second of audio is its cost per step times its steps per second. The cell counts HiFi-GAN V1 layer by layer.

Python · cell 11

HiFi-GAN V1 needs about 53 GFLOP per second of audio, 2.4 MFLOP per output sample. Almost all of it is in the MRF blocks, and the largest share is in the 128-channel blocks at 5.5 kHz. It is not in the biggest weight matrices, the 256-channel blocks at 689 Hz, which hold most of the 13.9 M parameters but run at a low rate.

A V100's peak of 15.7 TFLOP/s in 32-bit floats would allow about 300 times real time. So the measured 168 implies about 57 percent of peak, the efficiency of large, regular convolutions over many time steps at once. A WaveNet step is a handful of matrix-vector products for a single sample, a few percent of peak at best, repeated 22,050 times in sequence.

The arithmetic also says what to shrink for a CPU. The cost scales with channels squared, times kernel size, times rate. V2 cut the hidden width from 512 to 128 (0.92 M parameters, about 16 times fewer FLOPs, 9.7 times real time on the CPU), and V3 used fewer layers with wider dilations. Vocos removes the high-rate layers altogether, which is why its CPU speed is so far ahead.

One model from text to waveform

The two-stage pipeline, an acoustic model and then a vocoder, has a seam. The vocoder is trained on mels computed from real audio, and at inference it hears the acoustic model's smoother predictions. VITS (Kim, Kong & Son 2021) removes the seam by training both stages at once as a conditional variational autoencoder. Recall from §32.3 that a variational autoencoder (VAE) encodes data into a latent variable and decodes it back. It is trained by maximizing a lower bound on the likelihood, the ELBO, which is a reconstruction term minus a KL term that keeps the encoder's distribution close to a prior.

training:   linear spectrogram --posterior encoder--> z --HiFi-GAN decoder (on windows)--> waveform
                                                      |                                     |
            phonemes --text encoder--> Gaussians --(MAS alignment)--> prior on z via a flow  |
                                                                    L1 on mels, MPD + feature matching
inference:  phonemes --> text encoder --> stochastic durations --> sample prior --> inverse flow --> z --> decoder

In the diagram, a vocoder is still there. It is the HiFi-GAN V1 generator, decoding a learned latent $\mathbf{z}$ instead of a mel spectrogram, and it is trained with the same multi-period critic. On LJSpeech VITS scored 4.43 against 4.46 for the recordings, above Tacotron 2 with HiFi-GAN (3.77, or 4.25 with the vocoder fine-tuned on Tacotron 2's outputs) and above HiFi-GAN driven by Glow-TTS, a flow-based acoustic model (Kim et al. 2020), at 4.32 fine-tuned.

🔬 The parts of VITS

The posterior encoder reads a linear spectrogram of the real speech and outputs a Gaussian over latent frames $\mathbf{z}$ .
The prior comes from the text. The text encoder outputs a Gaussian per phoneme, and the monotonic alignment search of §37.1, a dynamic program that finds the most likely in-order assignment of frames to phonemes, expands them to frames. A normalizing flow makes this prior more expressive.
A flow-based stochastic duration predictor learns how long each phoneme lasts, so the same text can be spoken with different rhythms.
The decoder is the HiFi-GAN V1 generator, trained on random windows of $\mathbf{z}$ .

The losses are an L1 mel reconstruction loss (a Laplace likelihood in the ELBO), the KL term of the ELBO, and the least-squares adversarial and feature-matching losses with HiFi-GAN's multi-period discriminator.

The real way, in PyTorch

The block below builds a six-layer gated WaveNet stack (kernel 2, dilations 1 to 32) and checks causality with autograd. The gradient of output $n$ with respect to every input after $n$ must be exactly zero, and its support must be the receptive field. It then builds a HiFi-GAN generator twice. A tiny one with rates 4, 4 maps a random $(1, 80, T)$ mel to $(1, 1, 16T)$ samples, and the full V1 configuration is built only to count its parameters. Finally it runs one period sub-discriminator and computes the least-squares losses.

🔥 In PyTorchfor a machine with PyTorch

To use a trained vocoder, load published weights rather than training your own. Both calls below follow the libraries' own documentation as of 2026 (the vocos package and NVIDIA's BigVGAN repository). Each expects the mel configuration its model was trained on. Both models below use 100 bands at 24 kHz with hop 256, yet their mel formulas and floors differ, so each gets its mel from its own feature code. That is the first thing to check when a vocoder sounds wrong.

Where vocoders break

Train-test mismatch. A vocoder trained on mels of real audio meets an acoustic model's predictions at inference. Those are smoother, because the acoustic model was trained with a squared or absolute error and so predicts something close to an average. Tacotron 2 measured the cost. Its WaveNet trained on ground-truth mels and fed predicted ones scored 4.362, against 4.526 when trained on the predictions themselves. The fix is to train the vocoder on the acoustic model's teacher-forced predictions, made while the acoustic model is fed the true previous frames, so that they line up frame for frame with the real waveform. Tacotron 2 trained its WaveNet on such predictions from the start. HiFi-GAN fine-tuned an already trained V1 on them for 100,000 steps, and its MOS rose from 3.91 to 4.18 (ground truth 4.23 in that test).

Unseen conditions. A vocoder trained on one clean speaker generalizes only partly. HiFi-GAN V1 scored 3.77 against 3.79 for the recordings on nine held-out speakers of the VCTK corpus. But noisy recordings, singing, and music fall outside what a leaky-ReLU generator learned from one reader, which is BigVGAN's reason for periodic activations and large, diverse training data.

Characteristic artifacts. Each kind of artifact points to a cause.

What you hear	Likely cause	Where it is fixed
metallic buzz, wavering pitch	phase inconsistent from one pitch period to the next	the multi-period critic
faint whine at the frame rate and its harmonics	unequal polyphase gains in an upsampler (tonal artifact)	$k$ a multiple of $s$ , or nearest-neighbor upsampling
hiss or roughness in the high frequencies	aliasing from the activations	anti-aliased activations (BigVGAN)
muffled, smeared speech	oversmoothed input mels	fine-tuning on predicted mels

GAN training can also be unstable or drop modes, which is why the mel loss carries weight 45 and why BigVGAN needed gradient clipping at 112 M parameters.

Budgets. An on-device assistant has a CPU and tens of milliseconds, so it needs a V3-class model, LPCNet, or a Fourier-head vocoder. A podcast service on GPUs can afford V1 or Vocos, and a music application needs BigVGAN-class generalization. The last interview question below works through this choice.

🧭 What to remember

A vocoder undoes an analysis that threw away phase and pooled detail. The waveform given a mel spectrogram has many plausible versions, and their average is near silence, so the vocoder must be a generative model and its reconstruction losses must compare magnitudes, never raw samples.
WaveNet factorizes $p(\mathbf{x}\mid\mathbf{c})$ sample by sample with gated residual blocks of dilated causal convolutions. Their receptive field is $r = 1 + (k-1)\sum_l d_l$ , 3,070 samples for three cycles of dilations 1 to 512 with kernel 2, and mel conditioning makes long fields unnecessary.
Mu-law quantizes the logarithm of the amplitude, so its error is a fixed fraction of the sample and its SQNR is about $12/(\Delta_y\ln(1+\mu))^2$ , 38.1 dB at 8 bits at every level, where 8-bit linear coding loses a decibel for every decibel of level.
Sample-level autoregression costs $f_s$ steps in sequence per second of audio. WaveRNN and LPCNet make each step tiny, and Parallel WaveNet distills the model into a flow. WaveGlow generates in one parallel pass, diffusion vocoders in a few, and GANs in one.
HiFi-GAN pairs a transposed-convolution generator whose strides multiply to the hop with five period critics (primes 2 to 11) and three scale critics. It trains with least-squares losses (optimal critic $p_{\text{data}}/(p_{\text{data}}+p_G)$ , minimum exactly at $p_G = p_{\text{data}}$ ), feature matching (weight 2), and a mel L1 loss (weight 45). V1 runs 168 times real time on a V100 at 53 GFLOP per second of audio.
Upsampling leaves checkerboards when $s$ does not divide $k$ , and tones at the layer's input rate when the polyphase gains differ, and activations alias. BigVGAN answers with Snake and anti-aliased activations, Vocos with an inverse STFT head at the frame rate, and vocoders trained on real mels need fine-tuning on predicted ones.
Next, §37.3 replaces the hand-designed mel spectrogram by a learned discrete code, an autoencoder with residual vector quantization whose decoder is a vocoder trained with these same critics and losses.

van den Oord et al. (2016). WaveNet: A Generative Model for Raw Audio. Sections 2.1 to 2.5 give the dilated causal stack, mu-law with the 256-way softmax, the gated residual block, and both kinds of conditioning, and Table 1 the listening tests, including 8-bit mu-law against 16-bit PCM.
Kalchbrenner et al. (2018). Efficient Neural Audio Synthesis. WaveRNN, with the clearest accounting of why sample-level generation is slow (operations per step, memory bandwidth), the dual softmax, sparsity, and subscale generation.
van den Oord et al. (2018). Parallel WaveNet: Fast High-Fidelity Speech Synthesis. Probability density distillation of an autoregressive teacher into an inverse autoregressive flow, with the speed and MOS comparison quoted here.
Yamamoto, Song & Kim (2020). Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. The multi-resolution STFT loss with its spectral convergence and log-magnitude terms and the three resolutions of its Table 1.
Kong, Kim & Bae (2020). HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. The multi-period discriminator, the full objective with its weights, the speed and quality table, and ablations of every component. The official repository's config_v1.json holds the exact generator configuration.
Lee et al. (2023). BigVGAN: A Universal Neural Vocoder with Large-Scale Training. Snake activations, anti-aliased multi-periodicity composition, and what it takes to scale a GAN vocoder to 112 M parameters and to unseen domains.
Siuzdak (2024). Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis. The inverse-STFT head with magnitude and unit-circle phase outputs, and the speed comparison against time-domain vocoders.
Pons et al. (2021). Upsampling artifacts in neural audio synthesis. The signal-processing analysis of tonal and filtering artifacts from transposed and subpixel convolutions, the audio counterpart of the checkerboard.

Made for larger screens

Neural vocoders

What a vocoder has to invent

Why a regression cannot do it

WaveNet writes one sample at a time

One sample is one of 256 classes

Mu-law in and out

Dilated causal convolutions see far

The gated residual block

A WaveNet stack and its receptive field

Generating is sequential, and caching makes each step cheap

Fast WaveNet generation with per-layer queues

Four ways around one sample at a time

GAN vocoders

The least-squares game

Critics that listen at several periods and scales

Reconstruction losses that ignore phase

Multi-resolution STFT loss

HiFi-GAN's full objective

HiFi-GAN's discriminator plumbing and losses

The generator upsamples by the hop

Checkerboards and tonal artifacts

Where checkerboards and tones come from

Aliasing, and periodic activations

Skipping the upsampler with an inverse STFT

What it costs at real scale

One model from text to waveform

The real way, in PyTorch

Where vocoders break

Read more

Watch next