How a model learns to speak. Text becomes phonemes, durations, and mel frames, vocoders turn frames into waveforms, neural codecs turn sound into tokens, and language models over those tokens clone voices, make music, and hold full-duplex conversations, built from first principles to the voice models of 2026 and taken to interview depth.

Text to speech, or TTS, turns written text into the sound of a person reading it aloud. It is the voice of every assistant, screen reader, navigation app, and audiobook tool, and the last stage of every spoken chatbot. This section follows a sentence from its written form to a mel spectrogram, a picture of the sound's energy over time and frequency, which a second network turns into a waveform in §37.2. The hard part is length. A sentence of 30 characters becomes about 170 spectrogram frames, and the model has to decide which frames belong to which sound, in order, without skipping or repeating anything.

This chapter runs speech recognition in reverse. Chapter 36 turned audio into text, and this chapter turns text into audio. It then turns audio into discrete tokens that a language model can write, and ends with voice models that listen and speak at once (§37.5). It also builds on earlier parts of the book. From Chapter 35 it takes the mel spectrogram (§35.3) and the self-supervised speech encoders (§35.5). From the generative models it takes vector quantization (§32.5) and flow matching (§33.6), each recapped where it is used. A voice that sounds human can also impersonate one, so the chapter's last section covers how voice models are measured and secured (§37.6).

Two failures show what is at stake. Give a model trained on spelled-out text the line Dr. Smith lives on Smith Dr. and paid $1,200 on 3/4, and you may hear "D R Smith", "dollar one comma two hundred", and "three slash four". The writing was never turned into words. Now give an attention-based model a long sentence that uses the same short word several times. It may say one phrase twice, jump over another, and keep making sound after the text has run out. Here every word was known, and the model lost its place.

The second failure comes from a mismatch in length. Speech is far longer than its text, and the model has to stretch the text over it in order, never going back.

Here is the plan. First comes the front end, the classical machinery that turns writing into words and words into sound symbols. Then we state the length problem exactly and count its solutions. Then come three answers to it, in historical order. The first learns the alignment with attention, one frame at a time, and fails by skipping and repeating (Shen et al. 2018). The second predicts how many frames each sound lasts and copies its vector that many times (Ren et al. 2019), with the frame counts found by an exact dynamic program (Kim et al. 2020). The third generates a spectrogram of a chosen total length and lets the timing emerge inside it (Chen et al. 2024). Prosody, the pitch, loudness, and timing that the words leave open, runs through all three.

The pipeline, with the shape at every arrow

Every TTS system, from the rule-based synthesizers of the 1980s to the voice models of 2026, can be drawn as the same chain of five stages. Modern systems merge some of them, but the data still passes through the same forms.

In plain words, text becomes words, words become sound symbols, sound symbols become vectors, the vectors become a picture of the sound, and the picture becomes air pressure. Stages 1 and 2 are the front end. Stages 3 and 4 together are what papers call the TTS model. Stage 5 is the next section.

The mel spectrogram is the hinge between the two halves. Recall how it is made (§35.3). Slice the waveform into short overlapping windows, take the magnitude of each window's spectrum, pool it into $n_{\text{mels}}$ bands spaced the way hearing spaces pitch, and take a logarithm. Each of those steps has settings, and an acoustic model must produce exactly the recipe its vocoder was trained on. Two recipes cover most of the literature.

The first column is from the Tacotron 2 paper. The second is from the official HiFi-GAN repository's config_v1.json and meldataset.py, which NVIDIA's Tacotron 2 implementation and most LJSpeech models share. A 10 s utterance is $10\times 86.13 \approx 861$ frames in the second recipe and $10/0.0125 = 800$ frames in the first.

Each stage has its own audible failure. Telling which stage broke from a bad sample is the first thing a TTS engineer is asked to do.

One sentence, end to end

Before we build each stage properly, follow one sentence through all of them with toy parts, so that every shape has a number. The sentence is Dr. Smith paid $5. The first cell does the front end by hand, gives every phoneme a made-up duration (10 frames for a vowel, 6 for a consonant, and 4 more for each of the last two, since speakers slow down at the end), and expands the encoder's vectors to one per frame.

The second cell stands in for the acoustic model and the vocoder. Real systems go from vectors to a mel spectrogram and then to a waveform. Our toy goes straight to a waveform with hand-written rules, the way the formant synthesizers of the 1980s worked. Each vowel is a sum of harmonics of a falling pitch, boosted near two resonant frequencies of the mouth (the formants). Each hissing sound (/s/, /f/, /th/) is noise in one frequency band, and /z/ and /v/ add that hiss to a weak voiced sound. Each stop (/d/, /k/, /t/, /p/) is a silence and a short burst.

The third cell computes the log-mel spectrogram of that waveform with settings close to the second recipe in the table above (1,024-sample windows, hop 256, 80 bands up to 8 kHz, a floor of $10^{-5}$). It spaces its bands with the HTK mel formula, while the librosa function that HiFi-GAN calls defaults to the Slaney formula, which is one more setting a real acoustic model and its vocoder must share. That spectrogram is what a neural acoustic model would have to predict.

In the picture, each vowel shows as horizontal bars at its formants, each hiss as energy high up, and each stop as a dark gap that ends in a short burst across all bands. Every vertical line is a phoneme boundary, which is exactly what the durations decided. The rest of this section replaces the toy front end and acoustic model with real ones. §37.2 replaces the formant rules with a neural vocoder, and §37.4 shows the other route, where the acoustic model writes discrete audio tokens instead of mel frames.

The widget runs the same sentence through every stage on one time axis. Each chip is a phoneme, vowels in indigo and consonants in green, and the curves in the durations row show the length regulator copying each phoneme over its frames. Press Play to watch the stages fill in, or click a stage name to stop there. The codec tokens button switches to the other route, which §37.4 builds.

Text normalization turns writing into words

Written text is full of tokens that are not spoken as written. Numbers, dates, times, money, measures, abbreviations, URLs, and symbols all have to be expanded, and the right expansion depends on context.

The same token reads differently in context. 1985 is "nineteen eighty five" as a year and "one thousand nine hundred eighty five" as a count of people. St. is "Saint" before a name and "Street" after one. Dr. is "Doctor" before a name and "Drive" after a street name. 3/4 is "March fourth" in an American date, "the third of April" in a British one, and "three quarters" in a recipe. IV is "the fourth" after a king's name and "I V" on a hospital chart.

The word "normalization" means three different things in this book, so it is worth separating them. Unicode normalization (§2.5) makes byte sequences canonical and changes no words. Normalization for scoring speech recognition (§36.5) maps transcripts to one canonical form so that "Mr" and "mister" count as the same word. TTS normalization goes from written to spoken form, and it is the hard one, because it has to decide.

Can a neural network learn this map? Sproat and Jaitly (2016) built a large corpus of written text paired with its spoken form and trained recurrent sequence-to-sequence models on it. The models were very accurate on average, because almost every token maps to itself. But their errors were of a kind no product can ship. Their Table 4 lists English examples, such as 60 read as "six", 82.55 mm as "eighty two one five five meters", and an amount in pounds read as euros. If the text says 381 kg, a TTS system had better not say three hundred forty one kilograms. A plausible but wrong reading changes the message, which is worse than an awkward reading.

So production front ends keep a grammar. The classic design is a set of weighted finite-state transducers, hand-written rules per language and semiotic class that can only produce readings consistent with the token's digits and units. Sproat and Jaitly built such a grammar for measures and money and used it as a filter around the neural model, so that it could choose only among licensed readings. That removed most of the silly errors in those two classes and left the others untouched. The network still helps with classification, deciding which licensed reading the context calls for. In an interview, this is the answer to "why not just use a seq2seq model for normalization?"

The cell builds the classification step for a few classes, with one context rule for Dr. and St. (a title if the next word is capitalized, a street otherwise), and a verbalizer for numbers below 1,000. The second sentence breaks the rule.

In the second sentence St. is followed by the capitalized name "Jones" and is read as "Saint", though the writer meant Louis Street. The period after St. ends both the abbreviation and the sentence, and a human sees that "Jones followed" starts a new one. A real classifier looks at more words, at capitalization statistics, and at a list of street names, and it still makes errors. That is why production systems let users add exceptions.

import re

ONES = ("zero one two three four five six seven eight nine ten eleven twelve thirteen "
        "fourteen fifteen sixteen seventeen eighteen nineteen").split()
TENS = "_ _ twenty thirty forty fifty sixty seventy eighty ninety".split()

def verbalize(text):
    """Written form to spoken form, following the rules in the task."""

From letters to phonemes

After normalization every token is a word, and the next question is how it sounds. English spelling is a poor guide. "Though", "through", "tough", "cough", and "bough" share four letters and have five different vowel sounds. "Colonel" has no r in its spelling and one in its sound, and "knight" and "night" sound the same.

A phoneme is the smallest unit of sound that distinguishes words in a language, as /k/ and /b/ distinguish "cat" from "bat". Speech technology for American English usually writes phonemes in ARPAbet, an alphabet of plain ASCII symbols. The CMU Pronouncing Dictionary (CMUdict) uses 39 ARPAbet phonemes and has about 135,000 entries, more than 126,000 words, some with several pronunciations. Every vowel carries a stress mark, 0 for none, 1 for primary, and 2 for secondary, so "cat" is K AE1 T. The International Phonetic Alphabet (IPA) is the standard for every language, with the same word written /kæt/, and multilingual systems use it or a phoneme set per language.

A G2P module has three layers.

• A lexicon first. Look the word up in a pronunciation dictionary. This covers the frequent words and is exactly right for them.
• A learned model for the rest. Names, new words, and misspellings are not in the lexicon, so a small sequence model trained on the lexicon predicts their phonemes. Its accuracy on unseen words is far below the lexicon's, which is why names are the classic G2P error.
• Context for heteronyms. A heteronym is a spelling with several pronunciations that depend on meaning. "Read" is R IY1 D in "I will read" and R EH1 D in "I read it yesterday". "Lead" is a verb or a metal, and "record" puts its stress on the first syllable as a noun and on the second as a verb. A part-of-speech tagger or a small classifier picks among the lexicon's entries.

The cell implements the first and third layers with a few entries copied from CMUdict and a crude context rule. In place of the learned model it uses naive letter-to-sound rules, to show how badly rules alone do on English.

The rules get "phone" half right and "colonel", "though", and "knight" wrong. That is why rule-only G2P died out for English, though it still works for languages with regular spelling such as Spanish or Finnish.

Should the acoustic model read characters or phonemes? It can read characters and learn pronunciation from the audio, as the first Tacotron did (Wang et al. 2017). Whether it should depends on data. With 24 hours of one speaker, many words appear only a handful of times, and a character model has to learn "ough" from very few examples. So small-data systems feed phonemes. With tens of thousands of hours, the model sees enough examples to learn spelling-to-sound itself, and the large systems of 2024 to 2026 read raw text. Two flow-matching systems we meet later read characters (and Chinese pinyin), and CosyVoice 2 reads the subword tokens of the language model it is built on (Du et al. 2024). Even these systems keep a way to override a pronunciation, because a customer's product name has to be said right the first time.

The length problem

Here is the problem every acoustic model must solve. The encoder gives one vector per symbol, $T_{\text{txt}}$ of them, and the target has $T_{\text{mel}}$ frames. On LJSpeech $T_{\text{mel}}/T_{\text{txt}} \approx 5.7$ for characters, and more for phonemes, since a sentence has fewer phonemes than letters, spaces, and punctuation marks. Reading is monotonic, which means the frames follow the text in order. Every symbol is said. So the frames split into consecutive runs, one run per symbol.

In the example, $a = (1, \dots, 1, 2, \dots, 2, 3, \dots, 3)$ with six 1s, fourteen 2s, and eight 3s, and $\ell = (6, 14, 8)$. Two notes on indices. From here on $i$ is a text symbol and $j$ a frame, except inside attention, where we keep the convention of §3.6 that $i$ is a decoder step and $j$ a text position. Each subsection says which. And $\ell_i$ here is a duration in frames, unrelated to the span length of §35.4.

Could a model try every alignment and keep the best? Counting them answers that.

For 3 symbols and 5 frames there are $\binom{4}{2} = 6$, with durations $(1,1,3)$, $(1,2,2)$, $(1,3,1)$, $(2,1,2)$, $(2,2,1)$, and $(3,1,1)$. For 100 phonemes and 600 frames there are $\binom{599}{99} \approx 1.85\times 10^{115}$, so trying them all is out of the question. Speech recognition met the same count in §36.1, with the roles reversed. There the frames are the input and the symbols the output. Here the frames are what the model must produce.

Every acoustic model answers the length problem in one of three ways, and the rest of the section takes them in turn.

1. Learn the alignment with attention. A decoder produces one frame at a time, looks back at the text at each step, and decides when to stop. The alignment is implicit and soft.
2. Predict a duration for every symbol. A small network predicts $\ell_i$, and each encoder vector is copied $\ell_i$ times, so the whole spectrogram can be produced in one parallel pass. The alignment is explicit and hard, and it needs training targets.
3. Choose only the total length. A generative model fills in a spectrogram with a given number of frames and works out the timing inside. The alignment is implicit again, but generation is parallel.

Answer one, learn the alignment with attention

Tacotron 2 is an encoder-decoder that writes frames

Recall the encoder-decoder of §3.5. An encoder reads the input into one vector per position, and a decoder writes the output one step at a time, feeding each output back in as the next input. With attention (§3.6), at every step the decoder scores every encoder vector, turns the scores into weights with a softmax, and reads their weighted average, the context. Tacotron 2 (Shen et al. 2018) is this design with one change that matters a great deal. Its outputs are not tokens drawn from a softmax but continuous 80-dimensional mel frames, one per decoder step. In this subsection $i$ is a decoder step, so the decoder produces frame $i$ at step $i$, and $j$ is a text position.

• Encoder. Each character becomes a learned 512-dimensional vector. Three convolution layers, each with 512 filters that see 5 neighboring characters, add local context. One bidirectional LSTM with 256 units in each direction then gives the encoder states $\mathbf{h}_j \in \R^{512}$.
• Decoder. At step $i$ the previous frame $\mathbf{x}_{i-1} \in \R^{80}$ (the recorded one during training, the model's own during generation) passes through a pre-net, two fully connected layers of 256 ReLU units with dropout 0.5. Two LSTM layers of 1,024 units read it with the previous context. Attention over the encoder states gives a new context $\mathbf{c}_i$, and a linear layer maps the LSTM output and $\mathbf{c}_i$ to the frame $\hat{\mathbf{x}}_i^{\text{pre}} \in \R^{80}$.
• Stop token. From the same inputs, a linear layer and a sigmoid give $p_i^{\text{stop}}$, the probability that the utterance is over. Generation ends at the first frame with $p_i^{\text{stop}} > 0.5$.
• Post-net. After all frames exist, five convolution layers predict a correction that is added to the whole spectrogram, $\hat{\mathbf{X}}^{\text{post}} = \hat{\mathbf{X}}^{\text{pre}} + \operatorname{PostNet}(\hat{\mathbf{X}}^{\text{pre}})$. Because it runs after decoding, the post-net can look at future frames, which the decoder cannot.

The loss is the squared error before and after the post-net (which the paper says aids convergence), plus a binary cross-entropy that teaches the stop token to fire on the last frame and only there,

In plain words, make every frame right twice, once straight out of the decoder and once after the post-net's correction, and learn to say "done" exactly on the last frame. Removing the post-net cost about 0.1 in mean opinion score (MOS, the average listener rating on a 1 to 5 scale) in the paper's ablation, 4.429 against 4.526. Training feeds the recorded previous frame (teacher forcing), while generation feeds the model's own. This mismatch is the exposure bias of §3.5, here with continuous outputs, so a small error in one frame shifts the input of the next.

Why squeeze the previous frame through a narrow pre-net with heavy dropout? The paper found the bottleneck "essential for learning attention". Consecutive mel frames are nearly identical, so a decoder that sees the previous frame clearly can predict the next one by copying it and never needs to read the text. A noisy, narrow view of the previous frame makes copying unreliable, and the cheapest way left to predict the frame is to attend. This explanation is practitioners' reasoning consistent with the paper's finding, not a theorem. The paper also keeps the pre-net's dropout on at inference "to introduce output variation", so the same text gives slightly different speech on each run.

Why squared error gives muffled spectrograms

The squared-error loss has a cost that every later TTS model tries to avoid. §32.1 proved that the predictor minimizing expected squared error is the conditional mean, the average of all the outputs that are possible given the input. Given the text and the frames so far, the next frame is not determined. The speaker could be a little higher or lower in pitch, a little early or late. So a network trained with squared error learns the average of those possibilities. The average of two voiced frames at slightly different pitches is a frame whose harmonics are smeared out.

Later TTS models answer this in two ways. Some keep squared error but give the decoder what the text leaves open, such as the pitch and loudness of the recording, so there is less to average (FastSpeech 2, below). The rest sample one plausible output instead of predicting the average. Flows (Glow-TTS, below) model the full distribution of frames. Adversarial losses (§37.2) penalize outputs a discriminator can tell from real ones. Flow matching (later in this section) and diffusion sample one spectrogram, and codec language models (§37.4) sample discrete tokens. An interviewer who asks "why does my TTS sound muffled?" wants to hear the conditional mean.

Location-sensitive attention

Plain attention does not know where it looked a moment ago, and that is its weak point for speech. The additive score of §3.6 is a small network that reads the decoder state $\mathbf{s}_{i-1}$ and one encoder state $\mathbf{h}_j$ together, $e_{i,j} = \mathbf{w}^\top\tanh(\mathbf{W}\mathbf{s}_{i-1} + \mathbf{V}\mathbf{h}_j + \mathbf{b})$ (Bahdanau et al. 2014). If the same word occurs twice, its two encoder states are similar, the two scores nearly tie, and the decoder can jump between them. Speech recognition met exactly this problem (§36.4). The fix there was to feed the previous attention weights into the score through a convolution (Chorowski et al. 2015). Tacotron 2 uses that mechanism with one change. It feeds the cumulative weights, the total attention each position has received so far.

In plain words, the scorer sees three things for each text position. It sees what the decoder wants ($\mathbf{W}\mathbf{s}_{i-1}$), what is written there ($\mathbf{V}\mathbf{h}_j$), and how much attention that position and its 30 neighbors have already received ($\mathbf{U}\mathbf{f}_{i,j}$). The sum $\mathbf{f}_{i,j}$ is what deep-learning libraries call a one-dimensional convolution (Conv1d) with zero padding $(r-1)/2$ on each side. Strictly it is a cross-correlation, since the kernel is not flipped.

Why does this help? Two filters show what the location term can express, and they are worth working out once by hand.

The cell runs a toy decoder that generates on its own, on a sentence in which "the", "cat", "sat", and "by" each occur more than once. The encoder states hold content only, so repeated words have identical vectors. The decoder's query is the vector of the word it believes it is saying, plus noise. It reads the word its attention picks, moves on once that word has lasted its planned number of frames, and stops after the last word. The content scores are multiplied by $\beta = 6$, which sets how sharply the query picks out matching words. We compare content alone (location weight $\lambda = 0$) with content plus the stay-or-move feature of the example ($\lambda = 4$).

With content alone, every run goes wrong. The first one jumps backward 13 times, never reads 8 of the 13 words, and is still talking when it hits the cap of 150 steps, against 59 planned frames. That is the skip, the repeat, and the failure to stop, all in one sentence. With the location feature, all eight runs read every word once, in order, and stop on time.

The widget runs the same toy with sliders for $\beta$ and $\lambda$, and adds two things. A stop probability, drawn under the grid, plays the part of Tacotron 2's stop token. It is the attention on the last word times the share of that word's frames already said, and the speech ends once it passes 0.5. A third slider, a diagonal prior, lowers each score by more the farther that word is from the dashed line, where a speaker reading at a steady pace would be. The guided attention loss below trains attention toward the same line.

import numpy as np

def location_attention(s_prev, H, alpha_cum, W, V, U, F_loc, w, mask):
    """One step of location-sensitive attention. Return (alpha (T,), context (d,))."""
    raise NotImplementedError

How attention fails, and the fixes

Location features make attention-based TTS usable, but not safe. It fails in three ways, and every one is audible.

• Skipping. Attention jumps forward past a word, and the word is never said.
• Repeating. Attention stays on a word too long or jumps back, and a phrase is said twice or loops.
• Never stopping. Attention never settles on the last symbol in a way that fires the stop token, and the decoder babbles until a hard cap. NVIDIA's implementation caps generation at 1,000 decoder steps, which at 86.13 frames per second is 11.6 s, just above LJSpeech's longest clip (10.10 s).

The failure rate grows with length for two reasons. Errors compound through exposure bias, and long inputs reach positions the model never saw in training, where clips are short (LJSpeech's are at most 10.1 s). The fixes, from mildest to most radical, follow.

In plain words, attention weight on the diagonal of the (frame, text) grid costs nothing, weight far from it costs up to 1 per unit, and $g$ sets how far is far. Reading speed is roughly constant, so the true alignment is near the diagonal, and the loss gives attention a strong hint early in training. Tachibana et al. report that with it the attention became roughly correct after about 5,000 iterations, against about 50,000 without.

• Repair at inference. Tachibana et al. also force attention forward when it misbehaves. If the most attended position moves by less than $-1$ or more than $+3$ in one step, they replace the weights by a one-hot vector on the previous position plus one.
• Monotonic attention. A family of mechanisms makes attention move forward by construction. They include forward attention (Zhang et al. 2018), stepwise monotonic attention (He et al. 2019), and location-relative mechanisms that drop the content comparison from the alignment step and were built for long-form text (Battenberg et al. 2020).
• Chunking. Split long text at sentence boundaries and synthesize each sentence separately. This bounds the damage, at the cost of prosody that cannot span sentences.
• Remove attention. Predict how long each symbol lasts and lay the frames out explicitly. That is the next answer.

Answer two, predict a duration for every symbol

The length regulator

FastSpeech (Ren et al. 2019) replaces the attention between text and frames with an explicit expansion. If you know each symbol's duration, you can build the frame sequence directly by copying. In this subsection $i$ is a text symbol and $j$ a frame.

In plain words, copy each symbol's vector once for every frame it should last. Position encodings (§5.4) are added after the expansion, so the copies of one vector are no longer identical and the decoder can tell the frames of a run apart. A stack of Transformer blocks with convolutions then maps $\mathbf{H}_{\text{mel}}$ to the spectrogram in one parallel pass, with no frame waiting for the previous one.

Libraries expose the same control under two names. Some take a length scale, our $\gamma_{\text{spd}}$. Others take a speaking rate, its reciprocal. Check which before you set it to 1.3.

import numpy as np

def length_regulate(H, durations, gamma=1.0):
    """H (T_txt, d), durations (T_txt,) nonnegative. Return (H_mel, a)."""
    raise NotImplementedError

The duration predictor

At inference the durations have to come from somewhere, so FastSpeech trains a small network to predict them from the encoder states. It has two one-dimensional convolution layers with ReLU, each followed by layer normalization and dropout, and a linear layer that outputs one number per symbol. It is trained with squared error on the logarithm of the duration, $\mathcal{L}_{\text{dur}} = \frac{1}{T_{\text{txt}}}\sum_i\big(\hat z_i - \log\ell_i\big)^2$, with $\hat z_i$ the predictor's output for symbol $i$. During training the length regulator uses the target durations, not the predicted ones, which is teacher forcing for durations. Many implementations regress $\log(\ell_i + 1)$ instead, so that a duration of 0 is allowed, and invert the prediction with $\exp(\hat z_i) - 1$ before rounding.

Why the log? Durations are positive and skewed, a few frames for a stop consonant and dozens for a stressed vowel before a pause, and what matters is relative error. A two-frame error is large on a three-frame /t/ and invisible on a forty-frame vowel, and the log makes both the same kind of error. The log also has a consequence that is easy to miss.

In plain words, a duration predictor trained in the log domain predicts typical durations, not average ones, and the typical duration is shorter. With a spread of $\sigma = 0.4$ in log duration, the predicted total is $e^{-0.08} = 0.923$ times the average total, so the speech runs about 8 percent fast unless someone rescales it. That is one reason duration-based systems expose a global $\gamma_{\text{spd}}$ and tune it by ear. The PyTorch block at the end of the section measures the effect on a trained predictor.

Where duration targets come from

The predictor needs a target $\ell_i$ for every symbol of every training utterance, and recordings do not come with phoneme boundaries. There are three ways to get them.

From a teacher's attention. FastSpeech first trains an attention-based Transformer TTS model, the teacher, whose attention between frames and phonemes is an alignment. It runs the teacher once more over each training pair, feeding it the recorded frames (teacher forcing), and reads the durations off one attention head. Not every head is diagonal, so it picks the one that attends most sharply.

FastSpeech also trains on the teacher's generated spectrograms rather than the recordings (sequence-level knowledge distillation, Kim and Rush 2016). In the paper's ablation, removing that step cost 0.325 in comparative mean opinion score (CMOS), where listeners rate pairs of samples side by side. The FastSpeech 2 paper explains why it helps. A teacher's outputs vary less than real speech, so the student has less to average, but they also lose some detail of the recordings. The cost is a two-stage pipeline and targets that inherit the teacher's mistakes. A head that skips a phoneme gives it duration 0, and a head that jumps back gives a path that goes backward even though its counts look fine.

From a forced aligner. FastSpeech 2 (Ren et al. 2021) replaces the teacher with the Montreal Forced Aligner (McAuliffe et al. 2017). It is a classical hidden Markov model recognizer, which FastSpeech 2 trained on its own TTS corpus, that runs Viterbi forced alignment (§36.1) of each recording against its known phoneme sequence and outputs a start and end time for every phoneme. Dividing by the hop gives durations in frames. The authors chose it for more accurate alignments, and dropping the teacher cut total training time by a factor of 3.12, from 53.12 to 17.02 hours (their Table 2).

From the model itself. Glow-TTS learns the alignment together with the acoustic model, with no teacher and no external aligner, by searching for the best monotonic alignment at every training step. That search is the most interesting algorithm in this section.

Monotonic alignment search

Glow-TTS (Kim et al. 2020) is a normalizing flow, an invertible network trained by exact likelihood (§33.6). Its decoder maps the whole mel spectrogram to a latent sequence of the same shape, one vector $\mathbf{z}_j$ per frame for $j = 1, \dots, T_{\text{mel}}$, and it can be run backward to turn latents into a spectrogram. The text encoder outputs a mean $\boldsymbol\mu_i$ and a standard deviation $\boldsymbol\sigma_i$ for every symbol. The model says that frame $j$'s latent is Gaussian with the statistics of its symbol $a(j)$. So a good alignment is one under which each frame's latent looks like a sample from its symbol's Gaussian. Write

The change-of-variables formula adds a term that does not depend on the alignment, so the alignment enters the likelihood only through the score $S(a) = \sum_{j=1}^{T_{\text{mel}}}\log p_{a(j),j}$. Training alternates two steps. With the parameters fixed, find the best monotonic alignment $a^\star = \arg\max_a S(a)$. With the alignment fixed, take a gradient step on the log-likelihood. This is Viterbi training, or hard EM, the same alternation that trained hidden Markov model recognizers (§36.1). It maximizes the likelihood of the single best alignment rather than the sum over all of them.

The first step looks impossible, with $\binom{T_{\text{mel}}-1}{T_{\text{txt}}-1}$ alignments to search, but a dynamic program finds the best one exactly. It works like the Viterbi segmentation that found the best tokenization in §2.4.

The idea in plain words comes first. To know the best way to be on symbol $i$ at frame $j$, you only need the best ways to have been on symbol $i$ or on symbol $i - 1$ one frame earlier, because those are the only two places a monotonic alignment can come from. So fill a table one frame at a time, keeping the best score for every (symbol, frame) cell, then walk back from the last symbol at the last frame.

The last statement is why the search needs $T_{\text{mel}} \ge T_{\text{txt}}$, at least one frame per symbol, and why implementations fill only the cells with $i \le j$. Glow-TTS reports that the search takes under 20 ms per training iteration on a CPU, less than 2 percent of training time.

import numpy as np

def mas(loglik):
    """loglik (T_txt, T_mel). Return (durations, best_score)."""
    raise NotImplementedError

The cell below writes the search with plain loops, line for line as in the theorem, and runs it in the situation inside Glow-TTS. Twenty symbols get random four-dimensional means, and frames are drawn around the mean of their planted symbol with noise of standard deviation $\sigma$. The scores are $\log p_{i,j} = -\tfrac12\lVert\mathbf{z}_j - \boldsymbol\mu_i\rVert^2$, a Gaussian log-density up to a constant. For comparison the cell also reports the greedy alternative, which gives each frame its most likely symbol.

At low noise the search recovers every frame. At $\sigma = 0.5$ it still gets 99 percent of frames right, with an average duration error of 0.1 frames. The greedy assignment is right on only 69 percent of frames and jumps backward 31 times, an alignment no reader could follow. At $\sigma = 2$ the frames barely carry information about their symbols and the search degrades, but its answer is always a valid alignment, with every symbol said once and in order. That guarantee, not accuracy alone, is why hard monotonic alignment fixed the skips and repeats of attention.

At inference there is no spectrogram, so there are no latents to align. Glow-TTS therefore trains a FastSpeech-style duration predictor on the searched durations, with a stop-gradient on its input so that it does not pull on the encoder. At synthesis it predicts durations, expands the prior, samples latents from it with a temperature (the prior's standard deviation scaled by 0.333 gave the best MOS), and runs the flow backward in one parallel pass.

Its mel generation took about 40 ms whatever the length, 15.7 times faster than Tacotron 2 on average, and it stayed robust on texts far longer than anything in training. Its MOS of 4.01 against Tacotron 2's 3.88 (both with the same vocoder, which scored 4.19 on resynthesized recordings) shows that nothing was lost in quality. VITS (Kim et al. 2021) keeps the search, adds a duration predictor that samples durations instead of regressing them, and trains the acoustic model and the vocoder together, which §37.2 describes.

import numpy as np

def durations_from_attention(A):
    """A (heads, T_mel, T_txt). Return (head, durations)."""
    A = np.asarray(A, dtype=float)
    heads, T_mel, T_txt = A.shape
    return 0, np.bincount(A[0].argmax(axis=1), minlength=T_txt)

Prosody, what the words leave open

Say "you're going" as a statement and then as a question. The phonemes are the same, and what changes is the pitch at the end. Stress the words of "I didn't say he stole it" one at a time and you get several sentences with different meanings. Everything the words do not fix is prosody.

One text has many valid prosodies, so prosody is where the averaging of squared error hurts most. A model that predicts the average pitch contour of every way a sentence could be said produces flat, monotone speech. FastSpeech 2 (Ren et al. 2021) calls this the one-to-many mapping problem and attacks it directly. During training it gives the decoder the true prosody of the recording it is reconstructing, so the decoder never has to guess it. It trains small predictors to supply the same quantities at inference.

These predictors sit between the encoder and the decoder, in a block the paper calls the variance adaptor, and all three have the duration predictor's architecture.

• Duration, from forced-aligner targets, in the log domain, feeding the length regulator.
• Pitch, the $F_0$ of every frame, extracted from the recording. The predictor learns the log of it, normalized per utterance. The decoder's input is $F_0$ quantized into 256 levels on a log scale, embedded, and added to the expanded sequence.
• Energy, $E_t$ for every frame, quantized uniformly into 256 levels, embedded, and added the same way. The predictor regresses the unquantized value with squared error.

The decoder sees the true values in training and the predicted ones at inference, the same teacher forcing as for durations. Because pitch and energy are explicit inputs, they can also be edited, by shifting the predicted contour up or scaling the energy, which gives a controllable voice. FastPitch (Łańcucki 2021) is a common simpler variant that predicts one average pitch value per input symbol instead of a contour per frame.

Estimating pitch

All of this needs $F_0$ for every frame of every training recording. A pitch tracker that makes octave errors teaches the model to sing. A voiced frame is nearly periodic, with period $T_0 = f_s/F_0$ samples, since voiced sounds are sums of harmonics of $F_0$ (§35.1). In this subsection $\tau$ is a lag in samples, not a temperature.

The oldest method looks for the lag at which the signal best matches a shifted copy of itself. The autocorrelation of a frame $x[0], \dots, x[N-1]$ is $r[\tau] = \sum_{n=0}^{N-1-\tau}x[n]\,x[n+\tau]$. For a periodic signal it peaks at $\tau = T_0$ and its multiples. The estimate is $f_s/\tau^\star$, with $\tau^\star$ the highest peak in a search range $[f_s/F_{0,\max},\ f_s/F_{0,\min}]$, something like 60 to 400 Hz for adult speech. It makes two kinds of mistake, called octave errors. Picking $T_0/2$ reports twice the true pitch, which happens when the second harmonic is strong. Picking $2T_0$ or $3T_0$ reports half or a third of it.

De Cheveigné and Kawahara (2002) fixed these errors one at a time, and their estimator, YIN, is still the default in many toolkits (librosa has yin and its probabilistic successor pyin).

Each step fixes a specific failure. The difference function is zero at every multiple of the period of an exactly periodic signal. Unlike the autocorrelation, it does not shrink at long lags, because it sums over the same window $W$ for every lag. But $d[0] = 0$, and $d$ is small at small lags for any smooth signal, so a plain minimum search would pick a lag near zero and report a pitch far too high. Dividing by the running mean fixes that, since $d'$ starts at 1 and stays near 1 until a real dip. It also makes $d'$ free of units, so the threshold means the same thing for a whisper and a shout. Taking the first dip under the threshold rather than the deepest prevents the "too low" errors, because the dips at $2T_0$ and $3T_0$ are often as deep as the one at $T_0$. Interpolation recovers the fraction of a sample by which the true period misses an integer lag.

The cell builds a 25 ms frame of a 100 Hz voice whose second harmonic is three times as strong as the fundamental, and compares the two methods.

The autocorrelation's peak at half the period is 11 percent higher than its peak at the full period. The strong second harmonic adds there, and the sum over $N - \tau$ terms shrinks with the lag. So it reports 202.5 Hz, the octave above. YIN compares the same 400-sample window with copies shifted by up to 200 samples. Its normalized difference at the half-period lag is $d'[80] = 0.26$, well above the threshold, while the true period gives 0, so YIN reports 100 Hz. Real trackers then smooth over time. TTS models predict $\log F_0$, since pitch is perceived roughly on a log scale, together with a voiced or unvoiced flag, since $F_0$ does not exist on unvoiced frames.

import numpy as np

def yin_f0(frame, fs, fmin=60, fmax=400, threshold=0.1):
    """F0 of one frame in hertz, or 0.0 if unvoiced."""
    raise NotImplementedError

Style without labels

Pitch and energy predictors control the prosody a model already chose. Asking for a speaking style (cheerful, whispered, newsreader) needs something more. Global style tokens learn styles without labels (Wang et al. 2018). A reference encoder summarizes a recording into one vector, which attends over a small bank of learned embeddings, the style tokens, and the weighted sum conditions the decoder. Trained only to reconstruct speech, the tokens come to stand for factors such as speed and pitch range, and on noisy found data, for noise and speaker. At inference you pick token weights, or play in a reference clip and copy its style to a new text. The systems of 2024 to 2026 go further. Codec language models take a few seconds of reference audio as a prompt and continue in its voice and manner (§37.4), and recent systems such as Qwen3-TTS also accept a written description of the voice and style to produce (Hu et al. 2026).

Answer three, generate the frames and let timing emerge

Explicit durations fixed robustness, at a cost in rhythm. A duration predictor trained in the log domain gives the typical length of every phoneme, which sounds even and slightly mechanical, and it needs alignment targets from somewhere. Flow matching offers a different deal. Treat the whole spectrogram as a sample from a distribution conditioned on the text, and let the model decide internally where each sound goes.

Recall flow matching from §33.6. Draw a straight line from a noise sample to a data point. A network $\mathbf{v}_\theta(\mathbf{x}, t)$ learns the velocity along such lines, the direction and speed from noise toward data at each point and time. New samples are made by starting from fresh noise and integrating $\mathrm{d}\mathbf{x}/\mathrm{d}t = \mathbf{v}_\theta$ in a few dozen small steps. Chapter 33 runs time from data at $t = 0$ to noise at $t = 1$. The TTS papers run it the other way, and we write their convention here so that you can read their equations directly.

The trick that makes flow matching a TTS model is infilling. Hide part of a recording, show the model the rest and the full transcript, and train it to generate the hidden part.

In plain words, hide most of a recording, give the model the rest and the full transcript, and teach it to move noise toward the hidden part. Generation plays the same game with the hidden part placed after a reference recording, so the model continues in that voice. That is zero-shot voice cloning, a new voice from a few seconds of audio with no training on it. Every solver step processes all frames at once, so a 10 s utterance costs as many network passes as there are solver steps, 16 or 32 (twice that with the guidance described below), rather than one per frame.

The three main systems differ in how much timing they still supply.

• Voicebox (Le et al. 2023) still uses durations. It conditions on a phone sequence repeated to frame level, aligned by the Montreal Forced Aligner for training and expanded by a second, smaller duration model at inference. It works on 80-band log mels at 100 frames per second, masks the whole utterance with probability 0.3 and otherwise a contiguous span of 70 to 100 percent, and samples with a midpoint solver at 32 function evaluations. Trained on 60,000 hours of English audiobooks, it beat the codec language model VALL-E on zero-shot TTS, with a word error rate of 1.9 percent against 5.9 and speaker similarity 0.681 against 0.580, while being up to 20 times faster.
• E2 TTS (Eskimez et al. 2024) drops the phones and the durations. It pads the character sequence with filler tokens to the length of the spectrogram and lets the network work out which frames belong to which characters. It needs no duration model, no G2P, and no alignment search, and it matched or beat Voicebox.
• F5-TTS (Chen et al. 2024) keeps E2's character input but first refines it with a few convolution blocks before joining it to the audio. Its backbone is a diffusion Transformer with adaptive layer norm (§34.2). It reads 100-band log mels at 24 kHz with hop 256 (93.75 frames per second), and it was trained on about 95,000 hours of English and Chinese from the Emilia dataset.

Removing durations does not remove the length question. It shrinks it to one number per utterance, the total number of frames, which must be chosen before sampling. F5-TTS estimates it from the reference, assuming the target is read at the same rate in characters per frame.

With the length fixed, two sampling choices set the speed and quality of F5-TTS. Guidance strengthens the pull of the text and the reference at every step, at the price of a second network pass. The step schedule decides where on the path from noise to data the solver spends its steps. The box gives both, and the derivation after it shows which schedules are valid.

With $s_{\text{sway}} = -1$, nine of sixteen steps start in the noisiest 30 percent of the path, against five for uniform steps, and the first step is 13 times shorter. With 16 function evaluations F5-TTS reaches a real-time factor of 0.15, meaning 1.5 s of compute for 10 s of speech on one RTX 3090, and a word error rate of 2.53 percent on LibriSpeech-PC test-clean. With 32 evaluations it reaches 2.42 percent at a real-time factor of 0.31. Matcha-TTS (Mehta et al. 2024) is the small-data member of the family. It puts a flow-matching decoder on top of an encoder and a duration predictor, learns its alignment without an external aligner, and needs few synthesis steps.

Choosing among the three answers

The table is why many 2025 and 2026 production systems are hybrids. A language model over speech tokens decides content and timing one token at a time, the way Tacotron did but over far shorter sequences. A flow-matching stage like F5's then turns its tokens into a mel spectrogram (§37.4). So the timing decision did not go away. It moved into the language model, which decides how many tokens each word gets.

Speed has a structural reason too. For the 76.3 s answer of the opening, Tacotron 2 needs about 6,100 sequential decoder steps, each waiting for the previous frame, while FastSpeech needs one parallel pass and F5-TTS 16 or 32, doubled by guidance. On a GPU sequential steps are the expensive kind, since each one reads all the weights from memory to do a small amount of arithmetic (§20.1).

In PyTorch

The block builds FastSpeech's duration predictor and length regulator as modules. It trains the predictor on synthetic durations that depend on each symbol and its left neighbor, plus lognormal noise of spread 0.4, and checks the geometric-mean theorem. torch.repeat_interleave is the length regulator in one call.

The predicted total comes out several percent short of the true total, close to the $e^{-\sigma^2/2}$ of the theorem. The match is not exact, because the block regresses $\log(1+\ell)$ of rounded durations rather than $\log\ell$. A deployed system corrects that shortening with its global duration scale.

In practice you rarely build a front end or an acoustic model from scratch. The block below, not run here, phonemizes with eSpeak through the phonemizer package and synthesizes with a pretrained VITS model from Hugging Face (Meta's MMS English voice), whose speaking_rate is the reciprocal of our $\gamma_{\text{spd}}$.

The text passed to the model is already verbalized. A character model trained on spelled-out text has no reading for $1,200, which is the opening failure of this section.

Where TTS breaks

Read more

• Sproat & Jaitly (2016). RNN Approaches to Text Normalization: A Challenge. The data set and the argument for why text normalization errors must never change the message, with the neural models' telling mistakes.
• Shen et al. (2018). Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. Tacotron 2, with the architecture, the stop token, the MOS table, and an honest error analysis.
• Chorowski et al. (2015). Attention-Based Models for Speech Recognition. Location-aware attention and why content-only attention fails on long inputs.
• Ren et al. (2019). FastSpeech: Fast, Robust and Controllable Text to Speech. The length regulator, durations from teacher attention, and the robustness test on hard sentences.
• Ren et al. (2021). FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. Forced-aligner durations and the variance adaptor for pitch and energy.
• Kim et al. (2020). Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search. Monotonic alignment search, Algorithm 1, and Viterbi training inside a flow.
• de Cheveigné & Kawahara (2002). YIN, a fundamental frequency estimator for speech and music, Journal of the Acoustical Society of America 111(4), doi 10.1121/1.1458024. Each step of YIN and its effect on gross errors, a model of how to build an algorithm by fixing failures one at a time.
• Chen et al. (2024). F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. Flow-matching TTS without durations or phonemes, the total-length estimate, and sway sampling.

Watch next

▶

▶

▶