How a stream of spectrogram frames becomes words. The alignment problem and how CTC, transducers, and attention each solve it, Whisper and the speech-LLMs that lead the 2026 leaderboards, language models at decoding time, honest word error rates, and the streaming systems that transcribe while you talk, derived from first principles and taken to interview depth.

Speech recognition turns a recording of someone talking into the words they said. It runs live captions, dictation, voice assistants, and meeting notes, and many spoken interfaces to language models start with it. The hard part is not recognizing each sound. It is that nothing tells the recognizer which stretch of audio belongs to which word, and this section is about that problem and the tools every recognizer brings to it.

This chapter builds recognizers on the frame vectors of Chapter 35, one vector for each short slice of audio. Some are computed straight from the sound, as the energy in each band of frequencies (§35.3). Others come from an encoder pretrained on unlabeled speech, such as wav2vec 2.0 (§35.4). The chapter runs from the classical recognizer of the 1980s to the speech-LLMs of 2026, recognizers that hand an encoder's frames to a large language model (§36.4).

Here is the problem at real size. Say "she had your dark suit in greasy wash water all year" out loud. Every speaker in the TIMIT corpus reads this sentence (Garofolo et al. 1993), and it takes about three seconds.

A recognizer does not read the sound wave directly. It reads a log-mel spectrogram. The audio is cut into short overlapping windows, one every 10 ms. Each window becomes a row of 80 numbers, the log of the energy in 80 frequency bands spaced the way the ear spaces pitch. Each row is called a frame, and §35.3 builds them step by step. Three seconds become 300 frames of 80 numbers. The transcript has 11 words, or 52 characters with the spaces.

The recognizer must read the 300 frames and write the 11 words, and nothing says which frames belong to which word. The training data does not say either. Nearly every speech corpus pairs a recording with its transcript and nothing more, because marking word boundaries by hand takes far longer than typing the words. Deciding which frames go with which word is called alignment. It is the central problem of this chapter.

How many ways are there to cut 300 frames into 11 consecutive pieces, one per word? There are 1,351,709,558,800,311,711, about $1.35\times10^{18}$. For the 52 characters there are about $1.3\times10^{58}$. Nobody can try them one by one. Every recognizer ever built handles this set without listing it. It adds up the probability of all the cuts, or it searches for the best cut, or it lets attention find a soft version of one.

This section does five things. It counts the alignments exactly. It builds the classical recognizer, which ran nearly every deployed system until about 2015 and whose ideas are still everywhere. It builds the encoder network that modern recognizers share. It defines word error rate, the number every recognizer is judged by, and its speed companion. And it follows one clip through the whole pipeline.

The rest of the chapter builds on this section. §36.2 sums over alignments with a blank symbol. §36.3 lets each output depend on the ones before it. §36.4 lets attention find the alignment. §36.5 adds text knowledge at decoding time and makes evaluation honest. And §36.6 runs it all while the person is still talking.

The cell shows what a recognizer sees. It builds a synthetic spectrogram for the sentence. Each word gets its own smooth spectral shape and a random duration. Neighboring frames are blurred together, the way neighboring sounds blend, and noise goes on top.

Try to find the ten boundaries in the picture before you read the printed list. Some are easy, where the spectral shape jumps. Others hide in the blur, or sit between two words that happen to look alike. The durations are random, so "greasy" gets only 5 frames here and "dark" gets 61. Real speech is harder still, because a real word has no single spectral shape.

Why speech is hard

Four facts make speech recognition more than labeling each frame. Each one shapes every system in this chapter.

Speech has no spaces. Words run into each other with no silence between them, so finding the word boundaries is part of the job. Pauses do happen, but mostly between phrases. Short silences also occur inside words. The lips close for a moment before the p in "speech", for example.

Durations vary. A word can last twice as long when it is stressed, spoken slowly, or followed by a pause. Short words such as "in" and "a" can last under 50 ms. A phone is one of the basic sounds of a language, such as the "s" or the "ee" in "speech", and English has about 40 of them. So there is no fixed number of frames per word, per character, or per phone.

Sounds blend and vary. The mouth moves continuously, so each sound is colored by the sounds around it. This is called coarticulation. The same word also sounds different across speakers, accents, microphones, and rooms. So a phone is a whole distribution of spectral shapes, not one template.

Many word sequences sound alike. The classic pair is "recognize speech" and "wreck a nice beach". The CMU Pronouncing Dictionary writes the first as the phones R EH K AH G N AY Z S P IY CH and the second as R EH K AH N AY S B IY CH. The two strings are nearly the same, so the sound alone cannot decide. What decides is knowing which word sequences people actually say. That is the job of a language model, a probability distribution over word sequences (§3.1). Statistical language models became an engineering tool in speech recognition first, in Frederick Jelinek's group at IBM in the 1970s (Jelinek 1976).

So a recognizer must find the boundaries itself, accept any duration, tolerate variable sounds, and lean on a language model. An interviewer who asks why speech recognition is not frame classification wants these four reasons and one more. The frame labels are never given.

Counting the alignments

Start small. Take 5 frames and two labels, "h" then "i". The first frame must belong to "h" and the last to "i". Every label needs at least one frame, and the labels keep their order. So the only choice is where "h" ends, after frame 1, 2, 3, or 4.

h i i i i
h h i i i
h h h i i
h h h h i

That makes 4 alignments. Now take 6 frames and 3 labels. You choose where the first two labels end. That is a choice of two of the five gaps between frames, so there are $\binom{5}{2} = 10$ alignments.

To count in general we need the task in symbols. The input is a list of frames. The output is a much shorter list of labels whose length is not known in advance.

Neural recognizers first shrink the $T_{\text{in}}$ feature frames to $T$ encoder frames, as we will see below, and the alignment problem lives on those $T$ frames. Until then, $T$ is just the number of frames the model reads.

An alignment says which label each frame carries. We want alignments that keep the labels in order and give every label at least one frame.

In plain words, frame $t$ is assigned label number $g(t)$. Nondecreasing, or monotone, means the labels come in the same order as the audio. That holds for transcription and fails for translation, where word order changes. Surjection means every label gets at least one frame.

Check it on the examples. Five frames and two labels give $\binom{4}{1} = 4$, and six frames and three labels give $\binom{5}{2} = 10$. A common slip is to allow empty segments, as if a label could take zero frames. That count is $\binom{T+U-1}{U-1}$, which is larger and wrong here.

The count also has a recursive form, and that recursion is the seed of every algorithm in this chapter. Let $N(t, u)$ be the number of alignments of the first $t$ frames to the first $u$ labels. Frame $t$ carries label $u$. Either frame $t-1$ carried label $u$ too, so the label continues. Or frame $t-1$ ended label $u-1$, so label $u$ starts at frame $t$. So

In plain words, the number of ways to reach a cell of the table is the sum of the ways to reach the two cells it can come from. This is the rule that builds Pascal's triangle, and it fills a $T\times U$ table in $O(TU)$ additions. Replace "count the paths" by "add up the probabilities of the paths" and you get the forward algorithm. Replace it by "find the best path" and you get the Viterbi algorithm. Both appear below, and every later section of this chapter runs one of them.

The cell lists the alignments of the tiny example, checks the formula against brute force, and checks the recursion at real scale.

For a fixed number of labels the count is a polynomial in $T$ of degree $U - 1$. With five labels it grows like $T^4/4!$, not exponentially. For a fixed number of frames it is largest when $U - 1$ is about $(T - 1)/2$, the middle of a row of Pascal's triangle, where it comes within a factor of order $\sqrt{T}$ of $2^{T-1}$.

Real transcripts sit far from that middle. Conversational English runs at about 150 words a minute, which is about 2.5 words and 15 characters per second, or one character every 7 frames. So the counts are huge, but not as huge as they could be. The widget's toggle previews §36.2, where a frame may also carry a blank symbol that belongs to no label, which multiplies the number of paths again.

def n_alignments(T, U):
    raise NotImplementedError

In an interview, the count is the opening move. You state $\binom{T-1}{U-1}$, show it is far too large to list, and write the recursion that replaces the list. Every method in the chapter is that recursion with different numbers in the cells.

The classical recognizer

From the 1980s until about 2015, almost every deployed recognizer was built the same way. It has a decision rule and three models, a language model, a pronunciation dictionary, and an acoustic model made of hidden Markov models. The rest of the chapter keeps borrowing from it, so we build it piece by piece.

The noisy channel and the Bayes decision rule

Picture the speaker choosing a word sequence $W = (w_1, \dots, w_{|W|})$, and an acoustic channel (the vocal tract, the room, the microphone) turning it into the frames $\mathbf{X}$. The recognizer sees only $\mathbf{X}$ and must guess $W$. The natural guess is the word sequence that is most probable given what was heard. This noisy-channel picture of speech came from Jelinek's group at IBM (Jelinek 1976), and it leads to one formula.

In plain words, pick the word sequence that best explains the sound, weighted by how likely that sequence is to be said at all. The two factors have names. $p(\mathbf{X} \mid W)$ is the acoustic model, and $P(W)$ is the language model. The split lets each be trained on its own data, the acoustic model on transcribed audio and the language model on text, of which there is vastly more.

The rule minimizes sentence error. It does not minimize the number of wrong words, and three candidates are enough to show the difference.

Choosing the output that minimizes the expected number of word errors is called minimum Bayes risk decoding. §3.5 defines it for translation, and §36.5 specializes it to word errors. Interviewers like this point, because it separates people who know the formula from people who know what it optimizes.

Three models that work together

The language model. By the chain rule of probability, $P(W) = \prod_i P(w_i \mid w_1, \dots, w_{i-1})$. Classical systems approximated each factor with an n-gram model, which predicts a word from only the $n-1$ words before it, $P(w_i \mid w_{i-n+1}, \dots, w_{i-1})$. The probabilities come from counts in a large text corpus. Smoothing moves some probability to word sequences the corpus never contained. The standard method credits a word by how many different contexts it completes, which is called Kneser-Ney smoothing (§3.1). Trigram and 4-gram models trained on hundreds of millions to billions of words were typical.

The pronunciation lexicon. Most words appear rarely in transcribed audio, so words are not modeled directly. Instead a dictionary maps each word to a sequence of phones. The CMU Pronouncing Dictionary maps "speech" to S P IY CH and "suit" to S UW T. It has about 135,000 entries for some 126,000 words of North American English, written in a 39-phone alphabet called ARPAbet, with stress marks on the vowels. A word with several pronunciations gets several entries. A new word needs only a new dictionary line, not new audio.

The acoustic model. Each phone is modeled by a small hidden Markov model, defined next. The models of a word's phones are chained into a word model, and word models are chained into a sentence model. Real systems also give each phone a separate model for each pair of neighbors, because coarticulation changes it, and the box below says how.

Hidden Markov models

A hidden Markov model, or HMM, is a machine that moves through hidden states, one step per frame, and emits one frame from each state it visits. You see the frames, never the states. For speech, the states are parts of phones. So a path through the states is exactly an alignment of frames to phone parts, and everything we counted above becomes something we can score.

In plain words, the probability of the frames along one path is a product of one start probability, one transition per frame, and one emission per frame. The probability of the frames is the sum of that product over every path. Rabiner's tutorial, the classic reference, writes $a_{ij}$ for the transitions and $\pi_j$ for the initial probabilities. We keep its $b_j$ but write $P_{ij}$ and $P_{0j}$, because this book uses $a_{ij}$ for attention weights and $\pi$ for policies and permutations.

A word model for "suit" chains the 3-state models of S, UW, and T into a 9-state left-to-right model. Suppose a state stays put with probability $P_{jj} = q$. Then the number of frames it lasts is geometric with mean $1/(1-q)$. With $q = 0.8$ that is 5 frames, or 50 ms. This is how an HMM absorbs variable durations. The stay and advance moves are the "continue" and "start" of the counting recursion.

Forward and Viterbi on an HMM

Two questions about an HMM come up constantly. How probable are the frames, summed over every state path? And which single path is most probable? Both answers are the counting recursion again, with probabilities in place of counts. The forward algorithm adds up the probabilities of the paths into each state, and the Viterbi algorithm keeps only the best one. §2.4 ran the same two algorithms on a table of character positions to cut words into tokens. Here the table has frames down one side and states along the other.

In plain words, to be in state $j$ at frame $t$ you must have been in some state $i$ at frame $t-1$. The forward algorithm adds up the ways in, and Viterbi keeps the best way in. Each way in is weighted by its transition, and then multiplied by the emission of the current frame. $\operatorname{LSE}$ is the log-sum-exp, $\log\sum_i e^{a_i}$, which adds probabilities that are stored as logs.

Log space is not optional. A 39-dimensional Gaussian with unit variances has density about $e^{-55} \approx 10^{-24}$ at a typical point. After about 13 frames the product drops below the smallest normal double-precision number, about $10^{-308}$, and soon after it becomes exactly zero. Every implementation in this chapter adds logs and uses the stable log-sum-exp of §0.4, which subtracts the largest entry before exponentiating.

The cell runs both algorithms in log space and reproduces the hand computation.

Forced alignment, and training without frame labels

When the transcript is known, the HMM for the whole utterance is fixed. It is the chain of the states of its words' phones. Viterbi through that one chain gives the best state for every frame, and so the start and end of every phone and word. This is forced alignment. It answers the question of the opening, which frames belong to which word.

Forced alignment computed phone and word timestamps for decades, and it still does in many places. The Montreal Forced Aligner, a widely used open tool, is built on the Kaldi speech toolkit. Even the 2026 Qwen3-ASR release ships a separate forced-alignment model for timestamps (Qwen Team 2026).

Training has a chicken-and-egg problem. To estimate the emission densities you need to know which frames belong to which state. To know that, you need the emission densities. The classical answer is expectation maximization, or EM (§2.4). It alternates between guessing the hidden assignments with the current model and re-fitting the model to the guess. The cheap version for speech is Viterbi training. Force-align with the current model, treat the resulting frame labels as truth, re-estimate the densities and transitions from them, and repeat. It begins from a flat start, which gives every state an equal share of each utterance's frames.

How good is a forced alignment? The cell simulates utterances of 3 phones with 3 states each, 9 states in all. Each state lasts 2 to 7 frames at random. Emissions are 2-dimensional Gaussians whose means sit on a circle. Neighboring means are a chosen distance apart, the gap, measured in units of the noise standard deviation. The cell force-aligns each utterance with the true model and measures how far the two recovered phone boundaries fall from the true ones.

With neighboring states 4 standard deviations apart, 99 percent of boundaries land within one frame. At 2 they are within a frame 94 percent of the time, and at 1 not quite two thirds of the time. At 0.5 the error averages 2.7 frames. The alignment degrades gradually, never catastrophically, because the order constraint keeps every boundary between its neighbors. That is why forced alignment works even with a weak acoustic model, and why a flat start can get training going.

One detail of this setup surprises people.

import numpy as np

def force_align(log_emit, log_stay, log_next):
    raise NotImplementedError

From Gaussian mixtures to neural networks

For about three decades, each tied state's emission density $b_j(\mathbf{x})$ was a Gaussian mixture, a weighted sum of Gaussians. It was fit to 39-dimensional frames such as the MFCCs of §35.3. An MFCC frame is 13 cepstral coefficients, a compressed summary of a log-mel frame, plus their first and second differences across neighboring frames, 39 numbers in all. The mixtures were trained by EM. They were then refined discriminatively, which means trained to score the correct transcript above competing ones, not only to fit the data.

Around 2010 neural networks replaced the mixtures. A network reads a window of neighboring frames and outputs a softmax over the tied states. It is trained with cross-entropy on frame labels from a forced alignment. But the network gives a posterior $P(s_t = j \mid \mathbf{x}_t)$, the probability of the state given the frame. (Here $\mathbf{x}_t$ stands for the window around frame $t$.) The HMM needs a likelihood $b_j(\mathbf{x}_t)$, the probability of the frame given the state. Bayes rule converts one into the other.

In plain words, divide what the network believes by how common the state is, and you get a number the HMM can use as a likelihood. The division matters most when the priors are very unequal. Silence states fill a large share of training frames, so their posteriors are large partly because silence is common. Without the division the decoder over-predicts silence.

Decoding with a search graph

The decoder must search every word sequence, every pronunciation, and every state path at once. Classical systems compile all three models into one large graph ahead of time. Decoding is then a Viterbi search through that graph, frame by frame. At each frame it keeps only the best few thousand partial paths and drops the rest, which is called beam pruning. The search can also keep a lattice, a compact graph of the competing hypotheses, so a stronger language model can rescore them later.

Why the language model needs a weight

Plug the two models into the Bayes rule as they are, and the language model barely affects the result. So every classical system instead maximizes

Here $|W|$ is the number of words. $\lambda_{\text{LM}}$ is the language model weight, typically around 10. $\lambda_{\text{len}}$ is a reward per word, and its negative is called the word insertion penalty. Kaldi's documentation scales the acoustic score by 0.1 in most of its examples. That is the same as an LM weight of 10, since multiplying the whole score by 10 does not change the arg max. Why should a correct Bayes rule need a factor of 10?

Because the acoustic model is wrong in a specific way. Frames are 25 ms windows taken every 10 ms, so each audio sample sits in two or three consecutive frames. Speech also changes slowly, so neighboring frames are strongly correlated. The HMM scores them as independent, so it counts the same evidence several times. The language model counts its evidence once.

Here is the simplest version. Suppose every frame appeared three times in a row. The acoustic log-likelihood would triple, and the language model would need a weight of 3 to keep up. The theorem makes this precise and handles partial redundancy too.

In plain words, the LM weight is the number of frames the acoustic model counts for each independent piece of evidence. Too small a weight lets the overcounted audio shout down the language model. Too large a weight lets the language model shout down the audio.

The cell checks the theorem on a two-word decision. Each utterance has 4 blocks of 3 frames. The frame mean is 0 for the first word and 0.5 for the second, and the first word has prior 0.8. The cell sweeps the weight and records how often the decoder picks the word that was spoken, its sentence accuracy.

The accuracy peaks at weight 3 for exact copies, at 2 for $\rho = 0.5$ (the theorem's $1 + 0.5\cdot2$), and at 1 for independent frames. At the best weight, exact copies reach 0.815. One frame per block, with no copies, reaches 0.814 by the exact formula of Exercise 36.1f, the same up to simulation noise. That is because copies carry no new information. The right weight undoes the overcounting, but it cannot create evidence the copies never held. Independent frames carry more evidence and reach 0.866.

Real frames are only partly redundant, and the weight also absorbs other ways the acoustic model is miscalibrated. So in practice $\lambda_{\text{LM}}$ is tuned on held-out data rather than derived. The length reward fixes a second bias. Every word multiplies $P(W)$ by a conditional probability below 1, so the language model favors fewer words, and scaling it by 10 makes each word ten times as expensive. The decoder then drops short words and merges others. A positive $\lambda_{\text{len}}$, tuned together with $\lambda_{\text{LM}}$, gives back a fixed amount per word. Both weights return in §36.5. There an external language model is added to a neural recognizer by the same weighted sum, which is called shallow fusion.

import numpy as np
from scipy.stats import norm

def sentence_accuracy(lam, k, rho, m, mu, p1):
    raise NotImplementedError

def best_lm_weight(k, rho, m, mu, p1):
    raise NotImplementedError

An interviewer who asks why the LM weight is about 10 wants the overcounting argument. The frames overlap and are correlated, the acoustic model treats them as independent, and the weight undoes the overcount. The insertion penalty is the usual follow-up.

End-to-end recognizers

The classical system has many parts, each trained separately. Two of them are expensive to build, a pronunciation lexicon written by linguists and frame-level state labels from a bootstrapped aligner. An end-to-end recognizer replaces the whole stack with one network. It models $p_\theta(\mathbf{y} \mid \mathbf{X})$ directly over characters or subword tokens and is trained on (audio, transcript) pairs alone. The alignment problem does not go away. Each family of end-to-end models handles it differently, and the rest of the chapter takes them in turn. The table names them, and the sections it links define each one.

Conditionally independent means that, given the audio, the output at one frame does not depend on the outputs at the others. Streams means the model can write text while the audio is still arriving.

End-to-end models won for three reasons. There is one network and one loss, so no two parts have to agree with each other. No lexicon or frame labels are needed, so a new language needs only transcribed audio. And they keep improving with more data and parameters, which OpenAI's Whisper recognizer (§36.4) pushed to 680,000 hours of audio.

The classical system still teaches three lessons this chapter uses. Bayes rule is how an external language model is combined with an end-to-end model (§36.5). Forced alignment, now done with CTC, is how word timestamps are produced (§36.2). And finite-state graphs are still how many production systems bias recognition toward contact names and product terms.

The shared encoder

Every end-to-end family puts the same kind of network in front. This encoder turns the $T_{\text{in}}$ feature frames into $T$ vectors, one per encoder frame. Many recognizers start the encoder from self-supervised training, where a network learns from unlabeled audio by predicting masked parts of it (§35.4 and §35.5). Two decisions define the encoder, how much it shortens the sequence and what its blocks look like.

Subsampling and frame rates

Feature frames come 100 per second. That is more than a recognizer needs, since English speech has about 15 characters and 2.5 words per second. It is also expensive, because attention's cost grows with the square of the length (§4.7). So the encoder starts with a few strided convolutions. A strided convolution slides a small filter along time and keeps only every $s$-th output, which shortens the sequence by about a factor of $s$. The exact length follows the output-size rule of §0.8,

for an input of $n$ frames, kernel width $k$, stride $s$, and padding $p$ (zeros added at each end).

Conformer and Transformer recognizers in the common open toolkits, such as ESPnet and NeMo, use two convolutions of kernel 3 and stride 2. (The Conformer paper itself says only that its convolutional front end subsamples by 4.) That is $4\times$ subsampling, with encoder frames of $\Delta_{\text{enc}} = 40$ ms. With padding 1, 3,000 frames (30 s) become 1,500 and then 750. Without padding they become $\lfloor 2997/2\rfloor + 1 = 1{,}499$ and then 749. That one-frame difference is the off-by-one behind many length bookkeeping bugs.

FastConformer (Rekesh et al. 2023) uses three stride-2 layers for $8\times$ subsampling, which gives 80 ms frames, 12.5 per second. Its second and third layers are depthwise separable, a cheap kind of convolution described with the Conformer block below. Halving the length cuts the part of every attention layer's cost that grows with the square of the length by a factor of 4, compared with $4\times$ subsampling. Together with smaller convolution kernels inside the blocks (9 instead of the 31 of NeMo's Conformer), the paper reports an encoder 2.8 times faster than the original Conformer.

The catch is that $8\times$ leaves few frames per label. CTC needs at least one frame per output label. English speech runs at about 15 characters per second, more than the 12.5 frames per second that remain. The FastConformer paper notes that most training utterances of LibriSpeech, the standard benchmark of read audiobooks, would fail that condition with character tokens after $8\times$ subsampling. So its models use subword vocabularies instead, 128 tokens for its LibriSpeech CTC models and 1,024 for its transducers, and §36.2 derives the exact condition.

Whisper uses a kernel-3 convolution with stride 1 and a kernel-3 convolution with stride 2, both padded, for $2\times$. Its 30-second window of 3,000 frames becomes 1,500 encoder frames of 20 ms (§36.4).

The trend in 2025 and 2026 is toward 12.5 Hz. Qwen3-ASR's encoder downsamples 128-band features by 8 to 12.5 Hz (Qwen Team 2026). Voxtral Realtime runs its encoder at 50 Hz and downsamples by 4 before its language decoder, again to 12.5 Hz (Mistral AI 2026). The reason is cost. When the decoder is a large language model, every audio frame is a token in its context. A 20-minute recording at 12.5 Hz is $1{,}200\times12.5 = 15{,}000$ tokens, where 50 Hz would be 60,000. Qwen3-ASR accepts single recordings of up to 20 minutes. The opposing pressure is the frames-per-label budget that CTC and transducers need, which is why lower frame rates go with larger subword vocabularies.

The Conformer block

Self-attention lets every frame look at the whole utterance at once (§4.3). That suits long-range context such as speaking rate, speaker, and topic. But much of speech is local. The change in resonances that tells "b" from "d" lasts a few tens of milliseconds, and attention has no built-in preference for neighboring frames. A convolution has exactly that preference and no global view. The Conformer (Gulati et al. 2020) puts both in every block, and it or a close variant is the encoder of many recognizers in this chapter.

In plain words, each block does half a feed-forward step, lets every frame look at every other frame, mixes each frame with its near neighbors, and does the other half of the feed-forward step. Three terms in the definition need a word. Relative positions make an attention score depend on how far apart two frames are, not on where they sit in the utterance. The Conformer uses the version of Dai et al. (2019), which adds a learned projection of sinusoidal offset encodings and two learned bias vectors. A depthwise convolution filters each channel separately, so it costs $k_{\text{conv}}d$ weights instead of $k_{\text{conv}}d^2$. And BatchNorm normalizes each channel with statistics over the batch, unlike LayerNorm, which normalizes each frame over its channels.

Four facts about the design come up in interviews. The paper calls the convolution module the most important part of the block. The two half-step feed-forward modules beat one full one of the same size. The convolution sits after the attention. And many implementations, NVIDIA's NeMo toolkit among them, use $k_{\text{conv}} = 31$, because an odd kernel can be centered with equal padding on both sides. The box gives the reasons and the numbers.

How many parameters does a block have? Count module by module, as §5.6 does for a Transformer block. Every linear and convolutional layer has a bias, and every normalization has a gain and a shift. The relative-position attention adds a $d\times d$ projection of the position encodings (no bias) and two per-head bias vectors, together $2d$ numbers.

The PyTorch block below builds the Conformer block exactly as defined, with relative-position attention written out, and counts its parameters. It is self-contained, so it repeats a few definitions.

def encoder_frames(T_in, layers):
    raise NotImplementedError

def conformer_block_params(d, k_conv, ff_mult=4, rel_pos=True):
    raise NotImplementedError

In a design review or an interview, the encoder questions are about rates and sizes. How many frames does a clip become, how many labels must fit in them, and where do the parameters sit? The answers are the output-size rule, the frame rates above, and $24d^2 + (32 + k_{\text{conv}})d$ per block.

Word error rate

A recognizer outputs a word sequence, called the hypothesis. A person's transcript, the reference, says what was actually said. How wrong is the hypothesis? Counting the positions where the two differ fails at once, because one missing word shifts everything after it. The right count is the smallest number of word edits that turns the reference into the hypothesis.

The word "alignment" here pairs words with words, not frames with labels, but the idea is the same. Both read two sequences in order and decide what goes with what.

The minimum total cost is the Levenshtein distance, or edit distance. It can be computed with a table, because the best alignment of two word lists is built from best alignments of their prefixes.

In plain words, the last operation of the best alignment either deletes the last reference word, inserts the last hypothesis word, or pairs the two. So each cell is the cheapest of those three options. $\mathbb{1}[r_i \ne h_j]$ is 1 when the two words differ and 0 when they match.

So the total is well defined, but the split into $S$, $D$, and $I$ is not. Scoring tools break ties with rules. NIST's scorer sclite, the standard for decades, aligns with cost 3 for an insertion or deletion and 4 for a substitution. On the example above it prefers the one-of-each story, $4 + 3 + 3 = 10$ against $3\cdot4 = 12$. The cell fills the table for any pair.

The first two pairs below the table show that WER is not symmetric. Swapping reference and hypothesis keeps the edit distance (2) but changes the denominator, so "yes" against "yes yes yes" is 200 percent and the reverse is 67 percent. The last pair is 200 percent too. Its character error rate is 9/16, or 56 percent, because most of the letters survive.

def wer_align(ref, hyp):
    raise NotImplementedError

def wer(ref, hyp):
    raise NotImplementedError

Scoring a whole test set

A test set has thousands of utterances, and there are two ways to combine them. The right one pools. It divides the total errors by the total reference words, so that every reference word counts once. The wrong one averages the utterances' WERs, which lets short utterances dominate.

Some languages are written without spaces between words, such as Chinese, Japanese, and Thai. There, word segmentation is itself ambiguous, so systems report CER over characters. For English, CER is a finer diagnostic. A misspelled name costs a whole word in WER but only one or two characters in CER.

WER has five properties every interviewer expects you to know.

• It can exceed 100 percent, since insertions are unbounded.
• It is not symmetric, because the denominator is the reference length.
• It weighs every word the same, so "not" counts as much as "uh". A transcript can have a low WER and the opposite meaning.
• It depends on text normalization, the rules that decide whether "Mr." is the same word as "mister" or "twenty five" the same as "25". Normalization can move published numbers by several points, and §36.5 treats it in detail.
• It is a point estimate on a finite test set. Two systems cannot be called different without a confidence interval, also in §36.5.

WER also scores speech synthesis for intelligibility. You run a recognizer on the synthetic audio and score its transcript against the input text (§37.6).

def corpus_wer(refs, hyps):
    raise NotImplementedError

def cer(ref, hyp):
    raise NotImplementedError

In practice nobody writes the scorer by hand. The jiwer library computes WER, CER, and the alignment with unit costs.

Speed is reported next to accuracy

A recognizer that is accurate but slow is useless for live captions and expensive for archives. So accuracy is always reported next to speed.

For example, a system that transcribes one hour of audio in 36 seconds has RTF 0.01 and RTFx 100. RTFx measures throughput, not latency. A batch system can reach RTFx in the thousands while each utterance waits seconds for its batch. Latency, how long after a word is spoken it appears, is the subject of §36.6.

One clip through the whole pipeline

Here is the whole section on one input. We follow the three-second TIMIT sentence from sound to a scored transcript and print the shape of the data at each stage. The first cell turns sound into features, the second draws them, the third shortens the features into encoder frames, and the fourth reads a decoder's output and scores it.

Sound to features. A microphone samples air pressure 16,000 times a second, so three seconds is 48,000 numbers. The front end of §35.3 cuts them into overlapping 25 ms windows (400 samples), one every 10 ms (160 samples). It takes the power spectrum of each window with the Fourier transform (§35.2), which gives 201 frequency bins. It pools the bins into 80 triangular mel bands and takes the log. The cell builds a synthetic clip with the word timings of the first cell, each word a chord of three tones, and runs that front end on it.

The picture shows both ends of the front end. The top is the waveform, with the true word boundaries drawn in and the longer words labeled. The bottom is the log-mel spectrogram the recognizer reads, where each word's three tones are bright horizontal bands.

Features to encoder frames. The encoder's first layers are two strided convolutions with kernel 3, stride 2, and padding 1. They shorten the 300 frames to 150 and then 75, one vector every 40 ms, each of width $d$. The Conformer blocks after them transform the 75 vectors without changing how many there are. The cell runs such a stem with random weights to show the shapes. A trained stem has the same shapes and better numbers inside.

Encoder frames to text, and the score. A decoder turns the encoder frames into tokens, and §36.2 to §36.4 build three kinds of decoder. There is no trained model in this page, so the token stream below is written by hand, in the format of Whisper's decoder, with one wrong word ("in" heard as "and").

Whisper's decoder first reads three special tokens that set the task, <|startoftranscript|>, a language token <|en|>, and <|transcribe|>. It then writes text tokens between timestamp tokens such as <|1.76|>, which mark where each segment starts and ends in steps of 20 ms. It stops with <|endoftext|>. The real tokenizer splits rare words into pieces, but here each word is one token for readability. The cell reads off the segments and their times, then scores the transcript with the edit_table function from the WER cell.

These stages are also the checklist for debugging a recognizer. A wrong sample rate breaks the features. A length off by one breaks the CTC length condition. A missing task token changes what the decoder does. A different text normalization changes the score. An interviewer who asks you to design a transcription service expects you to walk through these stages with their shapes, rates, and costs. §36.4 follows a clip through a real Whisper decoder in the same way.

Real scale, and where the numbers break

LibriSpeech (Panayotov et al. 2015) is the field's standard benchmark. It holds 1,000 hours of read English audiobooks from the LibriVox project, sampled at 16 kHz. Its speakers were ranked by the WER of a baseline recognizer and split in the middle into "clean" and "other" pools. So test-clean (5.4 hours) and test-other (5.1 hours) differ in how hard the speakers are to recognize, not in recording conditions. Conformer L reached 2.1 and 4.3 percent WER on them without a language model, and 1.9 and 3.9 with one (Gulati et al. 2020). Larger Conformers pretrained on unlabeled audio reached 1.4 and 2.6 the same year (Zhang et al. 2020). Test-clean is close to saturated.

Robustness. A low WER on the kind of audio a model was trained for says little about other audio. Table 2 of the Whisper paper (Radford et al. 2022) compares a wav2vec 2.0 model fine-tuned on LibriSpeech with Whisper large-v2. Both score 2.7 percent on test-clean. Over 13 other test sets the first averages 29.3 percent and the second 12.8. §36.4 opens with that comparison and explains the difference, and §36.5 shows how to evaluate robustness honestly. This is why the field now reports averages over many test sets.

The speed column decides real costs. At RTFx 3,390, one GPU transcribes an hour of audio in $3600/3390 \approx 1.1$ seconds. At RTFx 146 it takes about 25 seconds. So a 1,000-hour archive takes 18 minutes against nearly 7 hours. The next three sections explain where that factor of more than 20 comes from. CTC (§36.2) and transducers (§36.3) emit from one pass of the encoder with a light decoder, while Whisper and LLM decoders (§36.4) generate one token at a time.

Read more

• Jurafsky and Martin. Speech and Language Processing, Chapter 16, Automatic Speech Recognition. The textbook treatment of the task, the end-to-end models, and WER scoring with sclite and significance tests.
• Rabiner (1989). A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. The classic tutorial on the forward, backward, and Viterbi algorithms and on Baum-Welch training. This section keeps its $b_j$ for emissions but writes $P_{ij}$ for its $a_{ij}$.
• Mohri, Pereira, and Riley (2002). Weighted Finite-State Transducers in Speech Recognition. How the language model, lexicon, context dependency, and HMMs compose into one search graph. Povey et al. (2011), The Kaldi Speech Recognition Toolkit, is the toolkit that made this the standard, and its lattice documentation shows the acoustic scale in use.
• Hinton et al. (2012). Deep Neural Networks for Acoustic Modeling in Speech Recognition. The shared view of four research groups on DNN-HMM hybrids, including the scaled likelihood and the Switchboard result.
• Gulati et al. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. Short and precise, with the block's figures, the three model sizes, and the ablations. Rekesh et al. (2023), Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition, adds 8x subsampling.
• Panayotov et al. (2015). Librispeech: An ASR Corpus Based on Public Domain Audio Books. How the benchmark was built and why "clean" and "other" differ. For the human-parity debate read Xiong et al. (2016), Achieving Human Parity in Conversational Speech Recognition, and Saon et al. (2017), English Conversational Telephone Speech Recognition by Humans and Machines.
• Prabhavalkar et al. (2023). End-to-End Speech Recognition: A Survey. A map of every end-to-end family this chapter derives, with their history and trade-offs.
• Srivastav et al. (2025). Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation. How WER and RTFx are measured across 86 systems, and what the 2026 frontier looks like. Read it with Table 2 of Radford et al. (2022), Robust Speech Recognition via Large-Scale Weak Supervision, on why in-distribution WER is not robustness.

Watch next

▶

▶