This section is about what happens after a speech recognizer is trained. There are two jobs. The first is to add knowledge from text when decoding, so the recognizer gets drug names, contact names, and domain phrases right. The second is to measure the result honestly, so that a claimed improvement is real. Both are daily work on a production speech team, and both come up in interviews.

Start with the text. An end-to-end recognizer is one network that maps audio straight to text, such as the CTC, transducer, and attention models of §36.2 to §36.4. It learns which word sequences are likely only from the transcripts paired with its training audio. Text without audio cannot train it. Yet text is far more plentiful than transcribed audio. The 960 hours of LibriSpeech training audio come with transcripts of about 9.4 million words (Meng et al. 2020). The same corpus ships about 803 million tokens of extra text from some 14,500 public-domain books (Panayotov et al. 2015), roughly 85 times as much. A clinic that wants its dictations transcribed is in the same position. It has years of typed notes full of drug names and no matching audio at all.

The tool for using that text is a language model (LM), a model trained on text alone that assigns a probability to every sequence of words or tokens. The classical recognizer of §36.1 had a slot for one. It picked the word sequence that maximized an acoustic score, how well the audio fits the words, plus a weighted LM score, so a better LM was a drop-in upgrade. An end-to-end network has no such slot. The first half of this section puts one back. We start with the obvious recipe, shallow fusion. Bayes rule shows exactly what it gets wrong, and two fixes, the density ratio and internal language model estimation, follow from it. Then we put an LM inside CTC beam search, one of the most asked speech coding questions, and cover second passes and contextual biasing toward names.

The second half is about the yardstick. The word error rate (WER) is the number of word substitutions, deletions, and insertions needed to turn the output into the reference, divided by the number of reference words (§36.1). We decode and train for WER itself rather than for whole-sentence accuracy. We make the recognizer's confidence in each word mean what it says. Then we make the error rate honest. The same output can score very differently under two text normalizers. One test set says little about accents, noise, and meetings. And a 0.2-point gain on LibriSpeech test-clean is often noise, because its 2,620 utterances come from only 40 speakers.

Shallow fusion

The simplest way to use an LM is to let the recognizer propose transcripts and let the LM vote on them. Each candidate gets the recognizer's log-probability plus a weighted LM log-probability, and the best total wins. Here is that idea on two candidates. The recognizer gives "met for men" a log-probability of $-2$ and "metformin" $-5$, because the common words sound almost the same and are far more familiar to it. A medical LM gives them $-15$ and $-10$. With the LM at weight 1 the totals are $-17$ and $-15$, so "metformin" wins.

Now the notation. The recognizer reads a sequence of audio feature frames $\mathbf X$, the log-mel frames of §35.3. It scores a label sequence $\mathbf y = (y_1, \dots, y_U)$ of characters or subword tokens with its posterior $p(\mathbf y \mid \mathbf X)$, the probability of the transcript given the audio. The length in tokens is $|\mathbf y| = U$. A separately trained LM assigns $p_{\text{LM}}(\mathbf y) = \prod_{u=1}^{U+1} p_{\text{LM}}(y_u \mid y_{\lt u})$, a product of next-token probabilities. The last factor, for $y_{U+1}$, is the end-of-sentence token, so the LM also says how likely the transcript is to stop where it does.

Recognizers decode with beam search (§3.5). It grows transcripts one token at a time and keeps only the $b$ best partial transcripts after each step, a set called the beam. Each candidate transcript is a hypothesis. The finished hypotheses, ranked by score, form an n-best list.

In plain words, the recognizer proposes, and an LM trained on far more text votes, with $\lambda_{\text{LM}}$ setting how much its vote counts. Compare the classical rule of §36.1, which maximizes $\log p(\mathbf X \mid W) + \lambda_{\text{LM}}\log P(W) + \lambda_{\text{len}}|W|$ over word sequences $W$. Shallow fusion is the same formula with one swap. The acoustic likelihood $p(\mathbf X \mid W)$, how probable the audio is given the words, is replaced by the posterior $p(\mathbf y \mid \mathbf X)$. That swap should make you suspicious, and the next part shows why.

The idea comes from machine translation, where Gulcehre et al. (2015) named both shallow fusion and a trained variant, deep fusion. For speech, Kannan et al. (2017) gave the careful study. Their test bed was the Wall Street Journal read-speech corpus (WSJ), with 81 hours of training audio. A character-level attention recognizer, the Listen, Attend and Spell model of §36.4, went from 10.3 to 6.9 percent test WER when fused with a character-level recurrent LM. That is a third fewer errors, because 81 hours of transcripts teach a poor LM. On Google Voice Search, with about 12,500 hours of training audio, the gain was smaller, from 7.7 to 7.0 percent. That is the 9.1 percent relative reduction usually quoted.

Why the length reward

Every token the LM scores costs some probability. Say the LM gives the typical next token a probability of 1 in 50. In natural-log units, called nats, that token costs $\log 50 = 3.9$ nats. A transcript one token longer pays 3.9 more nats, times $\lambda_{\text{LM}}$, whether or not the extra token was really said.

In general, let $H$ be the LM's average cost per token on typical text, its per-token cross-entropy in nats. Perplexity, the effective number of equally likely next tokens (§0.5), is $\operatorname{PPL} = e^H$, so $H = \log\operatorname{PPL}$. A hypothesis pays about $\lambda_{\text{LM}}H$ per token. The recognizer's posterior already charges every token once, since it too is a product of next-token probabilities. Training set that charge so the recognizer's transcripts come out the right length. Fusion adds a second charge per token on top. So the search prefers short hypotheses, drops words, and in attention models ends the sentence early. A reward of $\lambda_{\text{len}} \approx \lambda_{\text{LM}}H$ per token cancels the extra charge on average. That leaves the LM its real job, which is to prefer one word sequence over another of the same length.

Where the LM score enters in each family

Shallow fusion is one formula, but each family of recognizers has a different place to apply it. The rule is that the LM scores text, so it applies only where the text grows. Getting the place wrong is a classic bug.

• Attention encoder-decoders (§36.4) emit one token per decoder step. The LM term is added at every step, including the end-of-sentence token.
• Transducers (§36.3) emit either a blank or a label at each point of their lattice of frames by labels. A blank only moves to the next frame and gets the transducer's score alone. A label extends the text and gets $\lambda_{\text{LM}}\log p_{\text{LM}}(c\mid\text{prefix}) + \lambda_{\text{len}}$.
• CTC (§36.2) emits a symbol at every frame, but most frames are blanks or repeats that merge into the previous character. The LM applies only when the prefix actually grows by a character. It never applies on a blank or on a merged repeat. With a word-level n-gram LM, Hannun et al. (2014) applied it only when a space completes a word. Their LM weight $\alpha$ is our $\lambda_{\text{LM}}$. They wrote the length term as a factor $|W|^\beta$ on the word count, but decoders such as pyctcdecode use $\beta$ as a reward per word, which is our $\lambda_{\text{len}}$.

What an interviewer asks here is where the LM term goes in each family, and why a blank must not get it.

What shallow fusion gets wrong

An end-to-end recognizer already contains a language model. It learned one from its training transcripts, because predicting text from audio is easier when you know which texts are common. Shallow fusion adds a second LM on top without removing the first. This part makes that precise and fixes it.

A domain is a kind of speech with its own typical text, such as audiobooks, voice search, or medical dictation. Call the domain the recognizer was trained on the source and the one we deploy in the target. A recognizer trained by maximum likelihood on (audio, transcript) pairs from the source estimates the source posterior. Bayes rule splits that posterior into an acoustic part and a prior, the probability of the text before hearing any audio,

The prior $p_{\text{src}}(\mathbf y)$ is the internal language model, the LM the recognizer learned from its transcripts. Shallow fusion multiplies in a second prior, so it decodes with $p(\mathbf X\mid\mathbf y)\,p_{\text{src}}(\mathbf y)\,p_{\text{LM}}(\mathbf y)^{\lambda_{\text{LM}}}$. The source prior is still there. When the domains differ, it keeps pulling toward the audiobook words that the medical LM is supposed to overrule. When the domains agree, the same prior is counted twice. Neither is the right decision rule for the domain we deploy in.

Proof. Bayes rule in each domain gives $p_{\text{src}}(\mathbf y\mid\mathbf X) = p(\mathbf X\mid\mathbf y)\,p_{\text{src}}(\mathbf y)/p_{\text{src}}(\mathbf X)$ and $p_{\text{tgt}}(\mathbf y\mid\mathbf X) = p(\mathbf X\mid\mathbf y)\,p_{\text{tgt}}(\mathbf y)/p_{\text{tgt}}(\mathbf X)$, with the same $p(\mathbf X\mid\mathbf y)$ by assumption. Divide the second by the first. The acoustic likelihood cancels, leaving $p_{\text{tgt}}(\mathbf y\mid\mathbf X)/p_{\text{src}}(\mathbf y\mid\mathbf X) = \big(p_{\text{tgt}}(\mathbf y)/p_{\text{src}}(\mathbf y)\big)\big(p_{\text{src}}(\mathbf X)/p_{\text{tgt}}(\mathbf X)\big)$, which rearranges to the first display. The factor $p_{\text{src}}(\mathbf X)/p_{\text{tgt}}(\mathbf X)$ is positive and the same for every $\mathbf y$, so it does not change the arg max, and taking logs gives the second display. ∎

In plain words, divide out the prior the recognizer learned and multiply in the prior of the place you deploy. The classical hybrid recognizer of §36.1 made the same move one frame at a time. Its network output a posterior over HMM states, and dividing by each state's prior frequency turned it into a scaled likelihood. Here the move is applied to whole sentences. McDermott, Sak, and Variani (2019) proposed it for transducers under the name density ratio. They estimated $p_{\text{src}}$ with a separate LM trained on the recognizer's own training transcripts.

The assumption that matters is the shared acoustics. It holds when the same kind of microphone records the same kind of speakers saying different things. That is the usual shift in text, such as dictation versus voice search, or audiobooks versus medicine. It fails when the audio changes too, as with a new accent or far-field microphones, and then no reweighting of text priors can fix it.

Real models are not exact, so in practice every log term gets its own weight, tuned on held-out data.

The subtraction has a side effect on length. When $\lambda_{\text{ILM}}$ is close to $\lambda_{\text{LM}}$, subtracting $\log p_{\text{ILM}}$ gives back most of the per-token cost that $\log p_{\text{LM}}$ charges. So the density ratio needs much less length reward than shallow fusion.

The cell checks the arithmetic.

A simulation of domain shift

One example can be arranged to say anything, so let's simulate the two situations that matter. The test is a stream of single words, and each is decoded on its own.

• What is said. Words are drawn from a target-domain prior $p_{\text{tgt}}$ over 40 words with Zipf-like frequencies, where a few words are very common and most are rare.
• What is heard. Each word has an acoustic signature, a fixed point in 6 dimensions. The recognizer hears the signature plus Gaussian noise, so words with nearby signatures get confused.
• The recognizer. Its log posterior is the exact $\log p(\mathbf x\mid w) + \log p_{\text{src}}(w)$, up to a constant. So the only thing wrong with it is its prior.
• The external LM. A slightly noisy copy of $p_{\text{tgt}}$, as if trained on plenty of target text.

In the cross-domain case the internal prior favors 8 other words that the source domain used a lot. In the in-domain case the internal prior is the right one but noisily estimated, as if learned from too few transcripts. The Bayes oracle decodes with the true acoustics and the true target prior, the best any decoder can do.

Read the two panels. Cross-domain, the recognizer alone makes 34.6 percent errors. Shallow fusion bottoms out at 25.3 percent near weight 1, still far above the oracle's 18.7, because no weight on the new prior removes the old one. In-domain, shallow fusion's best weight is 0.4 and its gain is small (22.6 to 21.2 percent). The external LM mostly repeats a prior the posterior already has. Counting it twice at full weight makes things worse than no LM at all, 23.6 percent against 22.6.

The density ratio matches the oracle's 18.7 percent at weight 1 in both cases, as the theorem predicts. Its decision is $\log p(\mathbf x\mid w) + \log p_{\text{LM}}(w)$, the Bayes rule with the external LM standing in for the true prior. What changes in practice is the default. When you have an estimate of the internal LM, subtract it, and expect the tuned shallow-fusion weight to sit well below 1 when the domains match.

Estimating the internal language model

The density ratio needs $p_{\text{src}}(\mathbf y)$. McDermott and colleagues trained a separate LM on the recognizer's training transcripts. That costs a second model, and it estimates the prior of the training text rather than the prior the network actually learned. Meng et al. (2020) read the internal LM out of the recognizer itself instead. They remove the acoustic input and see what the network predicts from the previous labels alone.

Recall the transducer's joint network from §36.3. An encoder turns audio frame $t$ into a vector $\mathbf h^{\text{enc}}_t$. A prediction network turns the $u$ labels emitted so far into a vector $\mathbf h^{\text{pred}}_u$. The joint network adds projections of the two, applies a nonlinearity $\phi$ such as $\tanh$, and outputs logits over the $V$ labels plus the blank. Internal LM estimation deletes the encoder's contribution.

Why should deleting the acoustic term give the prior? If the joint network were additive in its two inputs, its softmax would factor into an acoustic part times a language part, and the network with the audio removed would be the language part. A $\tanh$ is not additive, so this is an approximation, not a theorem. It works well enough in practice, and the fact box below shows how well.

The tuned weights in those papers are far from the theorem's 1. Meng et al.'s best transducer setting on LibriSpeech was $\lambda_{\text{LM}} = 0.30$ and $\lambda_{\text{ILM}} = 0.14$. McDermott and colleagues call the need for weights at all the method's clear weakness. The theorem assumes exact models. Real posteriors are miscalibrated, real LMs are trained on other tokenizations and text styles, and the internal LM is only an estimate. The decision rule is right, and the weights absorb how far each piece is from its ideal. An interviewer will ask exactly this, why the theory says 1 and practice says 0.3.

The widget puts the decision rule in your hands. Each preset is a realistic 5-best list with its three component scores, shown when you hover a row. The curve underneath is the pooled WER on a synthetic dev set of 300 n-best lists built to behave like that domain. It is swept over $\lambda_{\text{LM}}$ at your current length reward. When the internal LM is subtracted, its weight is tied to the LM weight, $\lambda_{\text{ILM}} = \lambda_{\text{LM}}$. Press Sweep to raise $\lambda_{\text{LM}}$ from 0 to 2 and watch the winner change while the dot rides the curve.

def fused_score(h, lam_lm, lam_ilm, lam_len):
    return h["am"] + lam_lm * h["lm"]          # not finished

def rescore(nbest, lam_lm, lam_ilm=0.0, lam_len=0.0):
    raise NotImplementedError

def tune_lm_weight(dev, grid, lam_ilm=0.0, lam_len=0.0):
    raise NotImplementedError

import numpy as np

def e2e_conditionals(acoustic, log_src):
    raise NotImplementedError

def viterbi(local):
    raise NotImplementedError

A language model inside CTC prefix beam search

A CTC recognizer (§36.2) outputs, at every audio frame, a probability for each character and for a special blank symbol that means "nothing new at this frame". Given the audio, its outputs at different frames are independent, so no frame's choice depends on what was written at the others. That gives CTC the weakest internal LM of the three families, and the most to gain from an external one.

Hannun et al. (2014) showed how much on WSJ. A CTC model decoded greedily, taking the most probable symbol at each frame, made 35.8 percent word errors. Prefix beam search constrained to a 20,000-word dictionary made 24.4. Adding a word bigram LM brought it to 14.1. The character error rate (CER), the same edit-distance ratio counted on characters, was only 10.0 percent to begin with and fell to 5.7. Most of the characters were right all along. They were just assembled into misspelled words.

Here is the prefix beam search of §36.2 again, in brief. Write $\varnothing$ for the blank. A frame path is one symbol per frame, and it collapses to a transcript by merging repeated symbols and then deleting blanks, so "a a ∅ b" spells "ab". A prefix $\ell$ is a partial transcript. For each surviving prefix after $t$ frames the search keeps two numbers. $p_b(\ell)$ is the total probability of the frame paths that collapse to $\ell$ and end in a blank, and $p_{nb}(\ell)$ is the same for paths that end in a character. Write $p_{t+1}(a)$ for the model's probability of symbol $a$ at frame $t+1$. At frame $t+1$ every path grows in one of these ways.

• A blank keeps the text $\ell$. It adds $p_{t+1}(\varnothing)\,[p_b(\ell) + p_{nb}(\ell)]$ to the new $p_b(\ell)$.
• The last character $c$ of $\ell$ again, after a character, merges into it and keeps the text. It adds $p_{t+1}(c)\,p_{nb}(\ell)$ to the new $p_{nb}(\ell)$.
• The last character $c$ again, after a blank, starts a new copy. It adds $p_{t+1}(c)\,p_b(\ell)$ to $p_{nb}(\ell c)$.
• Any other character $c$ extends the text. It adds $p_{t+1}(c)\,[p_b(\ell) + p_{nb}(\ell)]$ to $p_{nb}(\ell c)$.

Paths that reach the same prefix are summed, and the $b$ best prefixes by $p_b + p_{nb}$ survive. Only the last two cases make the text longer, and they are where the LM goes.

The one rule that matters is where the factor goes. It is paid when the text grows, never when a frame passes. The smallest possible example shows why that is the right place.

In the example the fused score of "a" came out as its CTC probability times its LM probability, the same factor on every path that spells it. The theorem says this holds for every prefix after every frame, with the LM probability raised to its weight and the length reward included. When the beam prunes, the CTC sum covers only the frame paths that survived.

In plain words, the LM score belongs to the text, not to the frame path. Every path for a prefix spells the same text, so the LM factor can be folded into the running sums without breaking the merging that makes prefix beam search work. That is also why the LM must not touch blanks or merged repeats. Doing so would charge different paths for the same text differently, and the sums would stop meaning anything.

Real systems differ from the exercise below in the unit of the LM. A character LM fits this algorithm directly. The usual choice is a word n-gram LM, often built and queried with the KenLM library. It is applied only when a space or a word-initial subword token completes a word. Partial words are then unscored, and the length reward counts words. With subword vocabularies the decoder must also know which tokens start a word.

import numpy as np

def ctc_lm_beam_search(probs, alphabet, beam, lm, lam_lm=0.0, lam_len=0.0, blank=0):
    # placeholder: greedy decoding, which ignores the LM and does not sum over paths
    best = probs.argmax(1)
    out, prev = [], None
    for k in best:
        if k != prev and k != blank:

In production you would not write this by hand. The open-source pyctcdecode runs the same search over a CTC model's log-probabilities with a word-level n-gram LM, and it handles word boundaries for subword vocabularies. Its alpha is $\lambda_{\text{LM}}$, its beta is the per-word reward, and its optional hotwords are the phrase biasing of a later part of this section.

Beam search details that matter in speech

Beam search was built in §3.5 and revisited for language models in §6.4. It keeps a list of finished hypotheses, normalizes scores by length, and batches the beam with a cache. Only three things are specific to speech.

Why speech tolerates beam search. §6.4 showed that for open-ended text the most probable output is atypical and repetitive, so a better search makes the text worse. A recognizer's posterior is different. Given the audio there is one right transcript, and a good model concentrates its probability near it. So the most probable output is usually the answer, and a better search finds it more often.

Typical beams are small, 4 to 16 for attention decoders and 4 to 8 for transducers. Whisper uses 5, and the minimum word error rate experiments later in this section used 8. CTC prefix beams are often wider, because a step is cheap and an LM needs alternatives to choose from. Hannun et al. used 200, with a character-level CTC model and a word bigram LM. The failure that remains is the one §3.5 found, a short or empty hypothesis that scores well. Fusion makes it worse, since every token costs LM probability. The length reward and the end-of-sentence term control it.

Frame-synchronous and label-synchronous search. CTC prefix search and transducer beam search advance one frame at a time. All hypotheses at a step have consumed the same audio, so their scores are directly comparable. Attention decoders advance one token at a time, and hypotheses finish at different steps. So they must be compared at the end, with length normalization or a reward. Whisper's beam search keeps decoding until the number of finished hypotheses reaches the beam size times a patience factor, 1 by default. It then ranks the finished ones by total log-probability divided by length, unless a different length penalty is requested. With fusion, each finished hypothesis must include the LM's end-of-sentence log-probability, or the LM never pays for stopping.

Cost. An attention decoder runs $b$ decoder steps per output token, one per hypothesis, plus $b$ steps of the external LM if one is fused. That is why fusion with a large neural LM is expensive. CTC runs the encoder once and does the search afterward, usually on the CPU, at $O(T\,b\,V)$ for $T$ frames and $V$ symbols.

Rescoring and second passes

Fusion is not the only place to use an LM. A first pass can produce an n-best list, or a lattice, the compact graph of competing hypotheses from §36.1. A bigger model then rescores it, recomputing each hypothesis's score and picking a new winner. Rescoring is simple, and it can use models too large or too slow for the first pass. But it can only choose among what the first pass kept. Kannan et al. found n-best rescoring less effective than shallow fusion on WSJ for exactly that reason. The correct prefix was often pruned early and never reached the list.

The usual compromise is a small LM fused in the first pass, and a large LM or a second recognizer rescoring afterward. The two-pass streaming systems of §36.6 work this way. A streaming transducer or CTC model writes partial results while you talk, and when you stop, an attention decoder rescores its n-best list over the whole utterance (Sainath et al. 2020).

Large language models (LLMs) opened a third option, generative error correction. Instead of choosing among the n-best, an LLM reads the list and writes the transcript. Yang et al. (2023) showed that frozen LLMs prompted with an n-best list rescore about as well as LMs tuned to the domain. With fine-tuning, the output can beat the n-best oracle, the best hypothesis in the list, because the LLM can combine the right words of different hypotheses. Chen et al. (2023) released HyPoradise, more than 334,000 pairs of n-best lists and reference transcripts, as a benchmark for it.

The risk is the obvious one. A text-only corrector cannot hear the audio. It can replace a rare but correct word with a fluent wrong one, which is the hallucination problem of §36.4 moved to the post-processor. The 2026 speech-LLM recognizers of §36.4, which use an LLM as the decoder, fold this step into the first pass, where the audio is still in view.

Contextual biasing

Contextual biasing tilts recognition toward a list of phrases expected right now, such as contacts, song titles, product or drug names, or the words on a slide. A user says "call Siobhan". The recognizer has rarely or never seen the name, and the spellings that sound the same ("Shivon", "Chevon") are each more common in its training text. Yet the phone knows the contact list. Production voice assistants depend on biasing to use it.

The workhorse is shallow fusion with a small biasing model built from the phrase list on the fly. The phrases are split into subword tokens and stored in a prefix tree, a tree whose paths from the root spell the phrases. During beam search, every token that extends a match of some phrase earns a bonus. So a hypothesis that has begun the rare name survives pruning long enough to finish it. If the match breaks before the phrase is complete, the bonus it collected is taken back, so partial matches gain nothing in the end.

Why not just reward the whole name once it is complete? Because biasing only at word boundaries, the classical way, works poorly for end-to-end models. They write a name one subword at a time, and the rare name has usually left the beam before its last piece arrives. Pundak et al. (2018) put the bonus on every character instead. Failure arcs, the transitions taken when a match breaks, subtract what was added. Zhao et al. (2019) extended the approach to subword units. They also added common prefixes such as "call" and "text" that signal when a phrase list is relevant. The cell computes the running bonus for four hypotheses, with a bonus of 2 per matched token.

Whisper users have a biasing channel built in, the previous-text prompt of §36.4. Text passed to the reference implementation as initial_prompt goes after the <|startofprev|> token. The decoder reads it as the transcript of an earlier window, so the spellings in it ("Siobhan", "metformin") become likelier continuations. It works because training put real previous text there and masked the loss over it. The model learned to use that context, not to transcribe it.

The prompt has limits. Only the last 223 tokens of previous text are kept. So the prompt scrolls out once later windows' transcripts fill that space, unless carry_initial_prompt is set. It is also dropped after any decode at temperature above 0.5. And it is a prior, not evidence. Over silence or unclear audio the decoder can copy prompt words into the transcript, and a long or off-topic prompt seeds the hallucinations of §36.4.

Optimizing the metric itself

Picking the most probable transcript is called MAP decoding, for maximum a posteriori. §36.1 proved that it minimizes the chance of getting the whole sentence wrong. It does not minimize the expected number of wrong words, which is what WER counts. We fix that mismatch twice, once at decoding time and once at training time.

Minimum Bayes risk decoding

Minimum Bayes risk (MBR) decoding picks the output with the lowest expected loss under the model's own beliefs, instead of the most probable output. Think of it as asking each candidate "if the truth were drawn from what the model believes, how many words would you get wrong on average?" §3.5 used it for translation, with samples as stand-in references and a translation-quality score as the utility. In speech the loss is the word error count, and the model's beliefs come from the n-best list itself.

Why the expected count and not the expected WER of the utterance? Corpus WER is total errors over total reference words (§36.1). The reference lengths do not depend on what we output. So minimizing each utterance's expected error count minimizes the expected corpus error count, which is what the metric measures. The cost is $n^2$ edit distances per utterance, fine for $n \le 16$.

The gain from MBR is usually modest, because a good recognizer puts most of its probability on one hypothesis, and then MBR agrees with MAP. It helps most where the model is unsure, which is also where the errors are.

Minimum word error rate training

MBR changes the decision. Better still is a model whose own distribution puts its probability on low-error transcripts. Maximum likelihood training does not aim for that. It raises the probability of the reference and treats every other transcript as equally wrong, whether it has one error or ten. Prabhavalkar et al. (2017) trained attention models to minimize the expected number of word errors directly, with the same n-best list standing in for all transcripts.

In plain words, the loss is the expected number of errors under the model's own n-best, measured relative to the list's average. The constant $\bar e$ does not change the gradient, as the theorem below shows, but it keeps the loss centered. The cross-entropy term keeps the model anchored to the reference. Prabhavalkar et al. found it important for stable training, with $\lambda_{\text{CE}} = 0.01$.

Take the MBR list again, with $\hat p = (0.4, 0.3, 0.3)$, and suppose the reference is "call dan now". The errors are $e = (1, 0, 1)$, so the model's expected count is $\bar e_{\hat p} = 0.4 + 0.3 = 0.7$. The gradient is $0.4 \times 0.3 = 0.12$ for "call ann now", $0.3 \times (-0.7) = -0.21$ for "call dan now", and $0.3 \times 0.3 = 0.09$ for "call dan no". They sum to zero. A gradient descent step lowers the scores of the two wrong hypotheses and raises the right one, which gets the largest push.

The gradient has a familiar form. REINFORCE (§15.3) is the policy-gradient rule that raises the log-probability of each sampled output in proportion to its reward. Subtracting a baseline, a reference value, from the reward lowers the variance without changing the expected gradient (§15.4). The exact gradient of the expected errors $\mathbb E_{\mathbf y\sim p_\theta}[e(\mathbf y)]$ is $\mathbb E[(e(\mathbf y) - b)\nabla_\theta\log p_\theta(\mathbf y)]$ for any constant $b$. So MWER is REINFORCE with the negative error count as the reward, and with two changes.

• It replaces the expectation over all transcripts by the exact expectation over the renormalized n-best. That ignores the probability outside the list, a bias, but it has no sampling noise given the list.
• Its effective baseline is the list's model-weighted mean error, $\bar e_{\hat p}$.

Draw the outputs by sampling instead, weight them uniformly, and use their mean error as the baseline, and you have the estimator of GRPO (§18.4), without GRPO's standard-deviation scaling, clipping, and KL term. That link is a favorite interview follow-up.

import numpy as np

def mbr_choice(nbest_logp, hyps):
    return int(np.argmax(nbest_logp))            # placeholder: MAP, not MBR

def mwer_loss_and_grad(nbest_logp, errs):
    raise NotImplementedError

In PyTorch the loss is three lines, and autograd produces the theorem's gradient. The block scores a fixed 4-best list with a tiny recurrent decoder (a GRU), checks the gradient against the formula, and takes 60 Adam steps on MWER plus a small cross-entropy term. Real MWER training reruns beam search with the current model for every batch, so the list changes as the model does.

Word confidence and calibration

A transcript is often used together with a number for each word that says how far to trust it. A medical scribe sends the words it is unsure of to a human reviewer. A voice assistant asks "did you say Siobhan?" instead of calling the wrong person. Self-training transcribes untranscribed audio with the recognizer and trains on the result, keeping only the words the recognizer is sure of. Distil-Whisper (§36.4) could filter its machine transcripts by their WER against the transcripts that came with its training audio, but untranscribed audio has nothing to compare with, so the confidence is the only filter. Each of these uses compares the number with a threshold, so the number has to mean what it says. Words given a confidence of 0.8 should be right 80 percent of the time, which is called being calibrated.

In plain words, NCE compares the log loss of the confidences with the log loss of a lazy recognizer that reports its average accuracy $\bar o$ for every word. Log loss is a proper score (§0.11), one that is minimized only by reporting true probabilities. NCE is 1 for confidences that are 1 on every correct word and 0 on every wrong one. It is 0 for the lazy constant, and negative for confidences worse than that constant. The normalization makes it comparable across test sets with different error rates. Li et al. (2021) report it next to the area under the precision-recall curve.

Discrimination is a separate question, whether the confidences rank correct words above wrong ones. It is measured by the AUROC, the area under the ROC curve (§0.11). Here it is the probability that a randomly chosen correct word gets a higher confidence than a randomly chosen wrong one, so 0.5 is chance and 1 is a perfect ranking. A monotone remapping of the confidences changes their calibration but leaves their order, and so the AUROC, alone. Interviewers like this distinction, because a calibration fix cannot help a confidence that ranks words badly.

Raw confidences are too high, for five reasons.

• Slot posteriors are renormalized over what survived the beam. In the MBR example above, "call dan now" gets $c = 1$, 0.6, and 0.7 from its three-entry list. But the list held only half of the model's probability. If the other half disagreed about "dan", the model's own posterior for it would be 0.3.
• Token probabilities inherit the overconfidence of deep classifiers (Guo et al. 2017, in §0.11). Li et al. (2021) found that regularizers such as label smoothing move softmax confidences a lot.
• CTC frame posteriors are peaky (§36.2), with most frames near probability 1.
• Teacher forcing. An attention decoder was trained on correct histories (§3.5). After its first error, its probabilities describe a situation it never saw in training.
• Fusion. A fused score is not a probability at all, since the LM weight and the length reward rescale it.

The cell simulates a recognizer whose softmax is too sharp. Each word has one to three subword tokens. The audio determines a true distribution over 40 candidates for each token, and the token said is drawn from it. The recognizer's logits are the true logits times 2.5. A word is correct when all its tokens are. Its confidence is the product of its tokens' top probabilities, which would be exactly its probability of being correct if the token probabilities were calibrated.

The fix we test is temperature scaling, dividing all logits by one number $\tau$ before the softmax (§0.11). The cell fits $\tau$ by minimizing the log loss on a dev set of 20,000 other words, then applies it to the test words.

The simulation is kind, since its miscalibration is exactly a temperature. Real miscalibration is not, which is why more flexible maps exist. The AUROC moves at all because a temperature acts on whole logit vectors rather than on the confidence itself. Two words with the same raw confidence can end with different scaled ones, through the shapes of their token distributions and their numbers of tokens.

The fixes come in two kinds. The first is a monotone map, fitted on held-out words from the deployment domain. It repairs calibration and keeps the order, and with it the AUROC. It can be a temperature on the logits, or Platt scaling $c' = \sigma\big(a \operatorname{logit}(c) + b\big)$, a logistic curve on the word score with two fitted numbers $a$ and $b$, or a piecewise linear map. Li et al. (2021) fit piecewise linear maps on the LibriSpeech dev sets. They raised NCE and left the AUC unchanged, as a monotone map must. The second kind adds new information, which is the only way to rank words better.

Text normalization before scoring

WER is computed on strings, so two transcripts that say the same thing in different written forms count as different. Text normalization puts both the reference and the hypothesis into one canonical form before scoring. It can move a published number more than any modeling change.

Whisper made the problem impossible to ignore. It writes text the way the internet does, with casing, punctuation, and digits (§36.4), while most test sets have their own conventions. Its authors built a normalizer by repeatedly inspecting where naive WER penalized harmless differences. Their English pipeline makes these changes.

• Lowercase, and remove text between brackets or parentheses.
• Remove the fillers "hmm", "mm", "mhm", "mmm", "uh", and "um".
• Expand contractions ("won't" to "will not") and standardize titles ("mr" to "mister").
• Convert spelled numbers and currencies to digits ("ten thousand dollars" to "$10000").
• Map British to American spellings.
• Remove remaining punctuation and symbols, and collapse whitespace.

For languages without spaces between words (Chinese, Japanese, Thai, Lao, Burmese), the basic multilingual normalizer puts a space between every character, which turns WER into CER.

A normalizer should remove differences that do not change what was said and keep every difference that does. Casing, punctuation, spacing, number formats, contraction spellings, British and American spellings, and agreed fillers can go. Negations, content words, and anything a downstream reader would act on must stay. A normalizer that drops "not" because it is short makes a system that turns "do not resuscitate" into "do resuscitate" look perfect.

Even well-tested rules have edges. Apart from a few fixed phrases such as "'s been", Whisper's English rules expand "'s" to " is", so "John's car" becomes "john is car" in both reference and hypothesis. That is harmless for scoring, since both sides change the same way. It is also why a normalized string should never be shown to a user.

import re

def normalize(text):
    return " ".join(text.lower().split())        # lowercases and collapses spaces, nothing else

The real tools are short. openai-whisper ships its normalizer, and jiwer scores lists of strings with pooled corpus WER.

Robustness and fairness

A low WER on one test set says little about how a recognizer does in the world. §36.4 opened with the example. A wav2vec 2.0 model, a self-supervised speech encoder (§35.4) fine-tuned with CTC on LibriSpeech audiobooks, and Whisper large-v2 both score 2.7 percent on LibriSpeech test-clean. On 13 other test sets they average 29.3 and 12.8 percent. A test set drawn from the same distribution as the training data is called in-distribution. Radford et al. use two measures beyond it. Effective robustness (Taori et al. 2020, met in §36.4) is how much better a model does out of distribution than its in-distribution score predicts. Overall robustness is the average over many distributions. An honest evaluation reports both kinds of number and looks at the conditions that matter for the product.

Many domains, one system. Evaluate one system, with one set of settings, on many datasets of different styles. Then no tuning to one dataset, such as audio filtering or special text rules, can hide a weakness. Public benchmarks are now built this way.

Conditions. Far-field and overlapping meeting speech (AMI, the CHiME challenges) is far harder than close-talk read speech. Accents and dialects need their own sets, such as CORAAL, interviews in African American English. Noise robustness is a curve, not a number. Whisper's paper plots WER on LibriSpeech test-clean against the signal-to-noise ratio (SNR), for white noise and for pub noise. Many of the 14 models it was compared with, all trained wholly or partly on LibriSpeech, beat Whisper with almost no noise, at 40 dB SNR. Every one of them fell behind it under pub noise below 10 dB.

Fairness. Error rates differ across groups of speakers, and averages hide the gap.

So report WER per group (accent, dialect, gender, age, recording condition) next to the average, with intervals, which the next part makes honest.

Contamination is the remaining threat. Web-scale training data may contain the test sets themselves, or their audio with other transcripts, a problem §22.3 treats in general. The Open ASR Leaderboard has a partial guard. Every track includes at least one test set released under a non-commercial license (TED-LIUM 3, CoVoST 2, CORAAL), which commercially minded training sets are less likely to include. And each track uses several datasets, so one suspiciously good score stands out. An interviewer who asks "is your model robust?" wants to hear which domains, conditions, and groups you checked, and how you ruled out contamination.

Is the improvement real

Your new model scores 3.12 percent on a test set the size of LibriSpeech test-clean, against the old model's 3.29. Is it better? LibriSpeech test-clean has 2,620 utterances, 52,576 words, and 40 speakers, so 0.17 points is about 90 words. Whether 90 words is signal depends on how the errors are spread. The usual check treats each utterance as an independent draw, and that check overstates the evidence severalfold.

Corpus WER is a ratio of sums, $\widehat{\mathrm{WER}} = \sum_i e_i/\sum_i N_i$, where utterance $i$ has $e_i$ word errors and $N_i$ reference words. Its uncertainty is not that of a proportion of independent words, because errors cluster in hard utterances. They cluster even more within a speaker, since a speaker with an unusual accent makes every one of their utterances hard. So a test set of 40 speakers carries something closer to 40 independent pieces of evidence about new speakers than 2,620.

Two tools from earlier chapters combine here. The bootstrap (§0.11) estimates uncertainty by resampling the test items with replacement and recomputing the metric many times. The spread of the recomputed values is the uncertainty. A clustered standard error (§14.7) treats a group of correlated items as one unit.

In plain words, treat each speaker as one draw, keep each speaker's utterances together, and compare the two systems on the same draws. The linearized formula comes from a first-order Taylor expansion of the ratio around its value. The residual $E_g - \widehat{\mathrm{WER}}\,N_g$ is how many more errors speaker $g$ made than the corpus rate predicts for their word count. The factor $G/(G-1)$ is the usual small-sample correction.

Pairing matters as much as clustering. Two systems tend to fail on the same hard utterances, so their differences vary far less than either system's errors. Resampling the two systems independently would throw that away.

The cell simulates a test set shaped like LibriSpeech test-clean, 2,620 utterances of about 20 words from 40 speakers, about 52,600 words in all. Each speaker has their own error rate. System B helps some speakers more than others and a few not at all. And the two systems err on the same hard words, as real systems do.

The utterance-level interval, $-0.25$ to $-0.09$ points, excludes zero comfortably, and the bootstrap agrees with the formula. The speaker-level interval, about $-0.49$ to $+0.14$, is almost four times wider and includes zero. Which one is right? Repeat the whole experiment on many fresh test sets of 40 new speakers each. Count how often each 95 percent interval contains the true improvement, the one we would see averaged over all speakers.

The utterance interval covers the truth only about 40 percent of the time. It answers "would B beat A again on these same 40 speakers?", which is rarely the question. The speaker interval covers it about 95 percent of the time, as advertised. Utterance resampling is not wrong as a computation. It answers a narrower question, and in a paper that claims a better model for new users it overstates the evidence severalfold.

The widget runs the same comparison with the test set's shape on sliders. It draws the bootstrap replicates of $\mathrm{WER}_B - \mathrm{WER}_A$ for both resampling units on one axis, shades each 95 percent interval, and marks the true gain over all speakers with a dashed line. Before you move anything, predict two things. With the speaker effect at zero, should the two resampling units agree? And with a strong speaker effect, which interval shrinks as the number of speakers rises from 10 to 100 at a fixed number of utterances?

With no speaker effect the two intervals agree, because utterances from one speaker are then no more alike than utterances from different speakers. With a strong one, the speaker interval narrows as the speakers go from 10 to 100, while the utterance interval barely moves, since it treats the 2,620 utterances as independent draws either way. Press New test set a few times and count how often each interval misses the dashed line. Here is the whole check on the simulated test set, written out as you would report it.

import numpy as np

def wer_ci(errors, words, speakers, cluster="utterance", n_boot=2000, level=0.95, seed=0):
    r = np.asarray(errors) / np.asarray(words)
    m, s = r.mean(), r.std() / np.sqrt(len(r))
    return m, m - 1.96 * s, m + 1.96 * s           # a mean of per-utterance WERs, with a normal interval

def paired_wer_diff_ci(err_a, err_b, words, speakers, cluster="speaker", n_boot=2000, level=0.95, seed=0):

The tools side by side

Three more options sit outside this section, and all of them change the model.

• Fine-tuning on domain audio. When some transcribed audio from the domain exists, this is the most direct fix. With Whisper and a few hours of audio, a sound recipe is to freeze the encoder, which carries the acoustic robustness that weak supervision bought (§36.4). Adapt the decoder, fully or with the low-rank adapters of §14.5. If the audio itself is new (an accent, a microphone), the encoder needs adapting too. Keep a general dev set next to the domain one, since a fine-tune that wins on drug names can quietly forget everything else.
• Synthetic audio. Text without audio can be turned into training audio with a speech synthesizer (§37.1) and used the same way. That adapts the acoustic and language parts together.
• A speech-LLM recognizer. It brings its LLM's text knowledge into the decoder from pretraining (§36.4). Much of what fusion supplied comes built in, along with the instruction-following and hallucination risks of an LLM.

Read more

• Kannan et al. (2017). An analysis of incorporating an external language model into a sequence-to-sequence model. The careful study of shallow fusion across tasks, units, and LM types, with the 9.1 percent voice search result.
• McDermott, Sak, and Variani (2019). A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition. Bayes rule applied to end-to-end posteriors, and the cross-domain experiments behind it.
• Meng et al. (2020). Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition. Reading the internal LM out of transducers and attention models, with the joint softmax approximation that justifies it.
• Hannun et al. (2014). First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs. Prefix beam search for CTC with a language model and a word insertion term.
• Zhao et al. (2019). Shallow-Fusion End-to-End Contextual Biasing. Subword-level biasing with failure arcs, context prefixes, and anti-context tests for end-to-end models.
• Prabhavalkar et al. (2017). Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models. Expected word errors over an n-best as a training loss, and why sampling worked less well.
• Srivastav et al. (2025). Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation. How a public benchmark normalizes text, chooses test sets, guards against contamination, and measures speed.
• Koenecke et al. (2020). Racial disparities in automated speech recognition. The matched-snippet study of five commercial recognizers and the evidence that the gap is acoustic.

Watch next

▶

▶