This section is about speech recognizers that work like a translator. An encoder reads the whole recording and turns it into a sequence of vectors. A decoder then writes the transcript one token at a time. Before each token it uses attention to look back over those vectors for the part of the audio it needs. Nobody tells the model which stretch of audio belongs to which word, and attention works that out by itself.

Whisper, released by OpenAI in 2022, is the best-known recognizer of this kind. Most of the recognizers at the top of the 2026 benchmarks are built the same way, and many of them use a pretrained large language model as the decoder. So this design is what most people mean today by speech recognition. Interviewers ask three things about it. Where did Whisper's robustness come from? How does one decoder handle many tasks? And why does it sometimes write fluent text that nobody said?

Here is the setup from §36.1, in brief. A front end turns the audio into a log-mel spectrogram, one vector of 80 numbers every 10 ms, each number the log energy in one frequency band (§35.3). A 3-second sentence is therefore 300 frames, while its transcript is about a dozen words. The recognizer must map the frames to the words without being told which frames go with which word. That unknown correspondence is the alignment.

Quality is measured by the word error rate (WER). It counts the word substitutions, deletions, and insertions needed to turn the output into the reference transcript, and divides by the number of words in the reference. One wrong word in a ten-word sentence is a WER of 10 percent.

§36.2 and §36.3 handled the alignment with a dynamic program. It adds up the probability of every way of assigning frames to labels in order. Here we drop explicit alignments altogether and let attention find them. That freedom makes this the most flexible design in the chapter, and the same freedom causes its failures.

The section runs in this order. First the attention recognizer, how it fails, and how a CTC head repairs it. Then Whisper exactly, from its shapes and parameter count to its training data, its task tokens, and its timestamps, with one clip followed end to end. Then long recordings, hallucination, cheaper variants, and the speech-LLMs that lead in 2026.

Same score on audiobooks, very different elsewhere

The clearest reason to study Whisper is one comparison from its paper. Two recognizers score exactly the same on LibriSpeech, the most used benchmark in the field, which is people reading audiobooks aloud. On recordings of other kinds, from phone calls to meetings to dinner parties, one of them makes more than twice as many errors as the other.

LibriSpeech has two test sets. test-clean holds the speakers whose speech was easiest for an early recognizer, and test-other holds the harder ones. Both are read audiobooks.

The two models differ in many ways, but the paper argues that one difference dominates, the training data. wav2vec 2.0 learned to transcribe from 960 hours of audiobooks. Whisper learned from 680,000 hours of audio found on the internet, with whatever transcripts came with it.

The relative error reduction on one test set is 1 minus the ratio of the two WERs. Going from 10 to 4 percent is a 60 percent relative reduction. The cell checks the table's arithmetic and shows why "average relative reduction" needs a definition.

The paper's 55.2 percent is the mean of the 13 per-set reductions. The relative reduction of the two averages is 56.3 percent. Both are fair summaries, and they differ for a simple reason. The hard sets with large WERs dominate an average of WERs, but count once each in an average of ratios. In an interview, say which one you mean. §36.5 returns to how WER should be aggregated and compared.

What this changes in practice is the evaluation plan. A WER on the benchmark you trained for says little about the recordings your product will see. An interviewer who asks "your model has 2 percent WER on LibriSpeech, will it work on call-center audio?" wants this example and a test set drawn from the target audio.

Let attention find the alignment

The encoder-decoder of §3.5 was built for translation. It reads a source sentence and writes the target one token at a time. Each token is conditioned on the source and on the tokens already written. For speech the source is the audio, and nothing else changes.

Here is a tiny example first. Take the two-token transcript "the cat".

• The model gives the first token a probability given the audio, say $p(\text{the} \mid \text{audio}) = 0.9$.
• It gives the second token a probability given the first and the audio, $p(\text{cat} \mid \text{the}, \text{audio}) = 0.8$.
• It gives the end of the transcript a probability too, $p(\langle\text{eos}\rangle \mid \text{the cat}, \text{audio}) = 0.95$. Here $\langle\text{eos}\rangle$ is the end-of-sequence token, the decoder's way of saying stop.

The transcript's probability is the product, $0.9 \times 0.8 \times 0.95 = 0.684$. That is the chain rule of probability, and it involves no assumption at all.

In general, an encoder turns $T_{\text{in}}$ feature frames $\mathbf X = (\mathbf x_1, \dots, \mathbf x_{T_{\text{in}}})$ into $T$ encoder vectors, often fewer than $T_{\text{in}}$. The transcript is a label sequence $\mathbf y = (y_1, \dots, y_U)$ of tokens from a vocabulary $\mathcal V$ of size $V$, usually with $U$ much smaller than $T$.

In plain words, the probability of a transcript is the probability of its first token given the audio, times the second given the first and the audio, and so on until the model says stop. The factorization is exact, so with a flexible enough decoder this model class can represent any distribution over transcripts. Nothing in it mentions frames or alignments.

The alignment appears inside the decoder. At step $u$ it computes attention weights $\omega_{u,1}, \dots, \omega_{u,T}$ over the encoder frames, positive and summing to one, and reads the weighted average of the frames. Stack the weights of all steps into a $U \times T$ matrix. Row $u$ says where in the audio the decoder looked while writing token $u$. That matrix is a soft alignment, the same picture as the translation alignment matrices of §3.6. We write the weights $\omega$ rather than $\alpha$, because $\alpha$ is the CTC forward variable in this chapter.

For speech the matrix should come out close to monotonic, meaning that the decoder's focus only moves forward in time. It looks like a band from the top left to the bottom right, because words are spoken in the order they are written. Nothing in the model forces that.

The freedom has a payoff and a price. The decoder conditions every token on all of the audio and all of the text so far, which is why these models have the best WERs. It can also look in the wrong place, which is where their failures come from.

Listen, Attend and Spell

One of the first attention recognizers for large-vocabulary speech, and the best known, is Listen, Attend and Spell, or LAS (Chan et al. 2015). Its encoder, the listener, is a stack of bidirectional LSTMs over 40-dimensional log-mel frames at 10 ms. A bidirectional LSTM runs one recurrent network forward over the frames and one backward. So each output vector summarizes the whole utterance as seen from that frame (§3.3).

Its decoder, the speller, is a two-layer LSTM that outputs characters. At step $u$ it has a state $\mathbf s_u$. It forms a context vector $\mathbf c_u = \sum_{t}\omega_{u,t}\mathbf h_t$, the attention-weighted average of the listener's outputs, and outputs $p(y_u \mid y_{\lt u}, \mathbf X) = \operatorname{softmax}(\operatorname{MLP}(\mathbf s_u, \mathbf c_u))$.

The paper's key engineering move was in the listener. An utterance is hundreds to thousands of frames, while its transcript is tens of characters. With a plain bidirectional LSTM listener the model converged slowly, and it was still worse after a month of training. LAS therefore stacks three pyramidal layers on top of a first bidirectional LSTM. A pyramidal layer reads pairs of neighboring outputs of the layer below, glued end to end, so it runs over half as many steps,

Here $\mathbf h^{(j)}_i$ is the output of layer $j$ at step $i$, and $[\cdot\,;\cdot]$ joins two vectors into one. Three layers divide the length by $2^3 = 8$, so a 10-second utterance of 1,000 frames reaches the speller as 125 vectors. That is the same factor of 8 as the FastConformer's convolutional front end in §36.1. It also cuts the cost of attention, which is proportional to the number of encoder frames times the number of output steps. Every recognizer since shortens the frame sequence early, and an interviewer may ask you why. The answer is these two reasons, an easier alignment to learn and cheaper attention.

Where attention fails on speech

Translation needs attention that can jump around, since word order differs between languages. Speech almost never needs that, and an unconstrained attention pays for freedom it does not use.

Chorowski et al. (2015) found the clearest symptom. Their first attention model scored each frame by its content alone. It reached 18.7 percent phoneme error rate on TIMIT, a small corpus of read English sentences. But it failed on utterances much longer than those it was trained on. Content alone cannot tell two occurrences of the same sound apart, so on a long input the attention can latch onto the wrong one.

Their fix lets the score see where the attention was at the previous step, which they called location-aware attention. The scorer is told "you were looking around here last time", and it can learn to prefer the frames just after that. The model then handled long inputs, at 18 percent on single utterances and 20 percent on utterances made ten times longer by joining several together. A smoothed normalization that keeps the weights from collapsing onto single frames brought single utterances to 17.6 percent.

Even with location awareness, attention recognizers have four characteristic failures, and every one of them returns in Whisper.

• Skipping. The attention jumps forward, and the words in between are never transcribed.
• Looping. The attention jumps back or stays put, and the decoder writes the same phrase again and again.
• Stopping at the wrong time. The decision to stop is the decoder emitting $\langle\text{eos}\rangle$, a choice made mostly by what the decoder has learned about text. Stopping early drops the end of the utterance. Not stopping produces invented text.
• No streaming. Every decoder step may attend to every frame, so the encoder must see the whole utterance before the first word comes out. §36.6 shows what it takes to work around this.

CTC and transducers cannot jump around or loop. Their alignments move through the audio and the transcript in order by construction, and every frame is accounted for. They still make errors, deletions included, but every word they write must be supported by some frames. That suggests combining the two kinds of model.

Joint CTC and attention

Recall CTC from §36.2. It adds a blank symbol $\varnothing$ that means "no new label here", and it lets every encoder frame output either a label or the blank. A frame-by-frame path spells a transcript after merging repeated labels and then removing blanks, so the path "c c ∅ a t" spells "cat". The probability of a transcript is the total probability of every path that spells it, computed by the forward algorithm. Every such path moves through the transcript in order and gives every frame a symbol.

The idea is to give one encoder both kinds of output. A CTC head and an attention decoder sit on the same encoder, and both are trained at once (Kim, Hori, and Watanabe 2016). The joint score was later used for decoding as well (Hori et al. 2017).

In plain words, the encoder has to serve two readers. The CTC reader insists that the frames spell out the transcript in order. The attention reader brings the full history of the labels. During training, the CTC gradient reaches the encoder through the forward-backward algorithm, which only ever considers in-order alignments. So the encoder learns representations in which the right frames light up for each label, and the attention has a much easier job finding them.

The effect is large. The joint model learned a clean diagonal alignment by the 5th epoch, while the pure attention model had not found one by the 9th (Kim et al. 2016). On the WSJ and CHiME-4 evaluation sets it improved character error rate by 5.4 to 14.6 percent relative over both CTC and attention alone. Of the three weights they tried (0.2, 0.5, 0.8), $\lambda_{\text{ctc}} = 0.2$ was the best.

At decoding time the CTC score guards against exactly the failures listed above.

• A hypothesis that loops contains labels that no frames support. Every CTC path for it must make some frames emit labels to which they give tiny probability.
• A hypothesis that stops early leaves frames with confident label probabilities, and those frames must now emit blanks.

Either way $\log p_{\text{CTC}}$ drops by many nats, the unit of a natural logarithm. The attention decoder, which leans heavily on what it knows about text, may still have scored the bad hypothesis well.

The scores are combined in log space, and that choice matters. In log space a single near-zero factor vetoes a hypothesis. A linear mixture of probabilities, $\lambda p_{\text{CTC}} + (1-\lambda)p_{\text{att}}$, has no such veto, since it stays large whenever either term is large.

The simplest way to use the joint score is rescoring. Beam search with the attention decoder alone produces a short list of its best hypotheses, the n-best list. Then each hypothesis gets its full CTC likelihood from the forward algorithm of §36.2, and the joint score picks the winner. Here is that as an interview problem.

import numpy as np

def joint_rescore(hyps, att_logp, log_probs, lam, blank=0):
    # Placeholder: trusts the attention decoder alone.
    scores = np.asarray(att_logp, dtype=float)
    return int(np.argmax(scores)), scores

Rescoring can only choose among hypotheses the attention decoder already proposed. A stronger use puts the CTC score inside the beam search itself, so that bad partial hypotheses are pruned before they spread (Hori et al. 2017). That needs a CTC score for the beginning of a transcript, a prefix $\mathbf g$, rather than for a finished one. The score is the total CTC probability of all transcripts that start with that prefix. The box below gives the recursion, and a first reading can take it on trust.

In plain words, the chance that the transcript starts with "the c" is a sum over the moment $t$ at which "c" first appears. Each term is the chance that everything before $t$ spelled exactly "the", times the chance that frame $t$ says "c". Beam search keeps $r^n$ and $r^b$ with each hypothesis and extends them in $O(T)$ time per new label. That is the same bookkeeping as the CTC prefix beam search of §36.2.

The cell checks the theorem against brute-force enumeration of every transcript on a 4-frame problem.

This one-pass joint decoding is the default recipe of the open-source ESPnet toolkit. It adds attention scores, CTC prefix scores, and optionally an external language model. The WeNet toolkit uses the two-pass form instead. CTC prefix beam search proposes an n-best list, and the attention decoder rescores it. That two-pass design returns in §36.6, where the CTC first pass streams. An interviewer who asks how to stop an attention recognizer from looping wants this answer, a CTC head trained jointly and its score added in log space.

Whisper, exactly

Whisper's authors went the other way. Instead of new machinery for alignment, they used an off-the-shelf encoder-decoder Transformer, so that the architecture would not confound their study of data. Every piece below is one you have met.

• The log-mel front end is from §35.3, and the convolution arithmetic from §0.8.
• Sinusoidal positions, fixed sine and cosine waves added to the inputs so the model knows each frame's place, are from §5.4.
• The pre-LN block, which applies LayerNorm before each sublayer inside the residual branch, is from §5.6. LayerNorm rescales each vector to zero mean and unit variance, then applies a learned gain and shift.
• Cross-attention, where the decoder's queries read the encoder's keys and values, is from §4.6.
• GELU is a smooth version of the ReLU activation, $\operatorname{GELU}(x) = x\,\Phi(x)$, where $\Phi$ is the standard normal cumulative distribution function.

What is new is how they are put together and the exact constants, which an interviewer may well ask for.

Follow one window through the shapes. The stem's convolutions use the output-length formula of §0.8, $\lfloor (T_{\text{in}} + 2p - k)/s\rfloor + 1$ for input length $T_{\text{in}}$, padding $p$, kernel width $k$, and stride $s$. The first keeps $\lfloor (3000 + 2 - 3)/1\rfloor + 1 = 3000$ frames. The second halves them, $\lfloor (3000 + 2 - 3)/2\rfloor + 1 = 1500$. So the subsampling factor is 2, much less than the 4 of Conformer encoders or the 8 of FastConformer encoders in §36.1, and each encoder vector covers 20 ms.

That 20 ms is the finest time step the decoder can point to, and it fixes the resolution of the timestamp tokens below. The 448-token decoder context is ample for 30 seconds. Fast speech is about 3 words per second, so a window holds roughly 100 to 150 tokens.

The 30-second window is a training choice, not a law. The training audio was cut into 30-second segments, so every input has one shape. That makes batching simple and lets the encoder use one fixed table of 1,500 positions. The price is padding on short inputs and a sliding procedure for long ones, both of which come back below.

Counting Whisper's parameters

With the architecture pinned down, the parameter count is bookkeeping in the style of §5.6. Interviewers use it to check that you know where a model's weights live, and the answer here has a surprise in it.

Compare the counts of §5.6, $12d^2 + 13d$ and $16d^2 + 19d$ for blocks with a bias on every projection. The differences, $d$ per encoder block and $2d$ per decoder block, are exactly Whisper's missing key biases.

def whisper_params(d, n_layer, n_vocab, n_mels, n_text_ctx=448, n_text_layer=None):
    # Placeholder: 12 d^2 per block, the usual rule of thumb, which ignores everything else.
    L_dec = n_layer if n_text_layer is None else n_text_layer
    enc, dec = 12 * n_layer * d * d, 12 * L_dec * d * d
    return enc, dec, enc + dec

The cell prints the whole family, and how much of each decoder is the token embedding.

The embedding's share of the decoder falls from two thirds in tiny to 7 percent in large. The reason is that $Vd$ grows linearly in $d$ while the blocks grow like $Ld^2$. Turbo's 806,958,080 trainable parameters plus its 1,920,000-number sinusoidal table give 808,878,080, the 809M on its model card.

The block below rebuilds the architecture in PyTorch. It counts every size with the tensors placed on PyTorch's meta device, which records shapes without allocating memory, so even large costs nothing. Then it runs a tiny copy on a 30-second input of random numbers to check the shapes and the causal mask.

Weak supervision at scale

If the architecture is standard, the robustness in the opening example must come from somewhere else. The paper's answer is the training set. Before Whisper, the strongest English recognizers followed one of two recipes.

• Some were trained on curated, human-checked corpora. Examples are LibriSpeech's 960 hours, or the 5,140 hours that SpeechStew assembled from seven public datasets (Chan et al. 2021).
• Others were pretrained without labels and then fine-tuned on a small curated set, as wav2vec 2.0 is.

The first recipe gives clean labels from a narrow slice of the world. The second gives a robust encoder, but the part that maps it to text is learned from the narrow slice. That, the paper argues, is where the robustness is lost.

Weak supervision means training on labels that exist for some other reason. The internet holds an enormous amount of audio paired with text that someone typed for viewers, such as subtitles, captions, and show notes. These transcripts are noisy and sometimes misaligned. They are formatted every possible way, sometimes in the wrong language, and sometimes not written by a person at all.

Whisper's bet was that 680,000 hours of that beat a few thousand hours of carefully checked labels. Its data pipeline is the part of the paper most worth learning, because every later system copied some version of it.

How much did scale matter? The paper trained medium-sized models on subsets of 0.5 to 8 percent of the data and compared them with the full set.

A power law $\text{WER} \propto D^{-k}$ in the number of hours $D$ says that each doubling of data multiplies WER by the same factor $2^{-k}$. On log-log axes it is a straight line of slope $-k$. So the local exponent between two sizes is $k = -\Delta\log\text{WER}/\Delta\log D$, the same reading as the scaling laws of §9.1. For example, going from 3,405 to 6,811 hours took English WER from 30.5 to 19.6, so $k = \ln(30.5/19.6)/\ln 2 = 0.64$.

The English exponent decays from 0.64 to 0.04, while the multilingual one holds near 0.33 until the last step. The paper offers two readings and leaves the question open. English WER may be saturating as it approaches human level. Or the models may be too small and too briefly trained to use the full set.

Across languages, Figure 3 of the paper fits the log of each language's Fleurs WER against the log of its training hours, with $R^2 = 0.83$. WER halves for every 16-fold increase in data, an exponent of $1/4$, since $16^{-1/4} = 1/2$. Most languages had less than 1,000 hours. So, as the paper points out, targeted data for those languages is the cheapest way to improve them.

One more finding matters for design. For small models, training on many languages and tasks at once hurt English, which is called negative transfer. For the largest model it helped. Multilingual, multitask training is a bet that pays off only at scale, and an interviewer may ask you to say why. A small model has too little capacity to share, so each language crowds the others out.

Effective robustness

The robustness claim of the opening example has a precise form, borrowed from image classification. It separates two questions. How good is a model away from home? And how much better is it there than its score at home predicts?

In plain words, overall robustness asks how good the model is away from home. Effective robustness asks how much better it is away from home than its score at home would lead you to expect. The second removes the confound that a better model is usually better everywhere.

Effective robustness was defined for ImageNet accuracy as $\rho = \text{acc}_2 - \beta(\text{acc}_1)$, with $\beta$ a straight line fitted after a logit transform of both axes (Taori et al. 2020). Accuracy is higher-is-better, so for WER the sign flips, as above.

The Whisper paper uses the concept without publishing a fitted $\beta$, so the cell fits one itself. It draws a line in log-log space through the eleven models of the paper's Table 8 that saw only audiobook speech. Most of them are self-supervised encoders fine-tuned on LibriSpeech, wav2vec 2.0 (§35.4) and HuBERT (§35.5) among them. The cell's out-of-distribution average is over the 12 sets that are not LibriSpeech, so it differs a little from the 13-set average of the opening example.

The audiobook models scatter about the curve by 5.7 points. The wav2vec 2.0 model of the opening example sits 2.5 points below it, an ordinary member of its family. Whisper large-v2 sits 20.2 points below it, three and a half times the scatter. Its overall robustness is 13.4 percent against the baseline's 33.6.

The paper finds the same at every Whisper size. Even tiny, at 6.7 percent on test-clean with the beam search and temperature fallback described below, is roughly competitive out of distribution with the best LibriSpeech model. In practice, this is the number to ask for when someone shows you a new recognizer's benchmark score. A model that moves along the baseline curve improves away from home only as much as its home score predicts.

One caution before taking every number at face value. All of these WERs are computed after Whisper's own text normalizer. It lowercases, removes punctuation, and standardizes spellings and numbers before scoring. It was built by inspecting where naive WER penalized Whisper's formatting.

The paper checks it against an independently built normalizer. The effects are similar on most datasets, but Whisper benefits more on a few (WSJ, CallHome, Switchboard). How normalization moves WER, sometimes by half, is the subject of §36.5.

Pseudo-labels and self-training

Whisper's labels were weak but human, typed by someone for some other purpose. Its successors went one step further and let a model write the labels. The 4 million hours that large-v2 transcribed for large-v3 are one instance. The method now supplies most of the training transcripts at the frontier. §35.4 used it in passing, as the last step of a recipe for languages with little labeled data. Here it gets its full statement.

In plain words, the teacher turns cheap audio into labeled data, and the filter throws away the labels it probably got wrong. The student sees noisier inputs than the teacher did and must still produce the teacher's labels. That forces it to generalize, so it can end up better than its teacher rather than a copy of it. Without noise and without a filter, a student of the teacher's size has little to learn beyond the teacher's own decisions, mistakes included.

That last point is the method's central failure, confirmation bias. A teacher's errors are not random. They concentrate on the accents, noise conditions, and rare words it already handles badly, and a student trained on them learns them as truth.

The filter fights this by trading quantity for label quality, and the trade has an optimum. Keep everything, and the student learns too many wrong labels. Keep only the most confident labels, and it learns from too few, and only from the easiest utterances.

The cell makes the trade exact in a small simulation where every quantity can be computed. Each "utterance" is a point in $\R^{400}$ from one of two classes, with class means $\pm\boldsymbol\mu$, $\lVert\boldsymbol\mu\rVert = 1.5$, and unit Gaussian noise.

• The teacher's confidence $c$ is uniform on $[0.5, 1]$ and calibrated, so a pseudo-label is right with probability $c$.
• The student is the nearest-centroid classifier, which labels a point by the closer of the two class averages. Its exact test error is a sum of two Gaussian tail probabilities.

There are two cases. In the random case the teacher's errors fall anywhere. In the systematic case they fall only on one side of a nuisance direction, the positive side for one class and the negative side for the other. That stands in for a teacher that fails on one accent.

Both curves are U-shaped. With random errors the student is best near $\tau = 2/3$, at 9.5 percent error. Keeping everything costs only half a point. A strict filter costs much more, 22 percent at $\tau = 0.98$, because it throws away nearly all the data.

Systematic errors behave differently, and this is confirmation bias in miniature. Keeping everything now costs 12.2 percent against a best of 9.6, and the best threshold moves up, to $\tau = 0.75$. Random errors only dilute the signal, and more data averages them away. Systematic errors tilt the direction the student learns toward the nuisance, and no amount of data removes a tilt.

Real teachers err systematically. That is why production pipelines filter hard at first and relax the filter as the teacher improves. An interviewer who asks how you would set the filter wants this trade, and a held-out human-labeled test set, sliced by accent and noise, to measure it.

One decoder, many tasks

A recognizer in production does more than recognize. It must detect the language, find where speech is, mark times, and sometimes translate. Whisper does all of this with one decoder. It writes the task into the decoder's input as special tokens, reserved vocabulary entries that carry instructions rather than text (§2.5). Chat models later used the same device for their instruction formats.

In plain words, the forced prompt tells the decoder which job to do. From then on the decoder writes a transcript in which times are just more tokens.

Language identification comes for free. When the language is not given, look at the decoder's distribution over the 99 language tokens right after <|startoftranscript|>. That distribution is a language classifier. The same position's probability of <|nospeech|> is a score for whether anyone is speaking. That probability, read before any text is generated, is the no-speech probability that long-form decoding uses below.

The widget shows the forced prompt and the decoder's output for one 30-second window. Its last sentence starts at 24.30 s and runs past the end of the window.

The answer is five, a start and an end for each of the two complete sentences and only a start for the cut one. The reference implementation starts the next window at 9.86 s, the end of the last complete segment. It drops the unfinished segment and lets the next window, which now holds the whole sentence, transcribe it. The rule is short enough to implement in an interview.

def build_prompt(lang, task, timestamps, prev=None):
    raise NotImplementedError

def timestamp_token(seconds):
    raise NotImplementedError

def parse_segments(tokens, offset=0.0):
    raise NotImplementedError

Keeping timestamps well formed

The parser assumes the decoder wrote a well-formed sequence, with segments opened and closed by timestamps in increasing order. Nothing in a softmax guarantees that. A decoder left to itself can write text before any timestamp, two segment ends in a row, a time earlier than the last one, or a segment that ends where it starts.

The reference decoder rules all of these out by hand. Before every step it sets the logits of forbidden tokens to $-\infty$, so their probability is exactly zero. It does this in greedy decoding and in beam search alike, so every hypothesis it ever scores is well formed. Masking logits by rules is a general tool, used wherever an output must follow a grammar, such as JSON from a language model.

In plain words, rules 2 and 3 make the output a list of segments, each opened by one timestamp and closed by a later one, in time order. A closing timestamp must be followed by the next segment's opening (which may share its time) or by the end, and an opening must be followed by text. The clause "or does not exist" makes the very first timestamp an opening.

Rule 3's strictness forbids zero-length segments such as <|0.00|> " Thanks" <|0.00|>. A window whose segments all ended at 0.00 would send long-form decoding back to the same position forever, the infinite loop that the reference code's comment on this rule mentions. Rule 4 keeps the model from skipping the first words of a window.

Rule 5 is the subtle one. The model's belief that "a timestamp comes next" is spread over up to 1,501 tokens. Say 80 allowed timestamps have probability 0.01 each and the best word has 0.15. The timestamps together have 0.8, yet each one loses to the word. An argmax over single tokens would keep choosing words and never close a segment. Rule 5 first decides between the two groups, timestamps and everything else, by total probability, and only then lets the argmax choose within the winner.

import numpy as np

def apply_timestamp_rules(logits, tokens, sample_begin, ts_begin, eot, max_initial=50):
    # Placeholder: applies no rules.
    return np.array(logits, dtype=float)

One clip through Whisper, end to end

Here is everything so far, applied to one input. Take the 7-second English clip of the worked example above, two sentences from 0.40 to 3.12 s and from 3.60 to 6.86 s. The three cells follow it from the waveform to a scored transcript, with the shapes and numbers at every stage.

The first cell builds a synthetic voice, a buzz at about 120 Hz with its harmonics, switched on during the two sentences, plus faint room noise. It then computes the log-mel spectrogram with Whisper's window, hop, and scaling. For brevity it uses simple triangular mel filters. §35.3 reproduces Whisper's exact filterbank.

The second cell runs the convolutional stem with random weights at a toy width of 64. Random weights are enough to check the shapes. It also locates the speech in encoder frames, which is where a trained decoder's timestamps must point.

The third cell plays the decoder's part. We have no trained model here, so it takes the output tokens a trained decoder writes for this clip. The cell turns the tokens into timed segments, then scores the transcript with the word-level edit distance of §36.1, before and after a minimal normalization.

The last line is the reason §36.5 spends so long on normalization. The same transcript scores 55.6 or 11.1 percent depending only on whether capitals and punctuation are compared. Whisper writes formatted text, so a test set with lowercase references punishes it unless both sides are normalized first.

Long-form transcription

Whisper sees 30 seconds at a time, but meetings, lectures, and podcasts last hours. The paper's procedure is a loop. Transcribe a window, move the window to where the transcript says it ended, and repeat. That makes every window depend on the one before, through both the start position and the previous-text prompt. So one bad window can derail the rest. The paper found that reliable long-form transcription needed a set of heuristics, which the reference implementation ships as its defaults.

The compression ratio is a cheap loop detector that needs no model. A repetition loop is the most predictable text there is. A compressor such as zlib, the library behind gzip, replaces each repeat with a short reference back to the first copy. So a looping text compresses enormously. Ordinary text, with its varied words, barely compresses at the length of one window.

A full window of normal speech sits near 1.6. A short sentence is below 1, because zlib's fixed 6 bytes of header and checksum dominate very short inputs. Three repeats are still unremarkable, and long loops reach 5 to 14. The threshold of 2.4 sits between ordinary windows and windows dominated by repeats.

It has known blind spots. Legitimately repetitive speech, such as a chant or a list of numbers read aloud, can trip it. A hallucination that does not repeat passes it.

The widget plays out the fallback policy for one window. Pick a case or set the first decode's statistics, then press Decode and watch each retry.

Two cases deserve a second look.

• In "confident hallucination", the no-speech probability is high, but the text, a stock phrase, is confident. So the silence rule never fires. Sampling breaks the repetition, and a fluent sentence that nobody said is accepted.
• In "mumbling", every attempt is unconfident. So the policy runs all six temperatures and returns the last, most random sample.

The heuristics trade one kind of error for another. That is why production systems put a separate voice activity detector, a small model that marks where speech is, in front of the recognizer (§36.6). Now implement the policy yourself.

import zlib

def compression_ratio(text):
    b = text.encode("utf-8")
    return len(zlib.compress(b)) / len(b)          # placeholder, upside down

def decode_with_fallback(decode_fn, temperatures=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0), logprob_threshold=-1.0,
                         compression_ratio_threshold=2.4, no_speech_threshold=0.6):

Hallucination on silence

Every recognizer makes errors. Whisper's most notorious ones are of a different kind. Given a stretch of silence, music, or noise, it sometimes writes fluent text that nobody said. Typical outputs are "Thank you for watching", a request to subscribe, or a sentence unrelated to anything in the audio. This is called hallucination.

To see why it happens, look at what the decoder is. It is a language model over internet transcripts, conditioned on the audio through cross-attention. The paper calls it an audio-conditional language model. When the audio carries strong evidence, the evidence decides the words. When it carries none, only the language model is left to decide them. Its favorite things to say over a silent stretch of an internet video are the stock phrases that captioners type over silent stretches of internet videos.

A toy model makes this exact. Take a bigram language model, which predicts each word from the one before it (§3.1). Fit it on a tiny "caption corpus" in which stock phrases are common. Then let the audio add evidence for the token actually spoken in each output slot, with a strength $w$,

where $y^\star_u$ is the $u$-th token of the true transcript ($\langle\text{eos}\rangle$ after its last word), and $\mathbb 1[\cdot]$ is 1 when its condition holds and 0 otherwise.

In plain words, the next token's probability is the language model's opinion, boosted by a factor $e^w$ for the token the audio supports. This is a language-model prior times an acoustic evidence term, the same split as the noisy-channel recognizer of §36.1. The strength $w$ plays the part of how loud and clear the speech is, and silence is $w = 0$. Giving the evidence to output slot $u$ quietly assumes the decoder knows where it is in the audio. That is the alignment that cross-attention has to learn.

Four regimes appear.

• With no evidence ($w = 0$) the decoder writes the prior's favorite path, "thank you thank you ...", until it hits the length limit. Its average log-probability is high, $-0.41$, because a language model is confident about its own favorite phrase. Only the compression ratio, 5.67, flags it.
• With weak evidence ($w = 0.5$) it writes the stock phrase but stops where the audio ends.
• Just below $w = 0.647$ it writes "the river is high". That is fluent, plausible, and wrong, because "is" follows "river" more often in the corpus than "rose" does.
• Above $w = 0.647$ it is correct. The theorem after the next exercise derives that number exactly.

The wrong-but-fluent regime is the dangerous one in practice. Its compression ratio and confidence look normal, so no repetition detector or confidence check can see it.

import numpy as np

def greedy_decode(logP, y_true, w, bos, eos, max_len=30):
    raise NotImplementedError

def evidence_threshold(logP, y_true, bos, eos):
    raise NotImplementedError

In plain words, the audio must outvote the language model at its single strongest objection. The step that sets the threshold is the token the prior likes least compared with its favorite alternative. That is usually a rare word, a name, a number, or a technical term.

Three practical lessons follow. Hallucination and domain errors are two faces of one thing. Fine-tuning or prompting with the right vocabulary helps, because it lowers the prior's objection. And everything that weakens the evidence, such as noise, distance from the microphone, or hesitant speech, moves a recognizer toward its prior.

The measurement that made this a public issue came from outside the lab.

Why language models in general invent content, and how to measure and reduce it, is the subject of §19.8. For speech the mitigations are specific, and each works for a reason you can now state.

Making Whisper cheaper

Whisper's costs come from two places. The encoder always processes a full 30-second window, so a 3-second voice command pays for 30 seconds of encoder compute, 27 of them padding. And the decoder is autoregressive, writing one token per step, each step waiting for the last.

At batch size 1, every new token reads all of the decoder's weights from memory to do very little arithmetic on them. So decoding is memory-bound (§20.1). Its time is set by the bytes moved per token, not by the arithmetic. The cross-attention keys and values are computed once per window and cached. For Whisper large they take 245.8 MB in 16-bit precision (§4.6), and they too are read at every step.

Even at its compute-bound best the encoder takes a few milliseconds per window, while 100 decoded tokens take at least 55 ms. The decoder dominates the time by an order of magnitude, so that is where to cut. OpenAI's large-v3-turbo, released at the end of September 2024, does exactly that. It prunes large-v3's decoder from 32 layers to 4 and fine-tunes, with the encoder untouched.

Keeping the encoder matters for two reasons. The encoder carries the acoustic robustness that the weak supervision bought. And its cost is paid only once per window.

How much faster is turbo in practice? RTFx, from §36.1, is seconds of audio processed per second of compute. On the March 2026 Open ASR Leaderboard, turbo averages 7.83 percent WER at RTFx 200, against large-v3's 7.44 at RTFx 146. That speedup is measured at batch 64, where decoding is far less memory-bound, so it is smaller than the 5.3 times bound at batch 1. An interviewer who asks why turbo cut the decoder and not the encoder wants exactly this arithmetic.

Distil-Whisper is sequence-level knowledge distillation, a small student trained on a large teacher's outputs (§20.4). Its WER filter does the job a quality filter does for any pseudo-labeled corpus.

Its speculative mode is the speculative decoding of §20.5. A small draft model proposes several tokens, and the large model checks them all in one pass. Here it has a convenient twist. The draft shares the teacher's encoder, so the audio is encoded once for both. The acceptance rule keeps the output distribution exactly that of the large model, so greedy outputs are identical, token for token.

The paper also notes the price of a tiny decoder. On the Meanwhile set, one speaker with many uncommon words, Distil-Whisper is worse. That fits the evidence threshold of the last section. A smaller decoder is a weaker language model for rare words, and rare words are where the margin the audio must overcome is largest.

Whisper's timestamps mark segments, not words. Word-level times come from a second alignment step, either a separate CTC model or the decoder's own cross-attention.

Speech-LLMs, the 2026 frontier

A speech-LLM is a recognizer whose decoder is a pretrained large language model. §35.6 built the general version, an audio language model with three parts.

• An audio encoder turns the audio into frames.
• An adapter shortens the sequence of frames and projects each one into the LLM's embedding space, so the frames look like extra input tokens.
• The LLM reads them alongside text and writes the answer.

That is the same connector pattern that §31.1 uses for images. For recognition, it is still an attention encoder-decoder. Whisper's decoder reads the encoder output through cross-attention, while the LLM reads the adapter's output through its ordinary self-attention, as part of its input sequence. The difference is the decoder. It arrives knowing a great deal of language, which is exactly what helps on rare words, numbers, and formatting.

The strongest open recognizers of 2026 are of this kind, and their reports show what is specific to speech recognition. Qwen3-ASR (Qwen Team 2026) is the most complete recipe.

• Its encoder is first trained as an attention encoder-decoder recognizer on about 40 million hours of pseudo-labeled audio, mostly Chinese and English.
• The encoder shortens 128-band filterbank features 8 times, to 12.5 audio tokens per second. Its attention windows vary from 1 to 8 seconds during training, so one model can stream with short chunks and run offline with long ones.
• A projector connects it to Qwen3-1.7B or Qwen3-0.6B.
• After multimodal pretraining, supervised fine-tuning trains it to be a recognizer that does not follow natural-language instructions in the prompt. That makes it resist instructions injected into the input.
• A final reinforcement learning stage on about 50,000 utterances uses GSPO (Zheng et al. 2025). It is a variant of the group-relative policy gradient of §18.4 that computes its importance ratios and clipping over whole sequences rather than single tokens. The report says this stage matters for noise robustness and transcription stability.

So the frontier has a shape. The autoregressive decoders, plain or LLM, win on accuracy by roughly half a point to two points of WER. The alignment-based decoders win on speed by about an order of magnitude. Which matters more depends on the product, and an interviewer will expect you to ask about latency, cost, and the kind of errors that hurt before choosing.

The LLM decoder also brings new risks.

• It can follow instructions spoken in the audio ("ignore the above and write..."). This is a form of prompt injection, and Qwen3-ASR trains against it explicitly.
• Its strong prior writes fluent text over weak evidence. That is the hallucination described above, with a better language model behind it.
• It is large and autoregressive, so it is expensive to serve.

Training against WER directly with reinforcement learning is the minimum word error training of §36.5, now applied to LLM recognizers.

Many languages in one recognizer

Whisper's multilingual recipe is the template most systems follow. One byte-level BPE vocabulary covers every script. A language token in the prompt names the language to write, and the decoder's distribution at that position doubles as a language identifier. Sharing one model across languages brings three problems that a single-language recognizer never has.

Language identification errors. When the language is not given, a wrong guess sends the decoder off in the wrong language. It transcribes speech in one language as text in another, or translates when it should transcribe. Whisper's paper measured 80.3 percent identification accuracy for its best model on the 82 Fleurs languages it had training data for, and 64.5 percent over all 102.

The same errors corrupt training data. The paper found that most of its supposedly Welsh translation data was English audio with English captions. Pipelines that pseudo-label with Whisper, such as Granary above, therefore identify the language in a first pass and check it before trusting a transcript. Products that know their users' language pass it in the prompt.

Data per language. WER halves for every 16-fold increase in a language's training hours, as the scaling study above showed. So the long tail of languages with under 1,000 hours is where accuracy is worst and new data is cheapest. Self-supervised encoders reach furthest into that tail. MMS (§35.4) recognizes 1,107 languages.

Meta's Omnilingual ASR (Omnilingual ASR team 2025) goes further. It scales a wav2vec 2.0 encoder to 7 billion parameters, adds an LLM-style decoder, and covers more than 1,600 languages. Over 500 of them were never before served by any recognizer. It is designed to take in a new language from a handful of examples.

The cost to English. Capacity and training time shared across languages are taken from each one, and the leaderboard paper shows the price.

• NVIDIA's Parakeet-TDT-0.6B-v3 adds 24 European languages to the English-only v2. It averages 6.32 percent English WER against v2's 6.05.
• Canary-1B-v2 goes from 4 languages to 25. It averages 7.15 against Canary-1B's 6.50.
• Omnilingual ASR's 7B LLM model (v2) ranks only 53rd on English, at 8.14. Yet on the paper's multilingual sets (German, French, Italian, Spanish, and Portuguese) it averages 4.39 percent, ahead of Whisper large-v3's 4.81.

This is the trade that Whisper's study found as negative transfer in its small models. The study's other half, positive transfer in the largest, suggests that capacity shrinks the cost. Still, a product that serves one language may be better off with a model built for it.

The three families side by side

Here is the chapter's comparison of the three families, complete.

Read more

• Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. The Whisper paper. Its parts on data, the model, and the multitask format, on effective robustness, on data scaling, and on long-form heuristics are the ones this section draws on. The reference implementation is short and worth reading, especially model.py, decoding.py (the timestamp rules), timing.py (word times by DTW), and transcribe.py. Taori et al. (2020), Measuring Robustness to Natural Distribution Shifts in Image Classification, defines effective robustness.
• Chan et al. (2015). Listen, Attend and Spell. The pyramidal listener and the attention speller, with an honest comparison against a production hybrid.
• Chorowski et al. (2015). Attention-Based Models for Speech Recognition. Why content-only attention fails on long utterances, and location-aware attention.
• Kim, Hori, and Watanabe (2016). Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning. The joint loss. Hori et al. (2017), Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM, adds joint decoding with the CTC prefix score.
• Gandhi, von Platen, and Rush (2023). Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling. Pseudo-labels with a WER filter, a 2-layer decoder, and speculative decoding with identical outputs. Park et al. (2020), Improved Noisy Student Training for Automatic Speech Recognition, is the self-training recipe for ASR, and Koluguri et al. (2025), Granary: Speech Recognition and Translation Dataset in 25 European Languages, a pseudo-labeling pipeline at scale.
• Koenecke et al. (2024). Careless Whisper: Speech-to-Text Hallucination Harms. The measurement of hallucinations, their harms, and their link to stretches without speech.
• Qwen Team (2026). Qwen3-ASR Technical Report. A complete 2026 speech-LLM recipe, from a 40-million-hour attention encoder-decoder encoder to reinforcement learning with GSPO.
• Srivastav et al. (2025). Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation. One normalizer and one hardware setup for 86 models, the accuracy and speed trade-off between decoder families, and the English cost of multilingual coverage. Omnilingual ASR team (2025), Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages, is the far end of that coverage.

Watch next

▶

▶