Chapter 36 · Section 36.6 · 100 min

Streaming and production ASR

This section is about streaming speech recognition, which turns speech into text while the person is still talking. It is also about the systems that surround every deployed recognizer. Live captions must show each word as it is spoken. A voice assistant must answer about as quickly as a person would. Both need a recognizer that writes each word a fraction of a second after it is said, not after the recording ends.

Every recognizer in this chapter so far reads the whole recording first. Its encoder, the network that turns spectrogram frames into a sequence of vectors, is usually a Conformer, a stack of blocks that mix self-attention with convolution (§36.1). Self-attention lets every frame read every other frame, including frames from later in the recording. Whisper reads a whole 30-second window this way before its decoder writes a single word (§36.4). A transducer, the model of §36.3 that reads one encoder frame at a time and decides whether to emit a label, could write as the audio arrives. But it can do so only if its encoder never reads future audio. Reading everything first is fine for an archive and useless for a conversation.

How fast is "as quickly as a person"? Stivers et al. (2009) timed the gap between yes-or-no questions and their answers in recordings of natural conversation in ten languages, from Danish and Japanese to Tzeltal and Yélî Dnye. In every language the most common gap was between 0 and 200 ms. People start planning a reply before the question ends. A system that waits for silence and only then starts to think is already late.

Here is the plan. First we pin down what latency means and add up a real budget. Then we build encoders that see only a bounded amount of future audio. We prove why letting every layer peek ahead is a trap while chunks are not, and we make chunk-by-chunk inference exact with caches. Then we make the decoder stream too, in three different ways.

The rest of the section covers what surrounds the recognizer in production. A voice activity detector decides when someone is speaking, and an endpointer decides when they have finished. Diarization labels who spoke when. On-device models run on the phone behind a wake-word detector. Servers transcribe thousands of streams at once, and sizing them is arithmetic you can do on a napkin. Each of these is a standard system-design interview topic.

What latency means

The word "latency" covers several different quantities. Mixing them up is the most common mistake in discussions of streaming recognition, so we name each one.

Start with a tiny example. Say the word "mom" ends at 1.00 s and appears on screen at 1.25 s. Part of that 250 ms is spent waiting for audio after the word, which the model needs before it can decide. Part is spent computing. And part is the model choosing to wait a little longer before it commits. Each part has a different cause and a different fix.

First, the units. Speech enters the recognizer as feature frames, one log-mel spectrum every 10 ms. The first layers of the encoder subsample them, merging several frames into one. So one encoder frame covers $\Delta_{\text{enc}}$ of audio, 40 ms after 4x subsampling and 80 ms after 8x (§36.1). In this section encoder frames are numbered from 0, as in code. Frame $i$ covers the audio interval $[i\Delta_{\text{enc}}, (i+1)\Delta_{\text{enc}})$ , so frame 3 at 40 ms covers 120 to 160 ms.

📐 Latency in streaming ASR

Let the encoder output at frame $i$ depend on input frames up to $j_{\max}(i) \ge i$ .

The algorithmic latency of frame $i$ is $(j_{\max}(i) - i)\,\Delta_{\text{enc}}$ . It is the audio that must arrive after frame $i$ 's own audio ends before its output can exist. The architecture sets it (attention masks, convolution padding, chunking), and no hardware removes it.
The computational latency is the wall-clock time to process audio once it has arrived. The real-time factor RTF is processing time divided by audio duration (§36.1), so a chunk of duration $t_{\text{chunk}}$ takes $\mathrm{RTF}\cdot t_{\text{chunk}}$ to process. It depends on the hardware, the model size, batching, and load. It must stay below $t_{\text{chunk}}$ , or the system falls further behind with every chunk.
The emission delay is how long after the end of a word the model emits it, beyond what the first two force. The end of the word is located by forced alignment, which finds where each word of a known transcript sits in the audio (§36.1). Emission delay is learned behavior. FastEmit, a training change that pushes a transducer to emit labels sooner, reduces it (§36.3).
The finalization latency is the time from the end of the user's speech to the final transcript. It is the endpointer's wait plus any second pass.
The time to first token (TTFT) is the time from the start of the request to the first emitted token.

Outputs that may still change are partial results. Outputs that never change are final.

Which of these can a faster GPU reduce? Only the computational latency, plus the compute that sits inside the finalization latency and the time to first token, such as a second pass. The attention mask fixes the algorithmic latency, training fixes the emission delay, and a policy fixes how long the endpointer waits.

Papers usually report latency per word. It is the time from a word's forced-aligned end to the moment the system outputs it, averaged over words. Products care about the slow cases too, so they also report the 50th and 90th percentiles. When an interviewer asks how you would cut latency, first say which of the five you are attacking.

One more piece of arithmetic finishes the budget. A chunked encoder cuts the audio into chunks of $n_{\text{chunk}}$ encoder frames, each $t_{\text{chunk}} = n_{\text{chunk}}\Delta_{\text{enc}}$ seconds long. It processes a chunk only once all of it has arrived. A word can end anywhere inside its chunk, so it waits anywhere from 0 to $t_{\text{chunk}}$ for the chunk to fill. On average it waits $t_{\text{chunk}}/2$ .

✏️ A latency budget for a voice assistant

Take an on-device transducer with 80 ms encoder frames, chunks of 4 frames (320 ms), and no lookahead. It runs at an RTF of 0.1 on the phone, so a 320 ms chunk takes 32 ms to process. For its emission delay we take 40 ms. That is the 90th-percentile delay FastEmit reached on LibriSpeech (§36.3), measured from the end of speech to the last token, so the typical utterance does better. A word then appears on screen after $160 + 32 + 40 = 232$ ms on average.

The end of the turn costs more. Sainath et al. (2020) measured a median endpointer latency of 430 ms for their best on-device system. That is the time from the end of speech to closing the microphone. Their second pass took a median of 58 ms on a Pixel 4. So the final transcript is ready about 490 ms after the user stops. Only then can a language model start composing the answer, and a speech synthesizer start speaking it.

The cell keeps the budget explicit. Change an assumption and see what moves.

Python · cell 1

The budget shows where the time goes. Chunking costs a few hundred milliseconds, but the largest single item is deciding that the user has finished. A recognizer cannot know that someone has stopped until some silence has passed. A pause inside a sentence ("call... mom") sounds the same as the end of one. That is why the spoken dialogue systems of §37.5 predict the end of a turn from what has been said, instead of waiting for silence. In a design interview about a voice product, start the latency discussion at the endpointer, not at the encoder.

Encoders that do not look ahead

An encoder can stream if each of its output frames depends on a bounded amount of future input. Two parts of a Conformer block read future frames, its convolution and its self-attention. Both must be constrained.

Causal convolutions

A one-dimensional convolution with a kernel of $k$ taps computes each output as a weighted sum of $k$ neighboring inputs. In symbols, $h_i = \sum_{m} w_m x_{i+m}$ over a window of offsets $m$ (receptive fields are in §0.8). The usual "same" padding centers the window. For $k = 3$ the offsets are $-1, 0, 1$ , so output 5 reads inputs 4, 5, and 6, and input 6 lies in the future. In general a centered window of odd width $k$ , with offsets $m \in \{-\tfrac{k-1}{2}, \dots, \tfrac{k-1}{2}\}$ , reads $\tfrac{k-1}{2}$ future frames.

A causal convolution pads $k-1$ zeros on the left only and uses the offsets $m \in \{-(k-1), \dots, 0\}$ . After renaming the taps it reads

h_i = \sum_{m=0}^{k-1} w_m\, x_{i-m},

so output $i$ reads inputs $i-k+1$ through $i$ . With $k = 3$ , output 5 now reads inputs 3, 4, and 5. The parameters are exactly the same. Only the side of the padding changes, and it matters a great deal. A centered kernel of width 31, a size common in Conformer implementations, reads 15 future frames in every block. At 40 ms per frame that is 600 ms of audio.

Other parts of a standard recognizer use the future quietly.

Feature normalization. Normalizing the features by the mean and variance of the whole utterance uses audio that has not arrived yet. Streaming models drop it or use running statistics.
Batch normalization. The Conformer's convolution module normalizes with statistics computed over whole sequences during training (§0.7), so training sees numbers a streaming model never could. The cache-aware models below replace it with layer normalization, which normalizes each frame on its own.
The subsampling stem. The strided convolutions at the bottom of the encoder read ahead too, so they are made causal as well (Noroozi et al. 2023).

How lookahead accumulates across layers

Self-attention has no kernel, so how far it looks ahead is whatever its attention mask allows. The obvious first idea is to let each frame attend to the whole past and to $r$ future frames, in every layer. We call this per-frame lookahead. One such layer waits for $r$ frames. A stack of $L$ layers waits for far more, because the frame a layer peeks at has itself peeked further ahead in the layer below.

To state this precisely we need one word. The reach $f(i)$ of a layer is the last frame of the layer below that output frame $i$ reads.

∴ Lookahead accumulates across layers

Let every layer $\ell = 1, \dots, L$ compute its output at frame $i$ from the outputs of layer $\ell - 1$ at frames $j \le f(i)$ , where the reach $f$ is nondecreasing with $f(i) \ge i$ . Then the output of layer $L$ at frame $i$ depends only on input frames $j \le f^{L}(i)$ , the $L$ -fold composition $f(f(\cdots f(i)))$ . If every allowed connection carries signal, it does depend on frame $f^L(i)$ .

Per-frame lookahead of $r$ frames, $f(i) = i + r$ , gives $f^L(i) = i + Lr$ . The lookahead grows linearly with depth.
Chunked attention, in which frame $i$ sees up to the end of its own chunk, has $f(i) = e(i) = (c(i) + 1)\,n_{\text{chunk}} - 1$ with chunk index $c(i) = \lfloor i/n_{\text{chunk}}\rfloor$ . It gives $f^L(i) = e(i)$ for every $L$ . The lookahead is at most $n_{\text{chunk}} - 1$ frames at any depth.
Chunks that also see $n_{\text{right}}$ frames past their end in every layer, with $1 \le n_{\text{right}} \le n_{\text{chunk}}$ , give $f^L(i) = e(i) + (L-1)\,n_{\text{chunk}} + n_{\text{right}}$ . A centered kernel-3 convolution placed before the chunked attention of each layer is the case $n_{\text{right}} = 1$ . Placed after it, the growth is the same, one chunk per layer.

∂ Proof by induction on depth

For $L = 1$ the statement is the definition of $f$ . Suppose the output of layer $L$ at any frame $j$ depends on inputs up to $f^L(j)$ . The output of layer $L+1$ at frame $i$ reads layer- $L$ frames $j \le f(i)$ . Each of those depends on inputs up to $f^L(j) \le f^L(f(i)) = f^{L+1}(i)$ , because a composition of nondecreasing functions is nondecreasing. The frame $j = f(i)$ is among those read, and it depends on input $f^L(f(i))$ , so the bound is attained when connections carry signal.

For case 1, adding $r$ a total of $L$ times gives $i + Lr$ . For case 2, $e(i)$ lies in chunk $c(i)$ , so $e(e(i)) = e(i)$ and every further composition changes nothing. For case 3, $e(i) + n_{\text{right}}$ lies in chunk $c(i) + 1$ when $1 \le n_{\text{right}} \le n_{\text{chunk}}$ . So $f(f(i)) = e(i) + n_{\text{chunk}} + n_{\text{right}}$ , and each further layer adds one more chunk. ∎

In plain words, lookahead compounds through depth, the way a receptive field grows. With per-frame lookahead, layer 2 reads frame $i + r$ of layer 1, and that frame already read up to $i + 2r$ of the input. A chunk boundary is a wall that every layer respects, so the reach stops at the end of the chunk at any depth. But any leak through the wall lets the next chunk in at every layer. The leak can be a centered convolution, or lookahead frames that are ordinary members of the next chunk.

The same composition gives the reach of stacked sliding windows in §12.6, where $L$ windows of width $w$ see $L(w-1)$ tokens into the past. In a streaming encoder, reach into the past is free, since that audio has already arrived, and costs only memory. Reach into the future costs latency, one for one.

✏️ Lookahead by hand

Noroozi et al. (2023) give the standard example. A 17-layer encoder with 10 ms features and 4x subsampling has 40 ms frames. A lookahead of $r = 2$ frames per attention layer then waits $17\times 2\times 40 = 1{,}360$ ms. With 8x subsampling, as in NVIDIA's FastConformer encoder, frames are 80 ms, and even $r = 1$ waits $17\times 80 = 1{,}360$ ms. Per-frame lookahead in every layer is unusable at sensible latencies.

Now take chunks of 16 frames at 40 ms, 640 ms of audio. The first frame of each chunk waits $15 \times 40 = 600$ ms, the last frame waits nothing, and the average frame waits $7.5 \times 40 = 300$ ms, whatever the depth.

A real model shows case 1 at small scale. Moonshine v2 (Kudlur et al. 2026) is a streaming recognizer for edge devices, the successor of the Moonshine models of §36.4. Its encoder runs on 20 ms frames. Its first two and last two layers read 4 frames ahead, and the others are causal. Four layers of 4 frames give at most $4\times4 = 16$ frames, 320 ms, which is the bound the authors state.

The cell checks the theorem by brute force. It builds each rule as a boolean matrix over 320 encoder frames and composes it with itself layer by layer. Frame $i$ at layer $\ell$ reaches input $k$ if it reads some frame $j$ that reaches $k$ . The cell reports how much future audio output frame 0 depends on.

Python · cell 2

The per-frame rule climbs to 1,360 ms at 17 layers, as computed by hand. The chunked rule stays at 600 ms at every depth. The third column is the warning. One centered convolution per layer adds a whole chunk, 640 ms, per layer. After 17 layers a chunked encoder with centered convolutions, even of kernel 3, waits 10.9 seconds, far worse than per-frame lookahead.

This is why streaming Conformers use causal convolutions. The dynamic-chunk model of Zhang et al. (2020), described below, uses them precisely to keep its encoder independent of future frames. An interviewer who asks why a "chunked" model has seconds of latency is usually fishing for this leak.

The chunked attention mask

Streaming Conformers use case 2 of the theorem with one more setting, how many past chunks each frame may see. Here is the rule in plain words first. Each frame sees every frame of its own chunk, including the later ones, and a fixed number of whole chunks before it. It sees nothing after its chunk.

📐 Chunked streaming attention mask

For $T$ encoder frames numbered from 0, chunk size $n_{\text{chunk}}$ , and left context of $n_{\text{left}}$ chunks, let $c(i) = \lfloor i/n_{\text{chunk}}\rfloor$ . Frame $i$ may attend to frame $j$ if and only if

c(i) - n_{\text{left}} \le c(j) \le c(i).

As a boolean $T\times T$ matrix (True means may attend) it is block lower triangular, with at most $n_{\text{left}} + 1$ blocks in each band of rows. The additive mask, which is added to the attention scores before the softmax, puts $-\infty$ at every False entry, as in §4.4. With $n_{\text{left}}$ unlimited and $n_{\text{chunk}} = 1$ it is the causal mask.

The lookahead variant lets every frame of a chunk also see the first $n_{\text{right}}$ frames after its chunk, replacing the upper bound by $j \le (c(i) + 1)\,n_{\text{chunk}} - 1 + n_{\text{right}}$ . Measured from the end of a frame's own audio, the worst algorithmic latency of one such layer is that of the first frame of a chunk,

\big(n_{\text{chunk}} - 1 + n_{\text{right}}\big)\,\Delta_{\text{enc}},

plus the lookahead of any convolution that is not causal.

Here is the mask for $T = 12$ , $n_{\text{chunk}} = 4$ , and $n_{\text{left}} = 1$ , with one row per attending frame. Every frame sees its whole chunk, including the future frames inside it, and the chunk before it. It sees nothing else.

       0  1  2  3  4  5  6  7  8  9 10 11
   0   1  1  1  1  .  .  .  .  .  .  .  .
   1   1  1  1  1  .  .  .  .  .  .  .  .
   2   1  1  1  1  .  .  .  .  .  .  .  .
   3   1  1  1  1  .  .  .  .  .  .  .  .
   4   1  1  1  1  1  1  1  1  .  .  .  .
   5   1  1  1  1  1  1  1  1  .  .  .  .
   6   1  1  1  1  1  1  1  1  .  .  .  .
   7   1  1  1  1  1  1  1  1  .  .  .  .
   8   .  .  .  .  1  1  1  1  1  1  1  1
   9   .  .  .  .  1  1  1  1  1  1  1  1
  10   .  .  .  .  1  1  1  1  1  1  1  1
  11   .  .  .  .  1  1  1  1  1  1  1  1

The left context sets cost and memory, not latency. Each query sees at most $(n_{\text{left}} + 1)\,n_{\text{chunk}}$ keys, so attention costs the same per frame however long the stream runs. Each layer must keep the keys and values of the last $n_{\text{left}}\,n_{\text{chunk}}$ frames. That is a chunk-by-chunk version of the rolling cache of §12.6.

The latency, in contrast, varies from frame to frame. In the drawing, frame 0 waits for frames 1 to 3 and frame 3 waits for nothing. So the algorithmic latency of a chunked encoder lies between 0 and $(n_{\text{chunk}} - 1)\Delta_{\text{enc}}$ .

The lookahead variant runs into case 3 of the theorem. Suppose the $n_{\text{right}}$ extra frames are ordinary frames of the next chunk, and every layer uses the same mask. Then each layer pulls in one more chunk of future. The fix is to give each chunk private copies of its lookahead frames. They are computed afresh for that chunk and never taken from the next chunk's outputs. The lookahead then stays $n_{\text{right}}$ frames at any depth. The widget's "recomputed lookahead" setting draws this case.

🔬 Emformer, the model built on private lookahead copies

Emformer (Shi et al. 2020) computes each chunk's lookahead copies from that chunk, its past, and the copies themselves, never from the next chunk's outputs. It caches the keys and values of a fixed left context. It can also summarize older chunks into a small bank of memory vectors, so the context can reach further back without a longer cache. Its best results came from hybrid systems, in which the Emformer acoustic model scores the states of hidden Markov models (§36.1) rather than feeding an end-to-end decoder. They reached 2.50 and 5.62 percent WER on LibriSpeech test-clean and test-other at an average latency of 960 ms, and 3.01 and 7.09 percent at 80 ms. Here average latency is the lookahead plus half the chunk, and the 80 ms systems used no memory bank.

The widget draws, on top, the frames of every layer that the selected output frame depends on, down to the input, and below it one layer's mask. Start in "per-frame lookahead" with $r = 2$ and predict how far ahead the selected frame reaches as you add layers from 1 to 12, or press Stack layers. Then switch to "chunks" with chunk size 4 and repeat. Last, compare "shared lookahead" with "recomputed lookahead", which both let a chunk see 2 frames past its end, at 8 layers, and decide which one is case 3 of the theorem.

Core

A chunked streaming mask

Build the chunked mask of the definition and use it to check the theorem. Implement three functions.

chunk_mask(T, n_chunk, n_left, n_right=0) returns the boolean (T, T) mask, with True meaning "may attend". Frame i sees its own chunk, n_left whole chunks before it, and the n_right frames just past the end of its own chunk. n_left=None makes the whole past visible, and n_left=0 allows the frame's own chunk only. The last chunk may be shorter than n_chunk.
algorithmic_latency_ms(n_chunk, n_right, frame_ms, conv_lookahead_frames=0) returns the worst algorithmic latency of one chunked layer in milliseconds, measured from the end of the frame's own audio. A convolution stem that reads conv_lookahead_frames encoder frames ahead adds its lookahead on top.
max_dependency(mask, n_layers) returns an integer array whose entry i is the last input frame that output frame i depends on after n_layers layers that all use mask, for n_layers at least 1.

import numpy as np

def chunk_mask(T, n_chunk, n_left, n_right=0):
    i = np.arange(T)[:, None]
    j = np.arange(T)[None, :]
    return j <= i                      # placeholder: a causal mask, not a chunked one

def algorithmic_latency_ms(n_chunk, n_right, frame_ms, conv_lookahead_frames=0):

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Caches make streaming exact

A chunked mask defines what a streaming encoder computes. It does not say how to compute it as the audio arrives.

The wasteful way is buffered streaming. It runs a model on overlapping windows of audio and keeps only the output for the middle of each window. For example, each 1-second chunk is transcribed inside a 4-second buffer of past and future context. The context is recomputed every time. And if the model was trained on whole utterances, the context it sees at inference differs from what it saw in training.

Noroozi et al. (2023) showed how to do better with a cache-aware streaming encoder. The recipe has two halves.

Train with the streaming context. Use the chunked mask, causal convolutions, and layer normalization, so that training already sees exactly the contexts that streaming will provide.
Infer chunk by chunk with two caches per layer. Each causal depthwise convolution of kernel $k$ keeps the last $k-1$ inputs it saw. Each attention layer keeps the inputs of the last $n_{\text{left}}\,n_{\text{chunk}}$ frames, from which it computes keys and values for the new chunk's queries. Caching the keys and values themselves, as in the KV cache of §6.5, is equivalent. Normalization, feed-forward, and pointwise layers need no cache, since they act on each frame separately.

Why is this exact? Each layer's output at a frame depends only on the inputs its mask allows, and the caches hold exactly those inputs. So chunk-by-chunk inference reproduces the whole-utterance computation with the same mask, number for number, and no work is repeated.

The PyTorch block builds a two-layer encoder of causal depthwise convolutions and chunked attention with two chunks of left context. It runs the encoder on a whole utterance with the mask, then chunk by chunk with caches, and compares the two.

🔥 In PyTorchfor a machine with PyTorch

The streamed output matches the offline one to float32 rounding. Dropping the caches, which is what processing chunks independently amounts to, changes the output at order one. Every chunk loses its left context, and its convolution sees zeros where the previous chunk should be.

A recurrent or state-space encoder (§21.4) is causal by construction and streams the same way. It carries a fixed-size state from chunk to chunk in place of the attention cache.

📌 What cache-aware streaming buys

Noroozi et al. (2023) report WER on LibriSpeech test-other for models of about 114M parameters trained on LibriSpeech, each a FastConformer encoder decoded with a transducer. Average latency here is the encoder's algorithmic latency averaged over words, without compute.

model	average latency	WER
offline, full context	none (whole utterance)	5.0 percent
buffered streaming of the offline model (2 s chunks, 4 s buffers)	2,000 ms	11.3 percent
cache-aware, no lookahead	0 ms	9.5 percent
cache-aware, per-frame lookahead in every layer	1,360 ms	7.1 percent
cache-aware, chunked lookahead	1,360 ms	6.3 percent

The two 1,360 ms rows are not the same wait. Per-frame lookahead makes every frame wait 1,360 ms. The chunked model's 1,360 ms is an average, so the first frame of a chunk waits about twice that and the last frame waits nothing.

Sweeping the chunked models gives 9.0, 7.8, 7.5, 7.3, and 6.3 percent at average latencies of 40, 240, 520, 680, and 1,360 ms. Latency buys accuracy smoothly. Training with the streaming mask beats streaming an offline model by a wide margin.

One model for many latencies

A chunk size fixed at training fixes the latency, and different products want different latencies. Captions can wait a second, while a voice command cannot. Zhang et al. (2020) proposed dynamic chunk training in their U2 model. For each training batch they use full attention with probability one half. Otherwise they draw a chunk size uniformly between 1 and $\min(25, l_{\max} - 1)$ frames, where $l_{\max}$ is the longest utterance in the batch.

One model then works at any chunk size at inference, from fully streaming to offline, and the latency becomes a decoding setting. The model uses causal convolutions so that the encoder does not depend on future frames, for the reason the theorem gives.

Noroozi et al. (2023) do the same for cache-aware models, drawing a chunk size per batch. Trained on a large multi-domain set, their multi-latency transducer slightly beat single-latency models trained for each setting. On LibriSpeech test-other it scored 6.2, 5.5, and 5.2 percent against 6.4, 5.9, and 5.4 at 40, 240, and 520 ms. In practice, this means one trained model can serve every product's latency target.

Decoders that write before the audio ends

A streaming encoder is necessary but not sufficient. The decoder must also be able to emit words before the input ends.

Frame-synchronous decoders stream for free. A decoder is frame-synchronous when it makes its decisions one encoder frame at a time. CTC (§36.2) emits one symbol per encoder frame, possibly a blank that means "nothing new here". A transducer (§36.3) reads one encoder frame at a time and emits labels from it and from its own past labels. On top of a chunked encoder, both produce partial results chunk by chunk. Their remaining problem is emission delay, the habit of waiting for more context. FastEmit cut it by 150 to 300 ms on a Google Voice Search test set.

An attention decoder does not stream. Each label it writes cross-attends over all the encoder frames (§4.6), so the decoder cannot be sure of its first word until the input has ended. There are three families of fixes.

Two passes

The first fix keeps the attention decoder but uses it only at the end. Sainath et al. (2020) ran a streaming transducer as the first pass and showed its partial results while the user spoke. When the endpointer fired, they rescored the transducer's most likely hypotheses. The rescorer was an attention decoder that reads the whole encoder output, the Listen, Attend and Spell model of §36.4. A simple version rescores an n-best list, the first pass's few most likely hypotheses. Sainath et al. stored them instead as a lattice, a tree in which hypotheses that share a prefix share its branch, so the shared prefix is scored once.

Rescoring is not a search. It is a single forward pass that reads each hypothesis as given and scores it, the way a decoder is run during training (teacher forcing). With the lattice's branches batched together it took a median of 58 ms and a 90th percentile of 97 ms on a Pixel 4 CPU.

At the same latency, the system had 8 percent lower relative WER than a server-side conventional recognizer, with a model more than 400 times smaller. The U2 model of Zhang et al. (2020) is the same idea with CTC as the first pass, and it rescores an n-best list. The cost is a product decision rather than a technical one. The final transcript can differ from the partial one the user already saw, so the text changes at the end of each utterance.

Streaming an offline model by agreement

The second fix needs no retraining. Run the offline model again and again on a growing audio buffer, and show only what successive runs agree on. The rule for "agree" is called LocalAgreement. Whisper-Streaming (Macháček, Dabre and Bojar 2023) is the standard implementation, and it loops over three steps.

Re-decode. Every time at least MinChunkSize seconds of new audio have arrived, it transcribes the whole buffer again. If the previous update took longer than that, the next one starts at once. Up to 200 committed words from audio already trimmed out of the buffer go in as Whisper's text prompt, which keeps the style consistent.
Commit. It compares the new transcript with the previous one and commits what they share beyond what is already committed.
Trim. When the committed text contains the end of a sentence, it cuts the audio buffer there, which keeps the buffer at about 30 seconds, Whisper's window. If the buffer is still too long, it cuts at the end of the last committed segment instead. The released code defaults to cutting at Whisper's segment ends once the buffer passes 15 seconds.

✏️ LocalAgreement-2 by hand

step	hypothesis of the whole buffer	committed after the step
1	the	(nothing)
2	the cat sat	the
3	the cat sat on	the cat sat
4	the cat sat on a mat	the cat sat on
5	the cat sat on the mat	the cat sat on
6	the cat sat on the mat today	the cat sat on the mat

At step 5 the model changes its mind about "a". The remainders "a mat" and "the mat" share nothing, so nothing new is committed. The wrong word "a" was never shown as final. "today" appears at step 6 and will be committed only when the next hypothesis repeats it.

That last sentence sets the latency floor of the method. A word can be committed only after it has appeared in two consecutive hypotheses.

The authors measured this latency without computation at 1.02, 1.91, and 3.73 seconds for chunks of 0.5, 1, and 2 seconds. That is about twice the chunk size, as they expected from two consecutive updates having to agree. It sits above the $1.5c$ floor because real words fall short of the ideal. A word at the very end of the buffer is often transcribed wrongly or not at all, so it waits for a later pair of hypotheses to agree. Real computation added another 1.7 to 2.3 seconds, because every update re-encodes the buffer.

Compute is the second cost. Whisper's encoder always processes a full 30-second window (§36.4). With updates every second it runs about 30 encoder passes for every 30 seconds of audio, where offline transcription runs one, and the decoder rewrites the uncommitted text each time. Each stream also has its own buffer, so streams batch poorly. The delayed-streams authors below measured Whisper-Streaming's throughput on an H100 at 100 times lower than that of their natively streaming model (Zeghidour et al. 2025). If an interviewer asks how to stream Whisper, this method is the answer, and its two costs are the follow-up.

Interview

LocalAgreement

Implement local_agreement(hyps, n=2) from the definition. hyps is a list of hypotheses, each a list of words, produced one after another by re-decoding a growing buffer. Return one entry per hypothesis, the list of committed words after that hypothesis has been processed. Each entry must be its own list, not a reference to one list that keeps changing. Committed words are never removed or changed, even when a later hypothesis rewrites them.

def local_agreement(hyps, n=2):
    committed = []
    out = []
    for h in hyps:
        committed = list(h)            # placeholder: shows the latest hypothesis outright
        out.append(list(committed))
    return out

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Delayed streams

The third fix trains a model that streams by design. Line the transcript up with the audio before training, so that each word sits at the audio frame where it starts. Then shift the text later by a fixed delay, and train a decoder to write that shifted text while it reads the audio. At every moment the decoder has heard a fixed amount of audio beyond the word it is writing, and it never waits for more. This is delayed streams modeling (DSM, Zeghidour et al. 2025).

It differs from everything so far in how it treats alignment. CTC and transducers sum over all possible alignments, and attention decoders find their own. Delayed streams fix the alignment before training.

A forced aligner gives every word of the training transcripts a start time. For pretraining, the start times came from whisper-timestamped, a tool that adds word timestamps to Whisper's output. For fine-tuning on human transcripts, Whisper's timestamped transcript was matched to the human one word by word with dynamic time warping, a dynamic program that aligns two sequences. Audio and text then become two token streams at the same rate, one text slot per audio frame.

📐 Delayed streams modeling

Encode the audio at frame rate $f_r = 12.5$ Hz, frames of $\Delta_{\text{enc}} = 80$ ms, as tokens $X_0, X_1, \dots, X_{T-1}$ . Here $X_t$ is the set of discrete codes that Mimi, a neural audio codec (a network that compresses each audio frame into a few integers, §37.3), assigns to frame $t$ . Let the transcript be words $1, \dots, W$ in time order. Word $w$ has start time $s_w$ seconds from the aligner and text tokens $(x_{w,1}, \dots, x_{w,n_w})$ . Two special tokens, WORD and PAD, never occur inside words.

Aligned text stream. Set $Z_t = \text{PAD}$ for every $t$ . For each word, with start frame $i_w = \lfloor s_w f_r\rfloor$ , set $Z_{i_w} = \text{WORD}$ and $Z_{i_w + m} = x_{w,m}$ for $m = 1, \dots, n_w$ .
Delay. For a delay of $\tau$ frames ( $\tau$ is a delay here, not a temperature), $\tau_{\text{delay}} = \tau\Delta_{\text{enc}}$ seconds, the target stream is $Y_t = Z_{t-\tau}$ for $t \ge \tau$ and $Y_t = \text{PAD}$ for $t \lt \tau$ . The paper defines $\tau$ in steps and quotes delays in seconds, so its default of 2.5 s is about 31 steps.
Model. At step $t$ a causal decoder-only Transformer reads the sum of two learned embeddings, $\mathbf e_t = E_{\text{audio}}(X_t) + E_{\text{text}}(Y_{t-1})$ , and outputs a distribution over $Y_t$ . Here $E_{\text{audio}}$ sums one embedding per code of the frame, and a fixed start embedding stands in for $Y_{-1}$ at $t = 0$ . The model is trained with cross-entropy at every step, PAD and WORD steps included.
Inference. Audio tokens are fed as they arrive. $Y_t$ is sampled at each step and fed back at the next. A WORD sampled at step $t$ means a word started in frame $t - \tau$ , at time $(t - \tau)\Delta_{\text{enc}}$ .

In plain words, the model writes the text of frame $t - \tau$ while it already hears frame $t$ . It always has $\tau$ frames of future audio, and it never waits for more. The latency is $\tau_{\text{delay}}$ , an explicit number, plus the compute of one step.

Two more things follow from the construction. Since $Y_t = Z_{t - \tau}$ exactly, the WORD markers the model writes are word starts in the audio, shifted by $\tau$ . Shifting them back gives word timestamps to within one 80 ms frame. And because the loss covers the PAD and WORD steps, the model learns where words start, not only what they are.

✏️ Delayed streams by hand

Take "call mom now" with start times 0.10, 0.50, and 0.90 seconds, 16 frames of 80 ms, and $\tau = 2$ . Suppose the tokenizer splits "mom" into "m" and "om". The start frames are $\lfloor 0.10 \times 12.5\rfloor = \lfloor 1.25\rfloor = 1$ , $\lfloor 6.25\rfloor = 6$ , and $\lfloor 11.25\rfloor = 11$ . A dot is PAD and W is WORD.

$t$	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
$Z_t$	·	W	call	·	·	·	W	m	om	·	·	W	now	·	·	·
$Y_t$	·	·	·	W	call	·	·	·	W	m	om	·	·	W	now	·

At step 4 the input is $E_{\text{audio}}(X_4) + E_{\text{text}}(\text{W})$ , and the model must output "call". It has heard audio up to the end of frame 4, 400 ms, two frames past the frame that holds "call" in $Z$ . Read the W's of $Y$ at steps 3, 8, and 13 and subtract $\tau$ . That gives frames 1, 6, and 11, so timestamps of 80, 480, and 880 ms, each the start of the frame that contains the true start. The last token, "now", is written at step 14, once 1,200 ms of audio have arrived.

Most text steps are PAD. Words last several frames and take only a token or two each. Moshi, the full-duplex dialogue model of §37.5 whose aligned text stream DSM adapts, found that about 65 percent of the text steps of English conversation are PAD (Défossez et al. 2024).

At inference the model is fed the audio as it arrives and writes the text $\tau_{\text{delay}}$ behind it. There is no chunking, no re-decoding, and no search over alignments. Every stream advances one step per frame in lockstep, so many streams batch exactly like LLM decoding (§20.6).

The numbers are strong. DSM-ASR has a 2.6B-parameter decoder and a default delay of 2.5 seconds. It averaged 6.4 percent WER over the eight English short-form test sets of the Open ASR Leaderboard (§36.1). The offline models the paper compares with score 5.6 to 6.1 there, from Canary-Qwen-2.5B to Parakeet-TDT-0.6B-v2. It served 400 streams at once in real time on one H100.

A version trained with delays sampled per sequence from 0.25 to 4 seconds, and told the delay as an input, runs at any delay in that range. It lands within about 300 ms of the delay it is asked for. The same trick of shifting one stream against another generates speech instead of text in the codec language models of §37.4.

Why must the text lag at all? The authors give a two-line counterexample.

Speech is the same in a softer form. A word is often ambiguous until the sounds after it are heard, and the delay is how much of that future the model is allowed. An interviewer may ask you to choose $\tau$ . Measure WER against delay on your own traffic and take the shortest delay past which WER stops improving much. The curve depends on the language and the domain, so there is no universal answer.

Core

Delayed text streams

Build the delayed target stream of the definition and invert it to recover timestamps. Implement two functions.

build_delayed_text_stream(word_starts_s, word_tokens, n_frames, frame_ms, delay_frames) returns the delayed target stream $Y_0, \dots, Y_{n-1}$ as a list of n_frames strings, with "WORD" and "PAD" as the special tokens. word_starts_s holds the sorted start times in seconds, and word_tokens[w] is the list of word w's tokens. Follow the definition, with one rule for collisions. If a word's start frame $\lfloor s_w \cdot 1000 / \text{frame\_ms}\rfloor$ is already taken by an earlier word, start it at the first free frame after that earlier word instead. Drop tokens that would land at or past n_frames.
recover_word_starts(stream, frame_ms, delay_frames) is the inverse. It returns one (start_ms, tokens) pair per WORD in the stream. Here start_ms is a float, and tokens is the list of tokens that follow the WORD up to the next WORD or PAD.

import math

def build_delayed_text_stream(word_starts_s, word_tokens, n_frames, frame_ms, delay_frames):
    return ["PAD"] * n_frames          # placeholder: no words at all

def recover_word_starts(stream, frame_ms, delay_frames):
    return []                          # placeholder

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Delayed streams now run in production-grade open models. Voxtral Realtime (Mistral AI 2026) is a 4-billion-parameter delayed-streams recognizer for 13 languages. Its decoder writes a padding token while the acoustic evidence is incomplete. It writes a word-boundary token once a word is complete and the target delay has passed. On the FLEURS multilingual benchmark it came close to offline Whisper at a delay of 480 ms, 8.7 against 8.2 percent WER, and beat it at 960 ms with 7.7. At 2.4 seconds it came within one WER point of Mistral's own offline recognizer, 6.7 against 5.9.

🔬 Inside Voxtral Realtime, and an older streaming attention

Voxtral Realtime has a causal audio encoder trained from scratch. It runs at 50 Hz with sliding-window attention over 750 frames (15 seconds). An MLP adapter pools every 4 encoder frames into one, giving 12.5 Hz, and a decoder initialized from Ministral 3B reads the pooled frames. The whole model has about 4.4 billion parameters. The target delay is sampled from 80 to 2,400 ms during training. The delay is fed to the model, in training and at inference, through an adaptive normalization of the feed-forward branch of every block, so one model serves every delay in that range.

The older answer from attention-based recognition is monotonic chunkwise attention (MoChA, Chiu and Raffel 2017). The decoder learns to move a pointer forward through the encoder frames and to stop when it has seen enough. It then attends softly over a small chunk ending at the pointer, so each label needs only the audio up to its pointer. It was a strong streaming attention model in its time, but transducers and now delayed streams have largely replaced it.

The table puts the approaches side by side. It is the shape of a good answer to "how would you build streaming recognition".

approach	where its latency comes from	extra compute	accuracy against offline	needs special training	batches well
chunked encoder with CTC or a transducer	chunk size and lookahead, plus emission delay	none, with caches	about 25 percent more errors at 1.4 s, far more near zero lookahead	yes, the streaming mask	yes
two-pass rescoring	first pass for partials, endpointer plus a short rescoring for finals	one teacher-forced pass per utterance	better than the first pass alone	yes, both decoders	yes
LocalAgreement over an offline model	update interval times about 2, plus the time to re-decode the buffer	the encoder re-runs at every update	a little worse than offline, from short buffers	no	poorly
delayed streams (DSM, Voxtral Realtime)	the chosen delay, an explicit number	none, one decoder step per frame	approaches offline at 1 to 2.5 s, competitive at 0.5 s	yes, aligned streams	yes, in lockstep
monotonic attention (MoChA)	how far the learned pointer has moved	small	close to offline in 2017	yes	moderately

Voice activity detection and endpointing

A deployed recognizer almost never sees raw audio. In front of it sits a voice activity detector (VAD), which decides for each short frame whether someone is speaking. It earns its place in three ways.

It saves compute. Most of a phone's microphone input, and much of a meeting, is silence or noise.
It cuts long recordings into segments that fit the model, which batch transcription needs.
It guards encoder-decoder models against hallucination. §36.4 showed that Whisper's decoder, a language model conditioned on audio, writes fluent text when the audio carries no speech. So silence and music must never reach it. CTC and transducers have no free-running language model and are less exposed, but they still waste compute on silence.

The oldest VAD thresholds the energy of each frame, its loudness in decibels. It works in a quiet room and fails in noise, because a fan, music, or a door slam has as much energy as speech.

The cell builds a 30-second energy track of four utterances. It has dips between syllables, four 60 ms clicks, and a fan that switches on near the end. It compares a single threshold with hysteresis, which starts a speech region above one threshold and ends it only below a lower one. A thermostat works the same way, so that it does not flick on and off around one temperature.

Python · cell 3

The single threshold chatters. It flips on and off at the dips between syllables, giving 193 segments for 4 utterances, and it clips almost 3 seconds of speech. Hysteresis cuts the chatter to 18 segments and clips only 170 ms. Neither can tell the fan from a voice, and both report 3.5 to 3.7 seconds of false alarm, almost all of it the fan. Telling them apart is the job of a better frame classifier.

WebRTC's VAD, used in browsers, scores 10, 20, or 30 ms frames with Gaussian mixture models of speech and noise, and it has four aggressiveness modes. Neural VADs handle noise far better. Silero VAD, for example, is a model of about 2 MB. It outputs a speech probability for each 32 ms window at 16 kHz, in under a millisecond on one CPU thread. Every VAD, energy or neural, then needs the same post-processing to turn frame decisions into segments.

📐 VAD segmentation with hysteresis

Given speech probabilities $p_1, \dots, p_T$ for consecutive frames, an onset threshold $\theta_{\text{on}}$ , and an offset threshold $\theta_{\text{off}} \lt \theta_{\text{on}}$ ,

Hysteresis. In silence, a frame with $p_t \ge \theta_{\text{on}}$ starts a segment. In speech, a frame with $p_t \lt \theta_{\text{off}}$ ends it at that frame. Other frames keep the current state. A segment still open at the last frame ends there.
Bridge pauses. Merge consecutive segments separated by less than the minimum silence duration.
Drop bursts. Remove segments shorter than the minimum speech duration.
Pad. Extend each segment by a padding on both sides, clipped to the signal, and merge segments that now overlap or touch.

Silero VAD's defaults are $\theta_{\text{on}} = 0.5$ , $\theta_{\text{off}} = \max(\theta_{\text{on}} - 0.15, 0.01)$ , a minimum speech duration of 250 ms, a minimum silence of 100 ms, and 30 ms of padding. Its code differs from these four steps in small ways. It bridges pauses inside the hysteresis loop, and where padding would make two segments overlap it splits the gap between them instead of merging them.

The order matters. Dropping short segments before bridging pauses deletes real words that a pause split off. The sweep after the exercise shows this when it drops short segments without bridging.

Core

VAD with hysteresis

Implement vad_segments(probs, frame_ms, on, off, min_speech_ms, min_silence_ms, pad_ms). It follows the four steps of the definition exactly and returns a list of (start_ms, end_ms) pairs as floats, sorted by time.

Frame t covers [t * frame_ms, (t + 1) * frame_ms), so a segment of frames s to e - 1 spans s * frame_ms to e * frame_ms.
A pause is short when its length in milliseconds is less than min_silence_ms.
A segment is kept when its length is at least min_speech_ms.

import numpy as np

def vad_segments(probs, frame_ms, on, off, min_speech_ms, min_silence_ms, pad_ms):
    p = np.asarray(probs, dtype=float)
    segs, start = [], None
    for k, x in enumerate(p):                 # placeholder: one threshold, nothing else
        if x >= on and start is None:
            start = k

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

With a post-processor like yours, the cell sweeps the parameters on the energy track above, adding one at a time.

Python · cell 4

Each parameter trades one kind of error for another.

Minimum speech removes the clicks. Used alone, it also deletes fragments of real speech that pauses split off, and clipped speech doubles to 330 ms.
Minimum silence, applied first, bridges those pauses. That leaves 5 segments, the 4 utterances and the fan.
Padding recovers the soft onsets and endings that a threshold cuts, so clipped speech falls to 0, at the price of more false alarm.
A minimum silence of a full second goes too far. Clicks in the pauses become stepping stones that join whole utterances, and the last utterance swallows the fan. That leaves 3 segments and 7.8 seconds of false alarm.

The thresholds trade the same way. In practice they are tuned on held-out audio from the deployment domain. The errors are measured as durations of missed speech and of false alarm. On frames, these are the miss rate (one minus recall) and the false positive rate of §0.11.

The detection cost function of the pyannote.metrics library weights them 0.75 for a miss and 0.25 for a false alarm. That reflects the usual asymmetry. Clipped speech loses words for good, while extra audio mostly costs compute.

Endpointing

The endpointer decides that the user has finished, so the microphone can close and the system can respond. The classical endpointer declares the end of the query once the VAD has seen speech followed by a fixed interval of silence. The interval is a direct trade-off. Too short, and the system cuts people off at a pause and loses the rest of the sentence. Too long, and every answer arrives late.

Sainath et al. (2020) measured the trade-off on Google's Voice Search. With a separate acoustic endpointer, their WER rose from 6.2 percent without endpointing to 7.4 percent. Its median endpoint latency was 450 ms, and its 90th percentile 860 ms.

They then trained the transducer itself to emit an end-of-query token </s>. During training, the log-probability of that token gets a penalty for coming before the true end and another for coming more than a grace period after it,

\log p(\texttt{</s>} \mid \mathbf x_t) \;\leftarrow\; \log p(\texttt{</s>} \mid \mathbf x_t) - \max\big(0, \alpha_{\text{early}}(t_{\texttt{</s>}} - t)\big) - \max\big(0, \alpha_{\text{late}}(t - t_{\texttt{</s>}} - t_{\text{buffer}})\big).

Here $t$ is the current frame, $\mathbf x_t$ is the audio up to it, $t_{\texttt{</s>}}$ is the reference end frame, and $\alpha_{\text{early}}, \alpha_{\text{late}}, t_{\text{buffer}}$ are hyperparameters. In plain words, emitting the end token early is punished in proportion to how early it is. Emitting it late is punished only beyond a grace period of $t_{\text{buffer}}$ frames.

Combined with the acoustic endpointer, the trained end token reached 6.8 percent WER at median and 90th-percentile latencies of 430 and 790 ms. With the second pass it reached 6.1 percent at 430 and 780 ms. The server-side conventional system scored 6.6 percent at 460 and 870 ms. So the end token alone wins on latency, and with the second pass the system wins on both counts.

The reason is content. A model that has heard "call" can learn that requests rarely end there, which silence alone cannot tell it. Full-duplex dialogue models (§37.5) take the next step and decide when to speak from the content of the whole conversation.

Diarization, who spoke when

A meeting transcript is of little use without speaker labels. Diarization splits a recording by speaker. It answers "who spoke when" without knowing in advance who the speakers are or how many there are. Its output is a set of time segments, each with an anonymous label such as "speaker 2".

The labels are arbitrary. A system that calls Alice "speaker 2" and Bob "speaker 1" has done nothing wrong. So any scoring must first match the system's speakers, the hypothesis speakers, to the reference speakers of the human annotation.

📐 Diarization error rate

Score on short frames. In frame $t$ , let $N_{\text{ref}}(t)$ and $N_{\text{hyp}}(t)$ be the numbers of active reference and hypothesis speakers. Fix a one-to-one mapping $m$ from reference to hypothesis speakers, and let $N_{\text{corr}}(t)$ be the number of active reference speakers whose mapped hypothesis speaker is also active. Summing over frames, and multiplying by the frame duration,

t_{\text{miss}} = \sum_t \max\big(0, N_{\text{ref}} - N_{\text{hyp}}\big), \quad t_{\text{FA}} = \sum_t \max\big(0, N_{\text{hyp}} - N_{\text{ref}}\big), \quad t_{\text{conf}} = \sum_t \big(\min(N_{\text{ref}}, N_{\text{hyp}}) - N_{\text{corr}}\big),

\mathrm{DER} = \frac{t_{\text{FA}} + t_{\text{miss}} + t_{\text{conf}}}{t_{\text{speech}}}, \qquad t_{\text{speech}} = \sum_t N_{\text{ref}}(t),

with $m$ chosen to minimize DER. Overlapped speech counts once per speaker in $t_{\text{speech}}$ .

Two conventions change the number and must be reported. A collar removes the frames within a fixed distance of every reference boundary, often 0.25 s on each side, because hand-marked boundaries are imprecise. Tools state it differently. The classic md-eval scorer takes the 0.25 s per side, while pyannote.metrics takes the total width, so the same collar is collar=0.5 there. Overlap may be scored or skipped. DIHARD-style scoring, and the "Full" benchmark setup of the pyannote toolkit, use no collar and score overlap.

In plain words, a missed second is reference speech the system left uncovered. A false-alarm second is a speaker the system invented. A confused second went to the wrong speaker. DER adds them up and divides by the total speaking time.

The Hungarian algorithm is the same matching step that pairs predicted and true objects in the DETR detector of §27.6. In code it is one call to scipy.optimize.linear_sum_assignment.

∂ Proof that only the matched overlap depends on the mapping

The terms $t_{\text{miss}}$ , $t_{\text{FA}}$ , and $\sum_t\min(N_{\text{ref}}, N_{\text{hyp}})$ depend only on how many speakers are active in each frame, not on the mapping. Only $\sum_t N_{\text{corr}}(t)$ does. Count it the other way round. Each mapped pair $(r, h)$ contributes one to $N_{\text{corr}}(t)$ in exactly the frames where both are active, so $\sum_t N_{\text{corr}}(t) = \sum_{(r,h) \in m} O_{rh}$ . Minimizing DER is therefore maximizing that sum, which is an assignment problem on the matrix $O$ .

For greedy, take $O = \begin{pmatrix} 5 & 4 \\ 4 & 0\end{pmatrix}$ . Greedy takes the 5 and is left with the 0, a total of 5, while the crossed mapping gets $4 + 4 = 8$ . ∎

So scoring a diarization takes three steps. Build the co-occurrence table, solve the assignment, then count the three kinds of error frame by frame. Here are all three on a recording small enough to do by hand.

✏️ DER by hand

A 60-second recording has two reference speakers, A from 0 to 20 s and B from 25 to 55 s, so $t_{\text{speech}} = 50$ s. A system outputs speaker 1 from 0 to 17 s and again from 50 to 55 s, and speaker 2 from 23 to 50 s. The co-occurrences are $O_{A1} = 17$ , $O_{A2} = 0$ , $O_{B1} = 5$ , and $O_{B2} = 25$ . The best mapping is A to 1 and B to 2, with a total of 42 against 5 for the other. Then the errors are

17 to 20 s, missed (3 s),
23 to 25 s, a false alarm (2 s),
50 to 55 s, speech of B given to speaker 1, who is mapped to A (5 s of confusion),

\mathrm{DER} = \frac{2 + 3 + 5}{50} = 20\ \text{percent}.

Dividing by the 60-second duration instead would give 16.7 percent, a common mistake. A 0.25 s collar removes half a second around each of the four reference boundaries, and half of each of those windows lies in speech, so 1 s of reference speech goes. It also removes 0.25 s from each error, giving $(1.75 + 2.75 + 4.75)/49 = 18.9$ percent.

The widget scores a longer recording the same way. It starts with a system that is close to the reference, scored with no collar and with overlap included. The reference has three overlaps (8 to 9 s, 20.5 to 21 s, and 24 to 25 s), and the system never outputs two speakers at once. Before touching anything, guess what DER becomes with a 0.25 s collar and with overlap skipped. Then press "Rename system speakers" and predict whether anything changes.

The same close system scores 12.2 percent with no collar and overlap scored. It scores 5.0 percent with a 0.25 s collar, 4.5 percent with overlap skipped instead, and 0.9 percent with both. Either convention alone cuts the number by more than half, which is why a DER without its scoring setup is not a number you can compare. Asking "with what collar, and is overlap scored?" is the right reply to any DER claim in an interview or a paper review. The "two speakers merged" preset shows the third kind of error. Its system finds only two speakers, so where C talks alone, C's speech goes to a system speaker matched to A and counts as confusion, 25.8 percent DER with neither rule.

Interview

Diarization error rate

Implement der(ref, hyp, frame_ms), with no collar and overlap scored.

ref is a boolean array of shape (T, n_ref) whose column r marks the frames where reference speaker r is active. hyp is (T, n_hyp) for the system. Both may have overlap, and n_hyp may differ from n_ref.
Find the optimal one-to-one mapping with scipy.optimize.linear_sum_assignment on the co-occurrence matrix, then compute the terms of the definition.
Return a dict with keys "fa", "miss", "conf", and "total" (all in milliseconds, as floats), "der", and "mapping". The mapping is a dict from reference index to hypothesis index that includes only pairs with positive co-occurrence.
Raise ValueError if the reference has no speech.

import numpy as np

def der(ref, hyp, frame_ms):
    ref, hyp = np.asarray(ref, dtype=bool), np.asarray(hyp, dtype=bool)
    mapping = {r: r for r in range(min(ref.shape[1], hyp.shape[1]))}   # placeholder: map by label index
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The cascade, and why it misses overlap

The classical system is a cascade of four steps.

A VAD finds the speech.
The speech is cut into short windows, a second or two long, and each window gets a speaker embedding. That is a vector from a network trained to tell voices apart, so that two windows of the same voice get nearby vectors. The common networks, x-vectors and ECAPA-TDNN, are in §35.6.
The embeddings are clustered. Agglomerative clustering merges the closest clusters until their cosine distance passes a threshold. Spectral clustering instead estimates the number of speakers from the largest gap between consecutive eigenvalues of a matrix built from the pairwise similarities.
A resegmentation pass, often a hidden Markov model over the cluster labels, refines the boundaries.

The cascade is modular and works with any number of speakers. But each window gets exactly one speaker, so overlapped speech is missed by construction. The two-speaker CALLHOME telephone calls that Fujita et al. (2019) tested on had an average overlap ratio of 13 percent. And no part of the cascade is trained to minimize DER.

End-to-end diarization and permutation-invariant training

End-to-end neural diarization (EEND, Fujita et al. 2019) treats diarization as frame-level multi-label classification. A self-attention encoder reads the features of the whole recording. It has $n_{\text{spk}}$ output slots, one per possible speaker. For each frame $t$ and slot $c$ it outputs a logit $z_{t,c}$ , whose sigmoid is the probability that the slot's speaker is active. Overlap is no longer special, since two slots can be active at once.

The difficulty is the targets. The reference activities $y_{t,s} \in \{0, 1\}$ come with arbitrary speaker labels. (In this subsection $y$ is a speaker activity, not a transcript.) Nothing says which slot should be which speaker. Training slot 1 against "speaker 1" asks the network to predict a labeling that is a coin flip, and it learns to hedge.

The fix, borrowed from speech separation, is to score the network under whichever assignment of slots to speakers fits best. Write $\pi$ for a permutation, a one-to-one reordering of the $n_{\text{spk}}$ speakers. Write $\mathrm{BCE}(z, y) = \log(1 + e^{z}) - yz$ for the binary cross-entropy computed from a logit.

∴ Permutation-invariant training

The permutation-invariant loss

\mathcal L_{\text{PIT}} = \min_{\pi}\ \frac{1}{T\,n_{\text{spk}}}\sum_{t=1}^{T}\sum_{c=1}^{n_{\text{spk}}}\mathrm{BCE}\big(z_{t,c},\, y_{t,\pi(c)}\big)

is unchanged when the reference speakers are relabeled. It decomposes over slot and speaker pairs, $\mathcal L_{\text{PIT}} = \min_\pi \frac{1}{T n_{\text{spk}}}\sum_c C_{c,\pi(c)}$ with $C_{cs} = \sum_t \mathrm{BCE}(z_{t,c}, y_{t,s})$ . So the minimizing permutation is a linear assignment on the $n_{\text{spk}}\times n_{\text{spk}}$ cost matrix $\mathbf C$ . Brute force evaluates $n_{\text{spk}}!$ permutations, each costing $O(Tn_{\text{spk}})$ . Building $\mathbf C$ costs $O(T n_{\text{spk}}^2)$ , and the Hungarian algorithm then costs $O(n_{\text{spk}}^3)$ .

In plain words, the loss asks only that some consistent assignment of slots to speakers fits. It finds the best one with the same matching algorithm that scores DER.

∂ Proof of invariance and of the decomposition

Relabeling the reference replaces $y_{t,s}$ by $y_{t,\pi_0(s)}$ for a fixed permutation $\pi_0$ . The loss under $\pi$ for the relabeled targets equals the loss under $\pi_0\circ\pi$ for the original ones. As $\pi$ ranges over all permutations, so does $\pi_0\circ\pi$ , so the minimum is the same.

For the decomposition, swap the two sums, $\sum_t\sum_c \mathrm{BCE}(z_{t,c}, y_{t,\pi(c)}) = \sum_c \sum_t \mathrm{BCE}(z_{t,c}, y_{t,\pi(c)}) = \sum_c C_{c,\pi(c)}$ . That is the cost of the assignment that pairs slot $c$ with speaker $\pi(c)$ . Training differentiates through the minimizing permutation, which gives the gradient of the minimum wherever the minimizer is unique. ∎

The cell shows what the minimum buys. Each of 80 recordings has the same two voices, which a per-frame logistic model can tell apart from 2-dimensional features. But the reference lists them in a random order in each recording, as real labels do. The fixed pair of voices keeps the toy linear. A real EEND model sees different speakers in every recording and must group frames by voice, using attention over the whole recording.

Python · cell 5

Trained against the labels in their given order, the model stalls at a loss of 0.41. Even scored under the best order for each recording it is at 0.38, because it learned to hedge between the two voices in each slot. With PIT it reaches 0.088. The model picks one consistent assignment, say voice one in slot one, and the loss stops punishing it for the labels' arbitrary order.

Interview

Permutation-invariant loss

Implement the PIT loss twice, once by brute force and once with a linear assignment.

pit_bce(logits, targets) takes two arrays of shape (T, S) with $S \le 4$ . logits holds the model's $z_{t,c}$ and targets the 0 or 1 reference activities. Return (loss, perm). Here loss is $\mathcal L_{\text{PIT}}$ , the mean over all $T S$ entries under the best permutation, found by trying every permutation. perm is a tuple whose entry c is the reference column matched to output slot c, so that targets[:, list(perm)] lines up with logits. Compute BCE from logits in a numerically stable way.
pit_bce_hungarian(logits, targets) returns the same thing using the pairwise cost matrix and scipy.optimize.linear_sum_assignment.

import numpy as np

def pit_bce(logits, targets):
    z, y = np.asarray(logits, float), np.asarray(targets, float)
    loss = np.mean(np.logaddexp(0.0, z) - y * z)        # placeholder: fixed-order BCE
    return float(loss), tuple(range(z.shape[1]))

def pit_bce_hungarian(logits, targets):

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Two refinements are now standard. The first, the powerset formulation (Plaquet and Bredin 2023), replaces the $n_{\text{spk}}$ independent sigmoids with one softmax over sets of speakers. Each class is a set of speakers who talk at once, such as "nobody", "speaker 2 alone", or "speakers 1 and 3 together".

📐 Powerset classes for diarization

For a window with at most $n_{\text{spk}}$ local speakers, of whom at most $K$ speak at once, the powerset classes are the subsets of $\{1, \dots, n_{\text{spk}}\}$ with at most $K$ elements. There are

N_{\text{cls}} = \sum_{k=0}^{K} \binom{n_{\text{spk}}}{k}

of them. A frame whose activity vector $\mathbf y_t \in \{0, 1\}^{n_{\text{spk}}}$ has at most $K$ ones belongs to the class $A_t = \{s : y_{t,s} = 1\}$ , and each class maps back to its indicator vector. The model outputs a softmax over the $N_{\text{cls}}$ classes at each frame and is trained with cross-entropy against $A_t$ . It is decoded by taking the argmax class and mapping it back to speaker activities. Frames with more than $K$ active speakers cannot be represented.

Because speaker labels are arbitrary, the target is first relabeled in four steps. Convert the argmax prediction to activity vectors. Find the speaker permutation that best matches the target, as in permutation-invariant training. Permute the target's activity vectors. Convert them back to classes.

With $n_{\text{spk}} = 3$ and $K = 2$ there are $\binom{3}{0} + \binom{3}{1} + \binom{3}{2} = 7$ classes, namely silence, each speaker alone, and each pair. Three things follow.

No threshold. The argmax removes the detection threshold that multi-label EEND must tune.
The permutation problem stays. Swapping two speakers swaps their singleton classes and every pair that contains them, which is why the matching is done on activity vectors.
Windows must be short. Sixteen speakers with up to two at once would already need $1 + 16 + 120 = 137$ classes, most of them almost never seen in training. So the powerset model runs on short local windows with few speakers.

The second refinement, Sortformer (Park et al. 2024), makes the minimum unnecessary. It orders the targets by arrival time, so slot 1 is always whoever speaks first. The loss can then be plain binary cross-entropy against the sorted targets, which makes diarization easy to train alongside a speech-to-text model. Sorting alone did about as well as PIT, and the best reported model averages the two losses.

Production systems combine the two worlds. The pyannote.audio 3.x pipeline runs a powerset segmentation model on 10-second windows, with at most 3 speakers per window and at most 2 at once. It embeds each local speaker and clusters the embeddings across the whole recording. The end-to-end model handles overlap locally, and clustering handles any number of speakers globally.

Who said which word. Speaker-attributed recognition joins the transcript to the diarization. A common pipeline, of which WhisperX is a popular open-source example, takes word timestamps from CTC forced alignment (§36.2). Its code then gives each word the diarized speaker who overlaps it most. The combined error is measured by cpWER, defined for the CHiME-6 challenge on dinner-party recordings (Watanabe et al. 2020).

📐 Concatenated minimum-permutation WER (cpWER)

For one recording, let $R_1, \dots, R_{K_{\text{ref}}}$ be the reference speakers' word sequences, each the concatenation in time order of all that speaker's utterances. Let $H_1, \dots, H_{K_{\text{hyp}}}$ be the same for the hypothesis speakers. Pad the shorter list with empty sequences to $K = \max(K_{\text{ref}}, K_{\text{hyp}})$ , as the MeetEval toolkit does (CHiME-6 always had four of each). With $E(R, H)$ the word-level edit distance (substitutions plus deletions plus insertions, §36.1) and $N_r$ the number of words in $R_r$ ,

\mathrm{cpWER} = \min_{\pi}\ \frac{\sum_{r=1}^{K} E\big(R_r, H_{\pi(r)}\big)}{\sum_{r=1}^{K} N_r},

the minimum over the $K!$ one-to-one mappings $\pi$ of reference speakers to hypothesis speakers. A hypothesis speaker matched to an empty reference contributes only insertions, and a reference speaker matched to an empty hypothesis only deletions.

The numerator is a sum over matched pairs. So, exactly as for DER, the best mapping is a linear assignment on the $K\times K$ matrix of pairwise edit distances. CHiME-6 had four speakers per session and tried all 24 permutations, and the MeetEval toolkit (von Neumann et al. 2023) uses the Hungarian algorithm.

In plain words, a word with the right text but the wrong speaker usually costs two errors, a deletion from its true speaker's stream and an insertion in the other. Diarization errors therefore show up as word errors. A system that swaps the names of all the speakers but gets every word right still scores 0.

On-device recognition

Running the recognizer on the phone, watch, or car removes the network from the latency budget. It works offline, and it keeps the audio on the device, a privacy property some products require. The constraints are memory, compute, and energy, with tight latency on top.

Memory. Weights dominate. A 600M-parameter model takes about 2.4 GB in fp32 (4 bytes per weight), 1.2 GB in bf16, and 0.6 GB with int8 quantization (§20.2), and int4 halves int8 again. He et al. (2018) quantized a 120M-parameter streaming transducer to 8 bits, cutting its size 4 times to about 120 MB. With symmetric quantization its 90th-percentile real-time factor fell by 64 percent to 0.51, twice as fast as real time on a Pixel phone. It was faster than the conventional on-device system it replaced, with more than 20 percent lower relative WER.

Compute. A matrix multiply costs about 2 FLOPs per parameter per input vector, one multiply and one add (§5.6). An encoder of $N_{\text{enc}}$ parameters producing $1/\Delta_{\text{enc}}$ frames per second therefore needs about $2N_{\text{enc}}/\Delta_{\text{enc}}$ FLOPs per second of audio. A 100M-parameter encoder at 12.5 Hz needs $2\times10^8\times12.5 = 2.5$ GFLOP per second.

That is about 2.5 percent of a mobile accelerator that sustains $10^{11}$ operations per second. Kudlur et al. (2026) design for edge chips that sustain $10^{11}$ to $10^{12}$ operations per second. They note that many reach only tens to hundreds of billions in practice, because memory bandwidth and heat keep them below their peak. Lower frame rates help directly, which is one more reason for 8x subsampling.

Latency on device. A full-attention encoder must re-encode the whole utterance before the decoder can emit, so its time to first token grows with the utterance. Moonshine v2 uses a sliding-window encoder with no positional embeddings, whose cost per frame is constant. On a MacBook M3 its 34M, 123M, and 245M-parameter models respond in 50, 148, and 258 ms. Whisper large-v3 takes 11.3 seconds on the same machine (Kudlur et al. 2026).

Personalization. Contextual biasing boosts known phrases during decoding (§36.5). On device it can use the user's contact names and app names, which never leave the phone.

The wake word comes first

A phone or speaker that listens all day should not spend even the 2.5 GFLOP per second of the encoder above on audio that is almost never meant for it. In front of the VAD and the recognizer sits a keyword spotter. It is a tiny always-on model with one job, deciding whether the last second or so of audio contained the wake phrase ("Hey Siri", "OK Google").

A keyword spotter, its metrics, and its cascade are built in §35.6. Three facts from there matter here. First, the two errors have different units. False accepts are counted per hour of audio with no wake phrase, and false rejects are the fraction of real attempts that fail. Second, a detector is tuned to a fixed false-accept budget, such as Google's 0.1 per hour (one false wake in ten hours), and then judged by its false rejects. Third, it meets that budget cheaply as a cascade. A tiny first stage on a low-power processor fires on a loose threshold and wakes a larger checker, and a speaker verifier or a server-side recognizer can check again.

When the cascade accepts, the device hands the recognizer the recent audio it has been buffering, so the words said right after the wake phrase are not lost. From then on, the VAD and endpointer of the previous sections run the turn.

📌 Wake-word detectors in numbers

Apple's 2017 "Hey Siri" detector on the iPhone ran a network of 5 hidden layers of 32 units on the always-on processor. Only when it fired did the main processor recheck the audio with 5 layers of 192 units. The Apple Watch, with a smaller battery, ran a single pass of intermediate size on about 5 percent of its compute budget, and only after a wrist raise (Siri Team 2017).

Google's end-to-end keyword spotter for "OK Google" and "Hey Google" has about 40K parameters (about 160 KB in float32, 40 KB in int8). It does 20K multiply-accumulates per 20 ms of audio, a million per second. At 0.1 false accepts per hour it rejected 1.52 percent of clean, non-accented attempts, but 35.3 percent of attempts from real voice-search logs with background noise. A 700K-parameter version cut those to 0.87 and 8.9 percent (Alvarez and Park 2019).

✏️ Why two stages win

Take illustrative numbers. The first stage runs at 10 false accepts per hour and a 1 percent false-reject rate. The second stage passes 1 false trigger in 100 and rejects 1 percent of true ones.

False accepts. Together they make $10\times 0.01 = 0.1$ false accepts per hour, the budget.
False rejects. If the two stages err independently, the combined false-reject rate is $1 - 0.99^2 \approx 2$ percent.
Compute. The big model wakes only about 10 times an hour. If each check takes a second, it is busy under 0.3 percent of the time. An always-on stage the size of Google's 40K-parameter model costs a million multiply-accumulates per second, under a thousandth of the 100M-parameter encoder above.

In practice the errors of the two stages are correlated, since audio that is hard for one is often hard for the other. So the combined false-reject rate must be measured, not multiplied.

Every piece of the section now fits on one timeline. Follow one spoken command through it.

✏️ One command, from wake word to final transcript

A user says "Hey Google, call mom" and stops. The device runs the on-device transducer of the latency budget at the start of the section, with 320 ms chunks, 32 ms of compute per chunk, and 40 ms of emission delay. The endpointer and second pass take their medians from Sainath et al. (2020). The word timings and the wake-word timings are illustrative. Times are in seconds from the start of "Hey".

time (s)	what happens
0.00 to 0.60	"Hey Google" is spoken. The always-on first stage fires near its end, and the larger second stage confirms on the main processor by about 0.8 s.
0.60	The recognizer receives the audio buffered since the wake phrase ended, so nothing is lost. Its chunks start here, at 0.60, 0.92, 1.24, 1.56, and 1.88 s.
0.90 to 1.20	"call" is spoken. It ends inside the chunk that fills at 1.24 s.
1.31	"call" appears as a partial result, $1.24 + 0.032 + 0.04$ , 110 ms after the word ended.
1.35 to 1.60	"mom" is spoken, and the user stops. It ends inside the chunk that fills at 1.88 s.
1.95	"mom" appears as a partial, 350 ms after it ended.
2.03	The endpointer closes the microphone, 430 ms after the end of speech.
2.09	The second pass rescores the first pass's hypotheses in 58 ms, and the final transcript "call mom" is ready.

The partials arrive 110 to 350 ms after each word, depending on where the word falls in its chunk. The final transcript arrives 490 ms after the user stopped, and a person would have started answering within 200 ms. Most of the gap is the endpointer, which is where a design review should spend its time.

Serving at scale

Server-side recognition comes in two shapes, and each has its own arithmetic.

Batch transcription of stored audio (call centers, podcasts, meetings, media archives) maximizes throughput. A VAD cuts each recording into segments of up to about 30 seconds at pauses. Segments are sorted by length and batched, so little compute is wasted on padding. Long stretches without pauses are split into overlapping windows, and the transcripts are stitched together in the overlap by aligning their timestamps or their words. LLM decoders use the continuous batching of §20.6.

Throughput is measured as RTFx, audio duration divided by processing time (§36.1). At RTFx 3,390, one fully busy GPU transcribes an hour of audio in $3600/3390 = 1.06$ seconds.

The cell applies the same arithmetic to models from the Open ASR Leaderboard paper. Its RTFx values were measured on one A100-80GB GPU with batch 64 where memory allowed, with numbers as of 27 March 2026. The cell assumes $2 per GPU-hour and 60 percent utilization.

Python · cell 6

The fastest transducer and CTC models cost about a dollar per thousand hours and need 3 GPUs for the whole load. Whisper large-v3 costs more than $22 and needs 48. That factor of more than 20 in throughput is the decoder story of this chapter, in money. Frame-synchronous decoders emit in one pass, while autoregressive decoders generate token by token.

Streaming transcription is constrained by latency instead. Every stream sends a chunk every $t_{\text{chunk}}$ . The server must process all the chunks that arrived in one chunk period within that period, or it falls behind. Suppose one stream's chunk costs $t_{\text{comp}}$ of GPU time on its own, and batching chunks from many streams raises throughput by a factor $g_{\text{batch}}$ . Then one GPU sustains at most $\lfloor g_{\text{batch}}\, t_{\text{chunk}} / t_{\text{comp}}\rfloor$ streams. With 320 ms chunks, 4 ms per chunk, and $g_{\text{batch}} = 2.5$ , that is 200 streams.

The fleet is sized for peak concurrency, not daily volume. Each stream also holds its caches in GPU memory between chunks, which is the streaming version of the KV-cache budget of §20.6. In a system-design interview, produce two numbers, streams per GPU from compute and streams per GPU from cache memory. The smaller one wins.

Core

Sizing a fleet

Turn the serving arithmetic into code. Implement three functions.

gpus_for_batch(audio_hours_per_day, rtfx, utilization) returns the whole number of GPUs needed to keep up with a daily volume of batch audio.
streams_per_gpu(chunk_ms, compute_ms_per_chunk_per_stream, batch_efficiency) returns the largest whole number of real-time streams one GPU can serve, as defined in the text.
cost_per_audio_hour(gpu_dollars_per_hour, rtfx, utilization) returns the cost of one audio hour in dollars.

def gpus_for_batch(audio_hours_per_day, rtfx, utilization):
    return audio_hours_per_day / rtfx          # placeholder: forgets the day and the utilization

def streams_per_gpu(chunk_ms, compute_ms_per_chunk_per_stream, batch_efficiency):
    raise NotImplementedError

def cost_per_audio_hour(gpu_dollars_per_hour, rtfx, utilization):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Monitoring in production. Offline benchmarks stop being informative the day a system ships, because traffic drifts to new devices, new accents, and new product names. Mature teams sample live traffic, have people transcribe it, and track WER with the text normalization and confidence intervals of §36.5. They break it down by device, locale, and domain. They also watch

latency percentiles for partial and final results, separately,
the VAD's missed speech, visible as users who had to repeat themselves,
endpointer cut-offs,
and, for encoder-decoder models, hallucination detectors such as the compression-ratio and log-probability checks of §36.4.

⚠️ Five ways to fool yourself about streaming

A non-causal piece hides in the encoder. One centered convolution, utterance-level feature normalization, or a non-causal subsampling stem breaks every latency claim. Inside a chunked encoder, a centered convolution adds a chunk per layer. Check by perturbing future input frames and confirming that past outputs do not change.
Latency measured without computation. Simulations that assume instant processing hide 1.7 to 2.3 seconds for Whisper-Streaming. Measure on the target hardware under realistic load.
Averages hide tails. A mean latency of 300 ms with a 90th percentile of 2 seconds feels broken. Report percentiles, and report partial and final latency separately.
Training without the streaming mask. Streaming a model trained offline loses accuracy badly (11.3 against 6.3 percent in the cache-aware comparison). Train with the masks you will serve, and use dynamic chunks if you need several latencies.
A DER or VAD number without its conventions. On the same output in the widget above, the collar and the treatment of overlap move DER from 12.2 to 0.9 percent. Always state them.

🧭 What to remember

Latency is several quantities. Algorithmic latency comes from the architecture, computational latency from the hardware, emission delay from training, and finalization latency from the endpointer and any second pass. Gaps between turns in human conversation are mostly under 200 ms, and the endpointer, not the encoder, usually dominates the budget.
Lookahead composes through depth. A lookahead of $r$ frames in every layer costs $Lr$ frames (1,360 ms for 17 layers of 40 ms frames at $r = 2$ ). Chunked attention caps it at $n_{\text{chunk}} - 1$ frames at any depth, provided convolutions are causal and lookahead frames are recomputed privately for each chunk.
The chunked mask lets frame $i$ see chunks $c(i) - n_{\text{left}}$ to $c(i)$ . Trained with that mask and run with convolution and attention caches, a streaming encoder reproduces its training-time computation exactly, and dynamic chunk training lets one model serve many latencies.
Attention decoders stream in three ways. A two-pass system rescores a streaming first pass. LocalAgreement re-decodes an offline model and commits what two runs agree on, at the price of at least $1.5c$ of latency and repeated compute. Delayed streams align the text to the audio with WORD and PAD tokens, shift it by a fixed delay, and train a decoder-only model to write it as the audio arrives (Voxtral Realtime beats offline Whisper at 960 ms).
A cascaded wake-word detector, tuned to a false-accept budget per hour, decides when to listen at all. A VAD with hysteresis, pause bridging, burst removal, and padding gates compute and hallucination, and an endpointer trained into the recognizer closes the microphone sooner without cutting people off.
DER counts false alarm, miss, and confusion over total speaker time under the optimal mapping, a linear assignment, and its conventions must be stated. End-to-end diarization trains with permutation-invariant loss, which is also a linear assignment, and modern pipelines combine local powerset models with global clustering. For speaker-attributed transcripts, cpWER scores words and speakers together under the best speaker mapping.
Serving is arithmetic. GPUs equal hours per hour over RTFx times utilization, streams per GPU equal batched compute per chunk period, and on device int8 weights and low frame rates set the budget. The next chapter runs recognition in reverse, from text to speech, and builds up to full-duplex voice models that put recognition, a language model, and synthesis inside one turn-taking loop. The system design interviews that close the book ask for exactly this latency and cost arithmetic.

Noroozi et al. (2023). Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition. Causal convolutions, chunk-aware lookahead, the activation caches, and the multi-latency training, with the 1,360 ms lookahead example and the comparison against buffered streaming.
Zhang et al. (2020). Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition, and Shi et al. (2020), Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition. Dynamic chunk training with CTC-first-pass rescoring, and the memory bank with per-chunk right context. Chiu & Raffel (2017), Monotonic Chunkwise Attention, is the attention-based streaming method of the same period.
Macháček, Dabre & Bojar (2023). Turning Whisper into Real-Time Transcription System. LocalAgreement over an offline model, the buffer management, and an honest latency evaluation with and without computation.
Zeghidour et al. (2025). Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling, with Mistral AI (2026), Voxtral Realtime, and Kudlur et al. (2026), Moonshine v2: Ergodic Streaming Encoder ASR for Latency-Critical Speech Applications. Natively streaming recognition at the 2026 frontier, on servers and on edge devices. Défossez et al. (2024), Moshi: a speech-text foundation model for real-time dialogue, introduced the aligned text stream that DSM builds on. The Qwen3-ASR report, Qwen Team (2026), Qwen3-ASR Technical Report, describes its dynamic attention windows.
Sainath et al. (2020). A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency, and He et al. (2018), Streaming End-to-end Speech Recognition For Mobile Devices. Two-pass rescoring, the end-of-query endpointer with its penalties, quantization, and every latency measurement in this section's budget. For the wake word in front of them, Siri Team (2017), Hey Siri: An On-device DNN-powered Voice Trigger for Apple's Personal Assistant, and Alvarez & Park (2019), End-to-End Streaming Keyword Spotting.
Fujita et al. (2019). End-to-End Neural Speaker Diarization with Self-attention, Plaquet & Bredin (2023), Powerset multi-class cross entropy loss for neural speaker diarization, and Park et al. (2024), Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems. The three ways end-to-end diarization handles the permutation problem. Watanabe et al. (2020), CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings, defines cpWER, and von Neumann et al. (2023), MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systems, implements it.
pyannote.metrics reference, pyannote.audio, and Silero VAD. The exact definitions of DER, collars, and detection cost, a production diarization pipeline, and a production VAD with the post-processing defaults quoted here.
Stivers et al. (2009). Universals and cultural variation in turn-taking in conversation. The ten-language measurement of response gaps that sets the latency budget for spoken systems.

Made for larger screens

Streaming and production ASR

What latency means

Encoders that do not look ahead

Causal convolutions

How lookahead accumulates across layers

The chunked attention mask

A chunked streaming mask

Caches make streaming exact

One model for many latencies

Decoders that write before the audio ends

Two passes

Streaming an offline model by agreement

LocalAgreement

Delayed streams

Delayed text streams

Voice activity detection and endpointing

VAD with hysteresis

Endpointing

Diarization, who spoke when

Diarization error rate

The cascade, and why it misses overlap

End-to-end diarization and permutation-invariant training

Permutation-invariant loss

On-device recognition

The wake word comes first

Serving at scale

Sizing a fleet

Read more

Watch next