A transducer is a speech recognizer that reads the audio one frame at a time and, before each decision, also looks at the part of the transcript it has already written. The transcript is written in labels, characters or subword tokens. It is the standard model for recognizers that transcribe while you talk, such as dictation on phones. A variant of it is among the fastest accurate open recognizers of 2026. This section builds it, trains it, decodes with it, and makes it fit in GPU memory.

The transducer starts from connectionist temporal classification, CTC, the recognizer of §36.2, so here is CTC in a few sentences. An encoder frame is a vector that summarizes a few tens of milliseconds of audio. At every encoder frame the network outputs a distribution $p_t$ over the labels plus one extra symbol, the blank $\varnothing$, which means "no new label at this frame". A string of one symbol per frame spells a transcript once repeated symbols are merged and blanks are deleted. The probability of a transcript is the sum of the probabilities of all the strings that spell it.

Given the audio $\mathbf X$, CTC picks the symbol at each frame independently of the symbols at the other frames. A string $\mathbf a = (a_1,\dots,a_T)$ therefore has probability $p(\mathbf a\mid\mathbf X) = \prod_t p_t(a_t)$. In plain words, the network never sees the labels it has already written.

Here is what that costs. Suppose a name that sounds the same either way is spelled "Caitlin" by half the transcribers in your training data and "Katelyn" by the other half. A model that reads its own output would commit to one spelling and finish it. CTC picks the letters at different frames independently, so nothing stops it from writing "Caitelyn", a mixture that no transcriber produced. An encoder that hears the whole utterance does not help, because the audio is the same for both spellings. The missing information is not in the sound. It is in the labels.

An attention encoder-decoder, the design of §36.4, writes the transcript one label at a time and conditions each label on all the previous ones. Its decoder attends over the whole encoded utterance, though, so it must wait until the speaker has finished. We want both, dependence on previous labels and left-to-right processing of the audio. The transducer of Graves (2012) gives both. It is often called the RNN-Transducer, or RNN-T, after the recurrent networks it first used. It keeps CTC's blank and its sum over alignments, reads the audio frame by frame, and adds a small network that reads the labels written so far.

It matters in practice. Google's first all-neural recognizer to run on a Pixel phone was a transducer (He et al. 2018). On the Open ASR Leaderboard, a public benchmark of recognizers, the only top-ten model as of 27 March 2026 measured at more than a thousand times faster than real time is a transducer variant (Srivastav et al. 2025).

This section answers five questions. How does a transducer assign a probability to a transcript, and how do we compute it and its gradient? How do we decode with it? Why does its training need so much memory, and how is that cut? Why do streaming transducers emit words late, and how is that fixed? And what do the 2026 variants change? It ends with a full comparison against CTC.

A transducer has three networks

The encoder turns the audio into a sequence of vectors, the same shared encoder as in §36.1. It reads log-mel frames, the audio's spectrum every 10 ms on a perceptual frequency scale (§35.3). It subsamples them in time and runs a stack of Conformer blocks, Transformer blocks with an added convolution. Out comes one vector $\mathbf h^{\text{enc}}_t$ per encoder frame, $t = 1,\dots,T$, typically 40 or 80 ms apart.

The prediction network reads the labels emitted so far and summarizes them in a vector $\mathbf h^{\text{pred}}_u$, where $u$ is the number of labels emitted. It never sees audio. Its job looks like that of a small language model, a next-token predictor as in §6.2. Later in the section we will see how far that resemblance goes.

The joint network takes one vector of each kind and outputs a distribution over the labels plus the blank. Its decision is one of two. Emit a label and stay on this frame, or emit the blank and move to the next frame.

Decoding is a walk. Start at the first frame with nothing emitted. At every step, ask the joint network about the current frame and the current labels. If it says a label, write the label, update the prediction network, and stay on the same frame. Staying is needed because one frame of audio can hold more than one label. English at 150 words a minute, with about six characters a word counting the space, is 15 characters a second, more than one per 80 ms frame. If the joint network says blank, move to the next frame. After the blank at the last frame, stop.

The walk lives on a grid. The horizontal axis is the frame $t$ and the vertical axis is the number of labels emitted, $u$. A blank moves one step right and a label moves one step up. Every path from the bottom left corner to the top right corner, followed by one last blank, is an alignment, one way of placing the transcript's labels on the frames. The loss sums the probabilities of all of them.

The lattice and the joint distribution

Now the same picture in symbols. The vocabulary $\mathcal V$ has $V$ labels, characters or subword tokens (§2.2). Adding the blank gives the extended vocabulary $\bar{\mathcal V} = \mathcal V\cup\{\varnothing\}$ of $V+1$ symbols, exactly as in CTC.

In plain words, there is one distribution for every pair (frame, number of labels so far). It depends on the audio only through $\mathbf h^{\text{enc}}_t$ and on the transcript only through $y_1,\dots,y_u$. It does not depend on how the walk reached $(t, u)$. That is what will make dynamic programming possible.

The transducer differs from CTC in three ways, and each one changes an algorithm later in the section.

1. A CTC frame emits exactly one symbol. A transducer frame emits any number of labels and then exactly one blank, so the blank now means "I am done with this frame".
2. Nothing is merged. The two l's of "hello" are two label moves in a row, with no blank needed between them.
3. Every symbol is conditioned on the labels emitted before it, which is the whole point.

The first difference removes CTC's length condition. CTC needs $T \ge U + R(\mathbf y)$ frames, where $R(\mathbf y)$ counts adjacent repeated labels (§36.2). Each label takes a frame, and each repeat needs a blank between its copies. A transducer can emit any number of labels from any number of frames, even more labels than frames. That matters with long frames and short labels, for example characters at 80 ms frames.

The transducer also defines a proper distribution over transcripts of every length. The final blank is what makes it one.

In plain words, if you run the walk as a random process, it stops, and each run spells exactly one transcript. The probabilities of all transcripts therefore add up to one. That is why maximizing $p(\mathbf y\mid\mathbf X)$ pushes down every other transcript, including ones with extra labels at the end.

The smallest lattice worth drawing has two frames and one label. Here it is with every probability written out, so you can trace both of its alignments by hand.

The widget draws the same lattice at a larger size. Set $T = 6$ and $U = 3$ and build the path that emits all three labels at the first frame and then only blanks. Before you read its share of $p(\mathbf y\mid\mathbf X)$, predict how it compares with the share of the most likely path, which the "Most likely path" button draws. Then turn on the occupancy view, which shades each node by the probability that the alignment passes through it, and find the nodes that almost no alignment visits. Leave the FastEmit slider at zero for now. It belongs to the end of the section, where we explain why transducers emit late.

What conditional independence costs

The spelling problem of the opening can be made exact with the smallest possible example. Suppose the training data holds one input, two frames long. Half the time it is transcribed "ab" and half the time "ba", from identical audio. The best any model can do is to put probability $\tfrac12$ on each transcript. That is a loss of $\log 2 = 0.693$ nats per utterance, the entropy of the labels.

The cell lists the four strings of the optimal CTC model and the transcript each one collapses to.

The gap is $\log 4 - \log 2 = \log 2$ nats, one bit per utterance. CTC loses something like it whenever the audio leaves several transcripts open and the labels have structure. In practice CTC systems recover much of it with an external language model at decoding time (§36.5). The transducer builds the dependence into the model. An interviewer who asks "what does a transducer buy over CTC" wants this example.

Counting the alignments

How many alignments does the loss add up? Every alignment is a path through the lattice, so the question is how many lattice paths there are.

Proof. A path from $(1,0)$ to $(T,U)$ makes $T-1$ blank moves and $U$ label moves. Any order of these $T-1+U$ moves is a valid path, because a sequence with exactly $T-1$ blanks and $U$ labels can never cross either edge of the grid. Choosing which $U$ of the $T - 1 + U$ positions hold the labels fixes the path. The final blank is forced, so it adds no choice. ∎

For $(T,U) = (4,2)$ that is $\binom{5}{2} = 10$. The CTC count for the same pair is different, and it depends on the labels. The cell computes it with the CTC forward recursion of §36.2, run on probabilities that are all 1. With every probability equal to 1, the recursion counts paths instead of adding probabilities.

The output shows three differences. For "ab" over 4 frames CTC has 15 alignments and the transducer 10. For the repeated "aa", CTC drops to 5, because a blank must separate the copies, while the transducer count does not change. For three labels in three frames CTC has exactly one alignment, and in two frames none, while the transducer has 10 and 4.

At a realistic size, 20 seconds at 40 ms frames ($T = 500$) with 100 subword labels, the count is a number with 116 digits. As in CTC, only a dynamic program can sum over that many paths, and the next section builds it.

def n_transducer_alignments(T, U):
    raise NotImplementedError

The forward algorithm

To add up $10^{115}$ paths, group them by their last step. Every path into a node arrives either by a blank from the node on its left or by a label from the node below it. So the total probability of reaching a node is the sum of two terms. One is the total for the left neighbor times its blank probability. The other is the total for the lower neighbor times its label probability. Every lattice in this book is summed this way, from the segmentation lattice of §2.4 to CTC's trellis.

Watch the indexing of the label probability. The edge into $(t,u)$ from below carries $y(t,u-1) = p(y_u\mid t,u-1)$. That is the probability of the $u$-th label, evaluated at the node where only $u-1$ labels were known. Writing $p(y_u\mid t,u)$ there is the most common bug in transducer code, because it lets the model see a label before predicting it. Interviewers look for it.

The order of computation matters on hardware. Every node on an anti-diagonal depends only on the previous anti-diagonal. So a whole anti-diagonal can be computed in parallel, and $T+U-1$ sequential sweeps finish the lattice. GPU implementations sweep this wavefront, and so does the NumPy version below.

The loss in code

In code the frames are 0-based, so the text's node $(t, u)$ is lp[t - 1, u]. Everything is in log space. Early in training a 20-second utterance has a probability far below the smallest float, since 600 steps at a probability near 1/1,000 each multiply to about $10^{-1800}$. The forward pass sweeps anti-diagonals with np.logaddexp, the stable two-term log-sum-exp of §0.4. The cell checks it against a brute force that enumerates all $\binom{T-1+U}{U}$ paths of the counting theorem.

The forward table of the hand example is $\alpha(1,0) = 1$, $\alpha(2,0) = 0.6$, $\alpha(1,1) = 0.3$, and $\alpha(2,1) = 0.3\times0.7 + 0.6\times0.4 = 0.45$. Then $p = 0.45\times0.8 = 0.36$, the number we found by listing paths. On random joint distributions the dynamic program and the brute force agree to ten digits.

import numpy as np

def rnnt_loss(log_probs, y, blank=0):
    raise NotImplementedError

The gradient, and where it flows

Training needs $\partial\mathcal L/\partial\mathbf z_{t,u}$, the gradient of the loss with respect to the joint logits at every node. The forward and backward variables give it in closed form. It has the "predicted minus target" shape of softmax cross-entropy (§0.2) and of CTC's gradient (§36.2). It also has one difference from CTC, and that difference is what later makes pruning possible.

Two quantities appear. Both are probabilities under the posterior over alignments, the distribution over the alignments of $\mathbf y$ in which each alignment has probability $p(\mathbf a\mid\mathbf X)/p(\mathbf y\mid\mathbf X)$. The edge occupancies are the probabilities that the alignment leaves node $(t,u)$ by its blank edge or by its label edge,

with the convention $\beta(T+1,U) = 1$ for the final blank, and $\gamma_{t,u}(k) = 0$ for every other symbol $k$. The node occupancy is the probability that the alignment visits $(t,u)$ at all, $o_{t,u} = \alpha(t,u)\beta(t,u)/p(\mathbf y\mid\mathbf X)$. The alignment leaves every node it visits by exactly one edge, so $o_{t,u} = \gamma_{t,u}(\varnothing) + \gamma_{t,u}(y_{u+1})$.

In plain words, each node does a cross-entropy update toward soft targets, the posterior edge counts $\gamma_{t,u}$. The update is weighted by $o_{t,u}$, the probability that the alignment visits that node. That weight is the difference from CTC. A CTC frame is always visited, so every frame's softmax gets a full update. A transducer node off the likely alignments has $o_{t,u}\approx 0$, so its softmax is barely touched. Pruned training, later in this section, exploits exactly that.

The formula also saves memory. Computing the logit gradient directly, without ever storing the gradient with respect to the probabilities, is what Li et al. (2019) called function merging.

The cell checks the formula against central differences, the numerical derivative $(f(z+\epsilon) - f(z-\epsilon))/2\epsilon$. Then it sums the occupancy over anti-diagonals and over frames, as the pitfall above warned. The function also takes a FastEmit weight lam, which is zero here and is explained later in the section.

The analytic gradient matches to about $10^{-9}$. Every anti-diagonal holds exactly one unit of occupancy. The four column sums are 2.011, 1.083, 2.750, and 1.157. They add up to $T + U = 7$, the number of nodes on every path. The first column's 2.011 says the alignment visits about two nodes of frame 1, so it emits about one label there.

import numpy as np

def rnnt_grad(logits, y, blank=0):
    p = np.exp(_log_softmax(logits))
    return p        # placeholder: the softmax output, not the gradient

The loss and its gradient in PyTorch

Autograd will differentiate the forward recursion for us. That makes a good check of the formula and a bad production implementation, because it records $O(TU)$ small operations. The block runs the recursion with torch.logsumexp, lets autograd produce the gradient, and compares it with the closed form.

The loss matches the enumeration of all 35 alignments. Autograd's gradient matches the closed form to about $10^{-16}$, rounding error in float64.

In a real training loop you do not write the recursion. Libraries implement it in CUDA with the closed-form gradient, and they take the joint logits for a padded batch. The call below is the torchaudio interface, shown and not run, since torchaudio is not installed here. Its default fused_log_softmax=True is function merging, so you pass raw logits, not log-probabilities.

The joint network and the prediction network

The joint network decides how the audio and the label history combine. Graves (2012) used the simplest choice. The encoder outputs $V+1$ numbers $\mathbf f_t$ for frame $t$, the prediction network outputs $V+1$ numbers $\mathbf g_u$ after $u$ labels, and the joint distribution is the softmax of their sum, $p(k\mid t,u)\propto\exp(f_t(k) + g_u(k))$. In plain words, it multiplies an acoustic distribution by a linguistic one and renormalizes. It is cheap, since $e^{f_t(k)+g_u(k)} = e^{f_t(k)}e^{g_u(k)}$. That takes $(T+U+1)(V+1)$ exponentials instead of $T(U+1)(V+1)$, and no layer runs at every node.

Graves, Mohamed, and Hinton (2013) replaced the sum with a small feedforward network,

with $\mathbf W_{\text{enc}}\in\R^{d_{\text{joint}}\times d_{\text{enc}}}$, $\mathbf W_{\text{pred}}\in\R^{d_{\text{joint}}\times d_{\text{pred}}}$, and $\mathbf W_{\text{out}}\in\R^{(V+1)\times d_{\text{joint}}}$. The biases are $\mathbf b\in\R^{d_{\text{joint}}}$ and $\mathbf b_{\text{out}}\in\R^{V+1}$, and $\phi = \tanh$ in their paper (ReLU is also common). In plain words, project both vectors to a common width $d_{\text{joint}}$, add them, apply a nonlinearity, and map to $V+1$ logits. They reported that it generalized better and in particular made fewer deletion errors, words dropped from the transcript. This form, with $d_{\text{joint}}$ around 320 to 1,024, has been the joint network of most transducers since.

The formula decides the cost of training. The projections $\mathbf W_{\text{enc}}\mathbf h^{\text{enc}}_t$ and $\mathbf W_{\text{pred}}\mathbf h^{\text{pred}}_u$ are computed once per frame and once per label. They are then broadcast into a $T\times(U+1)\times d_{\text{joint}}$ grid, which is cheap. The output layer, however, runs at every node, $T(U+1)$ times, with $d_{\text{joint}}(V+1)$ multiply-adds each. That is where most of the memory and much of the compute of training go, and we measure it later in the section.

The prediction network starts from a start symbol (often the blank itself) and is fed each emitted label in turn. Graves (2012) used one LSTM layer, a recurrent network with a gated memory (§3.3). He et al. (2018) used two LSTM layers of 2,048 units. It needs surprisingly little context.

The evidence comes from LibriSpeech, a corpus of read audiobooks with two standard test sets, test-clean and the harder test-other. Results are given as word error rate (WER), the number of word substitutions, deletions, and insertions divided by the number of reference words (§36.1). Zhang et al. (2020) built the Transformer Transducer, with Transformer encoders for both the audio and the labels. They limited the label encoder's attention to the previous few labels. In a streaming configuration, 3 previous labels per layer gave 4.2 and 11.4 percent WER on test-clean and test-other, against 4.2 and 11.3 with 20.

Ghodsi et al. (2020) went further. They removed the recurrent layers and kept a stateless prediction network, one with no memory, that embeds only the last label. With wordpiece labels, a kind of subword token, it performed virtually as well as the original. Their reading is that the prediction network acts less like a language model and more like a helper for alignment. The encoder and the joint carry most of the linguistic knowledge. Modern open recipes follow suit. The pruned transducers of the open k2 toolkit embed the last two labels and apply a width-2 convolution (Kuang et al. 2022). NVIDIA's token-and-duration models concatenate the embeddings of the last two labels (Xu et al. 2023).

A stateless predictor also simplifies decoding. Two candidate transcripts that end in the same two labels have identical predictor outputs, so the output is computed once and reused. In an interview, this is the answer to "how much does the prediction network matter".

However small it is, the prediction network, together with the joint, learns a prior over label sequences from the transcripts paired with the audio. That prior is the transducer's internal language model (internal LM). It matters when you add an external language model trained on text, because a text prior would then be counted twice. In a standard transducer the internal LM is tangled up with the blank, since one softmax spreads its mass over the blank and the labels together. Variani et al. (2020) separated the two.

In plain words, HAT's label softmax answers "which label, given that one is emitted", a distribution over $\mathcal V$ alone. Deleting the acoustic term leaves a distribution over the next label given only the previous ones, the internal LM. Its log-probability can then be subtracted, with a weight, before an external LM is added. On a large Google voice search task with phoneme labels, decoded with a pronunciation lexicon and an external n-gram LM, HAT had 6.6 percent first-pass WER against 8.0 for a standard transducer. Both models had 115M parameters. §36.5 covers internal LM estimation and fusion in full.

Greedy and beam search decoding

Training sums over alignments. Decoding must find one likely transcript. The simplest way is the walk from the start of the section, taking the most likely symbol at every step, which is called greedy decoding.

The prediction network runs only on labels, never on blanks. Its job is to summarize the transcript, and a blank is not part of it. The cap exists because nothing forces the joint network to emit a blank. An undertrained model, or one fed unusual audio, can keep proposing labels on one frame, and without a cap the loop never ends. NeMo, NVIDIA's speech toolkit, allows 10 labels per frame in its default greedy decoder. When the cap is reached, the decoder moves on as if it had emitted a blank. Both rules, the cap and "feed only labels", are what an interviewer checks in a greedy decoder you write on a whiteboard.

import numpy as np

def rnnt_greedy(enc, pred_step, joint, start_state, start_token, blank=0, max_symbols=10):
    hyp = []
    h_pred, state = pred_step(start_state, start_token)
    for t in range(len(enc)):
        k = int(np.argmax(joint(enc[t], h_pred)))   # one symbol per frame, like CTC
        if k != blank:

Beam search

Greedy decoding commits to one alignment and can never take back a mistake. Beam search keeps the $b$ best partial transcripts, called hypotheses, at every step, where $b$ is the beam width. In a transducer that is harder than for CTC (§36.2) or for a left-to-right language model (§3.5), for two reasons. Hypotheses at the same frame can hold different numbers of labels. And the same label sequence can be reached by many alignments, whose probabilities should be added.

The common fix is to allow at most one label per frame during search. Kang et al. (2022) showed that under this one-label limit a label step can also advance the frame. Every hypothesis then moves in lockstep, one frame per step, and the search becomes as regular as CTC's prefix beam search. The open k2 recipes, called icefall, name the result modified beam search.

Each frame costs three things. There are $b$ joint evaluations, one per hypothesis, each giving all $V+1$ symbols. There is a top-$b$ selection over at most $b(V+1)$ merged candidates. And there are at most $b$ prediction network steps, one for each new history. For a batch of $B$ utterances, all hypotheses advance together, so a frame is one joint call on $B\,b$ rows. He et al. (2018) cached prediction network outputs by label history, since many hypotheses share histories. That saved 50 to 60 percent of the predictor computation on device.

What does this search maximize? The theorem below says exactly. It is worth knowing for an interview, and the proof is optional.

Compare $Q$ with the transducer likelihood. A transducer path that emits a label at frame $t$ must still pay for the blank that ends frame $t$. Here the label itself ends the frame, as if the blank after one label had probability 1, which is how Kang et al. describe it. $Q$ is the likelihood of what they call the modified transducer. It is also the likelihood of the token-and-duration transducer, defined later in this section, with every duration equal to 1. Exercise 36.3g tests that case.

Decoding a model trained with the ordinary loss this way costs little accuracy. On LibriSpeech, Kang et al. measured greedy WER rising from 2.68 and 6.28 percent on test-clean and test-other to 2.72 and 6.33 under the one-label limit. Their constrained transducer, which builds the limit into its loss, gave the same WER with and without the limit (2.76 and 6.46). Training with the modified likelihood itself gave 3.68 and 9.43 in their basic setup, much worse. With the smoothed trivial loss of pruned training, described later in this section, the modified transducer recovered to 2.62 and 6.41, close to the ordinary one. The usual practice is still to train the ordinary transducer and apply the limit only at decoding time, since it costs a few hundredths of a point.

import numpy as np

def rnnt_modified_beam_search(enc, predict, joint, beam, blank=0):
    raise NotImplementedError

The memory wall of the joint tensor

Transducer training is known for running out of GPU memory, and the reason is one tensor. CTC's output for a batch of $B$ utterances is a tensor of shape $B\times T\times(V+1)$. The transducer evaluates its joint network at every node of every lattice, so its logits form a four-dimensional tensor, $(U+1)$ times larger.

A naive autograd implementation stores more than the logits. It keeps the softmax output for the backward pass, the gradient with respect to the probabilities, and the gradient with respect to the logits. That is about three tensors of the logits' size. The joint's hidden layer adds $B\,T\,(U+1)\,d_{\text{joint}}$ more numbers, 4.1 GB in float32 at $d_{\text{joint}} = 640$.

The compute is lopsided too. The output layer alone costs $2\,T(U+1)\,d_{\text{joint}}(V+1)\approx 6.6\times10^{10}$ FLOPs (floating-point operations) per utterance here. Compare an encoder with about 118M parameters, the size of the large Conformer model of Gulati et al. (2020), whose parameters are almost all in its encoder. Counting two FLOPs per parameter per frame, it costs about $2\times118\text{M}\times500\approx1.2\times10^{11}$. So the joint's output layer alone costs about half as much as the whole encoder. Practitioners use five fixes, usually several at once.

1. Remove padding. Utterances in a batch have different lengths $T_n$ and $U_n$, and a padded joint has $B\max_n T_n\,(\max_n U_n+1)$ rows. Li et al. (2019) evaluated the joint per utterance and concatenated the results, $\sum_n T_n(U_n+1)$ rows.
2. Merge the softmax into the loss. With the gradient theorem, the logit gradient $o_{t,u}p(k\mid t,u) - \gamma_{t,u}(k)$ can overwrite the softmax output in place, so only one tensor of joint size lives at a time (Li et al. 2019). Torchaudio's fused_log_softmax=True is this, and it is the same idea as the fused kernels of §11.7.
3. Shrink the factors. Use a lower frame rate, fewer and longer subword units, and bfloat16 logits. Frames 80 ms apart halve $T$ compared with 40 ms, one reason for the 8x subsampling of NVIDIA's FastConformer encoder (§36.1).
4. Split the batch for the joint. Run the encoder on the whole batch, then evaluate the joint and the loss on sub-batches, one at a time, so the joint's peak memory is that of a sub-batch. NeMo calls this fused batch processing (fused_batch_size).
5. Prune the lattice. Evaluate the full joint only near the alignment, as the next subsection shows.

def joint_bytes(B, seconds, frame_ms, U, V, dtype_bytes=4, prune=None):
    T = int(seconds * 1000 / frame_ms)
    return B * T * U * V * dtype_bytes

def max_batch(budget_bytes, seconds, frame_ms, U, V, dtype_bytes=4, prune=None, tensors=1):
    raise NotImplementedError

The widget makes the same count for a batch of 32 utterances, with the clip length on a slider. Both $T$ and $U$ grow with the clip, so the joint grows with the square of its length, while a pruned joint grows only linearly, like CTC's output.

Pruned RNN-T

The gradient theorem said that a node's softmax gets a gradient proportional to its occupancy, the probability that the alignment visits it. In a trained model the occupancy is concentrated in a narrow band around the most likely alignment. Kuang et al. (2022) turned that into a two-pass algorithm. First a cheap joint finds the band. Then the full joint runs only inside it.

In plain words, a cheap additive joint guesses where the alignment runs. The expensive joint is then evaluated only in a band of $S_{\text{prune}}$ label positions around that guess, at every frame. The trivial joiner is the additive joint of Graves (2012) from earlier in this section. It is cheap because its normalizer never needs the full $T\times(U+1)\times(V+1)$ tensor.

The cell builds a lattice that looks trained, with $T = 200$ and $U = 40$. It picks a band of $S_{\text{prune}} = 5$ positions per frame from the node occupancy, which is simpler than the paper's edge-occupancy bound, and then enforces the constraints of the definition. Then it measures how much of the transcript's probability survives. It also checks that the trivial joiner's normalizer is one matrix product.

With 12 percent of the joint tensor, the pruned lattice keeps 99 percent of the transcript's probability. In the batch of the joint memory fact, $S_{\text{prune}} = 5$ shrinks the logits from 6.6 GB to $5/101$ of that, 328 MB.

Here is how the k2 recipes wire it, shown and not run, since k2 is not installed here. The trivial joiner uses its own small projections to $V+1$ outputs, and the full joiner sees only the pruned positions. In an interview about transducer memory, this two-pass design is the answer to give. Its one requirement is a trivial joiner good enough to find the band, which is why the trivial loss stays in the objective.

Streaming, emission delay, and FastEmit

Streaming means transcribing while the person is still talking. A transducer streams naturally because of what each decision needs. The joint at $(t,u)$ reads $\mathbf h^{\text{enc}}_t$ and $\mathbf h^{\text{pred}}_u$, and nothing else. Suppose the encoder is causal, seeing no future audio, or sees only a bounded lookahead (§36.6 builds such encoders with chunked attention masks). Then $\mathbf h^{\text{enc}}_t$ is ready a fixed time after frame $t$ is heard. The prediction network depends only on labels already emitted. Greedy and one-label-per-frame beam decoding move frame by frame, so the transcript grows while the person talks.

An attention decoder, by contrast, computes each label from cross-attention over all encoder frames (§4.6). So it cannot be sure of even its first label until the input has ended. Making it stream takes extra machinery, which §36.6 describes. "Why do transducers stream and encoder-decoders not" is a standard interview question, and these two paragraphs are the answer.

Streaming transducers have one systematic problem, and it comes from the loss. The loss sums over all alignments of the transcript. Nothing in it prefers an alignment that emits "weather" as the word ends over one that emits it 300 ms later. Only the total probability of the transcript counts. A streaming encoder, meanwhile, is more certain the more audio it has seen, so waiting pays. Training therefore drifts toward late emission, which the user sees as words appearing behind their speech. That lag is called emission delay.

Yu et al. (2020) proposed FastEmit, which changes the gradient rather than the alignment. Recall that the occupancy of a node splits into two parts, the probability of leaving by the blank and the probability of leaving by the label, $o_{t,u} = \gamma_{t,u}(\varnothing) + \gamma_{t,u}(y_{u+1})$. FastEmit multiplies the label part by $1 + \lambda_{\text{FE}}$, with a small weight $\lambda_{\text{FE}}\ge0$.

In plain words, at every node where the alignment might emit the next label now or wait, FastEmit's training target leans toward emitting now. The derivation gives the exact target. It also shows that no reweighting of alignments could do this, and that the update is not the gradient of any loss.

The next cell is an optional check of the second remark. It computes the Jacobian of the update with respect to the logits by central differences, on a lattice with $T = 3$, $U = 2$, and $V = 2$, and measures how far it is from symmetric. At $\lambda_{\text{FE}} = 0$ the update is the true gradient, so its Jacobian is the Hessian of the loss.

At $\lambda_{\text{FE}} = 0$ the asymmetry is about $10^{-10}$, rounding error. At $\lambda_{\text{FE}} = 0.3$ it is 0.022, about a tenth of the largest entry, 0.21. No scalar loss has this update as its gradient.

The next cell shows the mechanism on a toy. The joint is a free table of label logits, one per node, and label $u+1$ becomes audible at a known onset frame. Before the onset its logit is held low, at $-4$, the acoustics saying "not yet". The starting model emits late, 1.25 frames after each onset on average. We fine-tune the table for 100 gradient steps with and without FastEmit and report when each label is emitted. In this toy the late start is put in by hand. In a real model it comes from the incentive to hear more audio, which the toy does not have. So read the cell for the mechanism and the papers for the size of the effect.

Plain training barely moves the delay, from 1.25 to 1.21 frames, because the loss is indifferent to timing. With $\lambda_{\text{FE}} = 0.3$ the mean delay falls to 0.09 frames. Emissions sit on the onsets, and the occupancy band moves up and to the left. With $\lambda_{\text{FE}} = 1$ the delay is $-0.25$, so some labels are now emitted before they are audible, which a real model can do only by guessing. That is the trade-off FastEmit's weight controls.

The toy needs large weights because it runs only 100 steps. Real models use $\lambda_{\text{FE}}$ around 0.004 to 0.01 over a whole training run, and the token-and-duration models below use 0.01. The FastEmit slider of the lattice widget earlier in the section runs the same fine-tuning for 60 steps at every weight, so at zero it shows plain training. At $T = 12$ and $U = 4$ its mean delay goes from 1.22 frames at zero to 0.09 at 0.5 and $-0.06$ at 1.

Token-and-duration transducers and other variants

A transducer advances one frame per blank. So a 30-second clip at 80 ms frames costs at least 375 joint calls, even when most of it is silence or the middle of a long word. Xu et al. (2023) let the model also say how many frames each symbol covers, which lets the decoder skip frames.

In plain words, each step now says what to emit and how far to jump. Take $\mathcal D = \{0,1\}$ first. A label of duration 0 and a blank of duration 1 are exactly the two moves of the ordinary transducer. What the token-and-duration transducer (TDT) adds are label moves that also advance time, and longer jumps of both kinds. The terminal node $(T+1,U)$ plays the role of the final blank, but a path may now also reach it with a label. The gradient follows from forward and backward variables exactly as before, now with sums over durations.

At inference the duration is what saves time. Greedy TDT decoding takes the most likely symbol and the most likely duration at each step, and advances $t$ by that duration, for labels and blanks alike. A blank whose predicted duration is 0 must still advance one frame, since such blanks never occur in training. Silence and long sounds are crossed in a few steps instead of one per frame.

import numpy as np

def tdt_loss(log_tok, log_dur, y, durations, blank=0):
    raise NotImplementedError

Two other variants complete the 2026 toolbox. Multi-blank transducers (Xu et al. 2022), the precursor of TDT, add "big blanks" that each skip a fixed number of frames. And the hybrid autoregressive transducer described earlier separates the blank from the labels, so that the internal language model can be measured and subtracted. §36.5 uses that for fusion.

CTC against the transducer, at real scale

Here is the whole comparison in one table. Attention encoder-decoders join it in §36.4.

The 2026 picture needs one more measure from §36.1. RTFx, the inverse real-time factor, is seconds of audio transcribed per second of compute. An RTFx of 1,000 transcribes an hour of audio in 3.6 seconds.

The Open ASR Leaderboard paper (Srivastav et al. 2025) reports numbers as of 27 March 2026. WER is averaged over its English short-form test sets, and speed is measured on one A100-80GB GPU at batch 64 where memory allowed. NVIDIA's Parakeet-TDT-0.6B-v2 averaged 6.05 percent WER at an RTFx of 3,390. It ranked tenth overall and was by far the fastest of the top ten. The other open entries of the top ten ran at RTFx 145 to 525, with LLM or Transformer decoders. The best CTC model, Parakeet-CTC-1.1B, averaged 7.40 at RTFx 2,730 and ranked 34th. These two differ in size and training data, so the comparison is not controlled. The paper's general finding is the pattern to remember. Conformer encoders with attention or LLM decoders give the lowest WER, and CTC and TDT decoders give by far the highest throughput.

Duration prediction is only part of the transducer's speed. Greedy decoding is sequential within an utterance. Consider a batched decoder that loops over frames, with an inner loop over each frame's labels as in Exercise 36.3d. At each frame, every utterance in the batch waits for the one that emits the most labels there. The reference loop of that exercise is correct, not fast.

Read more

• Graves (2012). Sequence Transduction with Recurrent Neural Networks, and Graves, Mohamed & Hinton (2013), Speech Recognition with Deep Recurrent Neural Networks. The transducer, its lattice, the forward-backward algorithm with the anti-diagonal identity, the gradient, and beam search, in eight pages, then the feedforward joint network used ever since.
• He et al. (2018). Streaming End-to-end Speech Recognition For Mobile Devices, and Sainath et al. (2020), A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency. How transducers reached phones, with every engineering choice and its measured effect.
• Li et al. (2019). Improving RNN Transducer Modeling for End-to-End Speech Recognition, and Kuang et al. (2022), Pruned RNN-T for fast, memory-efficient ASR training. Padding removal and function merging, then pruning, the memory fixes everyone uses.
• Yu et al. (2020). FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization. Why transducers emit late and the one-line gradient change that fixes it, with the latency and WER sweep.
• Xu et al. (2023). Efficient Sequence Transduction by Jointly Predicting Tokens and Durations. The TDT lattice, its gradients, and the speed-ups behind Parakeet-TDT.
• Variani et al. (2020). Hybrid Autoregressive Transducer (hat). Separating blank from labels to expose the internal language model.
• Kang et al. (2022). Fast and parallel decoding for transducer, and Bataev et al. (2024), Label-Looping: Highly Efficient Decoding for Transducers. One-label-per-frame beam search and the constrained transducer, then the batched greedy decoder behind today's throughput numbers.
• Zhang et al. (2020). Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss. Attention encoders, and how few previous labels the prediction network needs.
• Ghodsi et al. (2020). RNN-Transducer with Stateless Prediction Network. Removing the recurrence from the prediction network, and the evidence that it helps alignment more than it models language.
• Li et al. (2020). On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition. Transducers against attention encoder-decoders on 65,000 hours, streaming and offline.
• Srivastav et al. (2025). Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation. How the leaderboard measures WER and RTFx, and where transducer, CTC, and LLM-decoder systems land.

Watch next

▶

▶