Chapter 37 · Section 37.5 · 100 min

Speech-to-speech and full duplex

This section is about voice assistants, the programs you talk to and that talk back, and about the newest of them, which listen and speak at the same time. A voice assistant has to hear what you say, decide that you have finished, work out an answer, and say it aloud. Each of those steps takes time, and people notice a slow reply at once. People also expect more than tidy turns. In real conversation a listener says "mm-hmm" while the other person talks, finishes their sentences, and cuts in when they have heard enough.

Two measurements set the bar. The first is speed. In recordings of natural conversation in ten languages, the most common gap between a yes-or-no question and its answer is under 200 ms, and every language's average gap is under half a second (Stivers et al. 2009, with the details in §36.6). The second is overlap. In recorded meetings, a speaker is talking over someone else during about 12 percent of all speaking time, and during about 16 percent in one large annotated corpus (Çetin and Shriberg 2006). The authors of the Moshi model cite this study as "10 to 20%" to motivate their design (Défossez et al. 2024). A system that strictly takes turns, where the user speaks and then the machine speaks, can never overlap at all.

The earlier sections of this chapter built the parts. A neural audio codec turns sound into a short grid of integers and back (§37.3). The codec used in this section, called Mimi, cuts audio sampled 24,000 times a second into frames of 80 ms, 12.5 frames per second, and describes each frame with 8 integers called codes. The first code of a frame mostly records what is said, and the other seven record how it sounds. A codec language model predicts such codes one frame at a time, the way a text model predicts the next word (§37.4). This section turns those parts into a conversation partner.

Here is the plan. We first add up the time of one voice turn and prove which parts streaming can hide and which it cannot. We look at how a system decides that you have finished, and at what a chain of separate models loses besides time. Then come four ways to put speech and text inside one model. The center of the section is Moshi, a model that listens and speaks at once. We derive its layout of 17 token streams, its latency floor of 160 ms, its running text of what it is saying, and its serving cost. We end with exact definitions of turn-taking events, with how full-duplex behavior is measured, and with what a production voice agent needs so that it can be interrupted safely. Interviews for voice products ask about every one of these.

The cascade, and three changes that shorten the wait

The classic design is a cascade, a chain of separate models in which each hands its output to the next. A voice activity detector (VAD) marks which short frames of audio contain speech. An endpointer watches the VAD and declares that the user's turn is over, usually after a fixed stretch of silence. A speech recognizer (Chapter 36) turns the audio into text, a text language model writes a reply, and a text-to-speech system (§37.1) reads the reply aloud.

In the plainest cascade, each model waits until the one before it has finished. Three changes shorten the wait. Streaming lets the stages overlap, so the synthesizer starts on the reply's first clause while the language model is still writing the rest. An end-to-end model removes the hand-offs, with one network that hears audio and produces audio. A full-duplex model drops the turns altogether. It listens and speaks at the same moment, like a person on the phone.

📐 Half duplex and full duplex

A spoken dialogue system is half duplex if at every moment it is in exactly one of two modes. It is either listening (it takes in the user's audio and emits none) or speaking (it emits audio and treats the user's audio, at most, as a signal to stop). A rule outside the generative model decides where turns begin and end, usually an endpointer.

A system is full duplex if it takes in the user's audio and produces its own output audio at every time step. Listening and speaking can then overlap, and the model itself decides at each step whether to speak, stay silent, give a short acknowledgment, or stop.

A cascade with streaming stages is still half duplex. A model that is fed the user's audio at every step and generates its own audio at every step, possibly silence, is full duplex.

In plain words, half duplex is a walkie-talkie whose talk button some rule presses for you, and full duplex is a telephone. What changes in practice is who decides when to speak. In a half-duplex system a hand-written rule decides, and in a full-duplex system the model learns it. An interviewer may ask whether a fast streaming cascade counts as full duplex. It does not, however fast it is, because a rule still owns the turns.

Where the time goes in a voice turn

To compare designs we need one clock. It starts at the instant the user's speech really ends, the end of the last word as a forced aligner would place it. A forced aligner is a tool that finds where each word of a known transcript sits in the audio (§36.1). The clock stops at the first sample of the reply that reaches the user's ear.

📐 Latency of a voice turn

The latency of a voice turn $t_{\text{turn}}$ is the time from the true end of the user's speech to the first audio sample of the reply at the user's device. For a cascade it is built from five components, all in milliseconds.

The endpointing wait $t_{\text{end}}$ , the time the endpointer needs before it declares the turn over. For an endpointer that waits for silence, it is the silence threshold itself (§36.6).
The final recognition time $t_{\text{ASR}}$ , from the endpoint to the final transcript.
The language model's time to first token $t_{\text{LLM}}$ . It is mostly the prefill, the forward pass over the whole conversation so far that must finish before the first new token can be written (§20.1). We also count the time to decode the tokens the next stage needs before it can start.
The synthesizer's first-packet latency $t_{\text{TTS}}$ , from receiving its first text to emitting its first chunk of audio. For a codec language model it is the frames it must generate before the codec decoder can emit sound, plus their compute (§37.4).
The network time $t_{\text{net}}$ , the extra delay of sending audio up to the server and back, plus any playback buffer.

A stage streams if it can start on the first chunk of its input. A stage that does not stream waits for its whole input.

Here is a budget with numbers. They are assumptions, chosen to be typical, and they are also the defaults of the widget below.

✏️ A cascade budget

The endpointer waits for 500 ms of silence. Final recognition takes 150 ms. The language model prefills in 100 ms and then decodes at 25 ms per token, and its answer is 60 tokens long. The synthesizer needs one clause of 10 tokens before it can start, and then emits its first packet in 150 ms. The network adds 100 ms.

If synthesis waits for the whole answer, the language model's part is $100 + 60\times 25 = 1{,}600$ ms, and

t_{\text{turn}} = 500 + 150 + 1{,}600 + 150 + 100 = 2{,}500 \text{ ms}.

If synthesis streams from the first clause, the language model's part shrinks to $100 + 10\times 25 = 350$ ms, and $t_{\text{turn}} = 500 + 150 + 350 + 150 + 100 = 1{,}250$ ms. Of those 1,250 ms, the 600 ms of endpointing and network stay the same whatever hardware you buy. The other 650 ms are compute, which a faster stack could shrink.

The cell keeps both budgets explicit, so you can change one assumption and see what moves.

Python · cell 1

Streaming saved 1,250 ms, and all of it was generation time. It did not touch the endpointing or the network. That is not a feature of these particular numbers. It holds for every pipeline, as the next result shows.

∴ What streaming can hide

Let a pipeline have stages $1, \dots, m$ . Stage $i$ begins at time $b_i$ , with $b_1 = 0$ at the true end of speech. It emits its first output chunk at $b_i + a_i$ and its last at $b_i + e_i$ , where $0 \le a_i \le e_i$ . The reply is heard when the last stage emits its first chunk, at $t_{\text{turn}} = b_m + a_m$ .

If every stage waits for its whole input, $t_{\text{turn}} = \sum_{i=1}^{m-1} e_i + a_m$ .
If every stage $i \ge 2$ streams, $t_{\text{turn}} = \sum_{i=1}^{m} a_i$ , which is never larger.
If stage 1 is an endpointer, its output is a single decision, so $a_1 = e_1 = t_{\text{end}}$ , and no stage can begin before it. If the last stage is the network, with $a_m = t_{\text{net}}$ , then every turn-based design has $t_{\text{turn}} \ge t_{\text{end}} + t_{\text{net}}$ , whatever its compute.

∂ Proof that streaming hides only generation time

A stage that waits for its whole input begins when its predecessor emits its last chunk, $b_i = b_{i-1} + e_{i-1}$ . A streaming stage begins at its predecessor's first chunk, $b_i = b_{i-1} + a_{i-1}$ . Unrolling the first recursion from $b_1 = 0$ gives $b_m = \sum_{i<m} e_i$ , hence item 1. Unrolling the second gives $b_m = \sum_{i<m} a_i$ , hence item 2. Since $a_i \le e_i$ term by term, the streaming sum is at most the serial one.

For item 3, every $b_i$ is $b_{i-1}$ plus a nonnegative term, so $b_m \ge b_2 \ge e_1 = t_{\text{end}}$ , and $t_{\text{turn}} = b_m + a_m \ge t_{\text{end}} + t_{\text{net}}$ . ∎

In plain words, streaming replaces "wait for everything" with "wait for the first piece" at every hand-off. It cannot overlap anything with the decision that the user has finished, because no later stage knows what to do until that decision exists. A faster GPU shrinks $t_{\text{ASR}}$ , $t_{\text{LLM}}$ , and $t_{\text{TTS}}$ . It does nothing for $t_{\text{end}}$ or $t_{\text{net}}$ .

One escape remains, speculation. The system starts the language model on the partial transcript before the endpoint fires, and throws the work away if the user keeps talking. It is the same draft-then-check idea that speculative decoding uses inside a model (§20.5). Recognition and the first tokens can then hide behind the endpointing wait, at the price of wasted compute. The wait itself, and the network, remain. So when an interviewer asks how to make a voice assistant feel faster, the strongest answer starts with the endpointer, not the GPU.

The widget's last two designs preview the rest of the section. An end-to-end model that still waits for an endpoint keeps $t_{\text{end}}$ , but it removes the recognizer and the hand-off to a separate synthesizer. A full-duplex model has no endpoint at all. Its floor is set by its codec frame, which we derive for Moshi below.

Warm-up

Time to first sound

Write first_sound_ms(stages, streaming). It returns $t_{\text{turn}}$ , the time in milliseconds at which the last stage of a pipeline emits its first chunk. Each stage is a dict with five keys. The keys startup and chunk are times in milliseconds, n_chunks is how many chunks the stage emits, streams_in says whether the stage can start on the first chunk of its input, and name is a label. A stage that begins at time $b$ emits its $k$ -th chunk at time $b +$ startup $+ k\cdot$ chunk, for $k = 1, \dots,$ n_chunks.

The first stage begins at 0, the true end of speech. Every later stage begins when the stage before it emits its last chunk. The one exception is a stage with streams_in true when streaming is true. That stage begins when the stage before it emits its first chunk.

The tests use the numbers of the worked example, with the reply's audio split into thirty chunks of 50 ms. The pipeline has an endpointer (startup 500, one chunk of 0 ms), final recognition (150, one chunk of 0), the language model emitting six clauses (startup 100, chunk 250), the synthesizer (100, thirty chunks of 50), and the network (100, one chunk of 0).

def first_sound_ms(stages, streaming):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Real budgets, read from the papers

Published systems report budgets like ours. Each one starts its clock in a different place, though, and that is the most common trap when comparing them.

✏️ Two measured budgets

Freeze-Omni (Wang et al. 2024) is a speech-to-speech model built around a 7B text model whose weights are never changed. The authors start the clock when their turn detector fires and stop it at the first audio chunk. On average the language model takes 478 ms to produce its first chunk of text tokens. Passing that chunk into the speech decoder takes 15 ms, the decoder takes 237 ms to produce its first chunk of speech tokens, and the codec takes 11 ms to decode the first audio. The paper reports a total of 745 ms on average, a little more than the 741 ms the four averages add to, and 1,020 ms at the 90th percentile, the time that 9 turns in 10 beat. The turn detector itself fires 160 to 320 ms after the real end of speech, and the network adds 200 to 300 ms. So the authors estimate about 1.2 s in real use.

Qwen3-Omni (Xu et al. 2025) reports a "theoretical" first-packet latency of 234 ms for a single request. It is the sum of 72 ms to preprocess and encode the last chunk of input, 88 ms to the text model's first token, 57 ms to the speech model's first token, 14 ms for a small module that predicts the remaining codes of the first frame, and 3 ms for the codec decoder. With six requests served at once, the same sum is 1,172 ms. The 234 ms leaves out both the decision that the user has finished and the network.

Freeze-Omni's 1.2 s in real use is five times Qwen3-Omni's 234 ms, and part of that gap is only where each clock starts. The same caution applies to the headline numbers below, the fastest that people and published systems reach.

Deciding that the user has finished

The theorem says the endpointing wait is the floor of every turn-based design, so it pays to look at it closely. A silence endpointer declares the turn over after a fixed silence of $\theta$ ms. Speech is full of pauses, though. People breathe, think, and search for words, and a pause inside a turn can be longer than the gap at the end of one (Ekstedt and Skantze 2022 build their turn-taking model on this fact). So $\theta$ trades two errors. If it is short, the system cuts people off mid-sentence. If it is long, every answer is late by $\theta$ .

The cell makes the trade-off concrete on synthetic turns. Each turn has a random number of pauses inside it, Poisson with a mean of 2, and each pause has a log-normal length with a median of 350 ms. These are plausible assumptions, not measurements. A silence endpointer cuts the user off whenever a pause inside the turn reaches $\theta$ . The cell then tries a content-aware endpointer. At every silence it asks a classifier whether the words so far form a complete turn. It waits 200 ms if the answer is yes and 1,200 ms if it is no. The classifier is assumed right 90 percent of the time at real ends and 97 percent of the time at pauses.

Python · cell 2

Under these assumptions the silence endpointer must wait 950 ms to cut off fewer than 7 percent of turns. The content-aware one cuts off about as many, 7.2 percent, with a mean delay of 296 ms. The exact numbers depend on the assumptions, but the shape does not. Silence alone cannot tell a pause from an ending, so its two errors move in opposite directions as $\theta$ moves. Only information about what has been said, or how it was said (a falling pitch, a drawn-out last syllable), can reduce both at once.

Three lines of work use that information.

An end-of-query token. Sainath et al. (2020) trained the recognizer itself to emit a token that marks the end of the user's request, as §36.6 describes.
Turn ends predicted from words. TurnGPT (Ekstedt and Skantze 2020) is a Transformer language model over the dialogue's words. After each token it predicts the probability that the current turn ends there.
Future voice activity predicted from audio. Voice Activity Projection (VAP, Ekstedt and Skantze 2022) reads the conversation's audio and each speaker's voice activity so far, and predicts who will be speaking during the next 2 seconds.

VAP's training labels come for free, from the voice activity of recorded two-channel dialogue. Freeze-Omni takes a cheaper route. It adds a classification head to its language model that predicts, at the last frame of each audio chunk, whether to keep listening, to stop and answer, or to stop without answering. In practice a content-aware endpointer is the cheapest large latency win a cascade can get, and interviewers expect you to name it.

What a cascade loses besides time

A cascade passes text from the recognizer to the language model, and from the language model to the synthesizer. Text is a narrow channel. Moshi's authors list three things a cascade loses, and each is worth seeing concretely.

The first is everything that is not words. "Fine." said flatly and "Fine!" said brightly have the same transcript. Emotion, sarcasm, hesitation, laughter, a sigh, the identity of the speaker, and background sounds such as a baby crying all vanish at the recognizer. The synthesizer then has to guess the prosody, the rhythm, pitch, and loudness of speech, from text alone.

The second is overlap. A pipeline that waits for the user to stop cannot say "mm-hmm" while they talk, cannot finish their sentence, and handles being interrupted only through an extra rule. The authors of dGSLM, a two-channel dialogue model we meet below (Nguyen et al. 2023), measured this. In their dialogue statistics, which Moshi's paper reprints, a cascaded system produced exactly 0.0 s of overlap and 0.0 s of pauses within one speaker's speech per minute of dialogue. Human dialogues in the same measurement had 3.6 s of overlap and 5.5 s of such pauses per minute. The third loss is latency, which we measured above.

What a cascade keeps is just as real. Each part is a mature component, trained and tested on its own. The language model is a full text model with all its reasoning, its tools, its safety training, and its system prompt. The transcript is an audit log, which a bank or a hospital may be required to keep. And you can swap the voice, the language, or the model independently. Every end-to-end design below gives up some of this to win back the three losses, and weighing that trade is what an interviewer will ask you to do.

Putting speech and text in one model

An end-to-end model needs two things a text model lacks. On the input side it must read audio. It can take continuous features from an audio encoder, fed into the language model through a small adapter network (§35.6, §31.1), the way speech recognizers built on language models do (§36.4). Or it can read discrete speech tokens. On the output side it must write speech tokens that a decoder turns into a waveform. These are either codec codes (§37.3) or speech units. A speech unit is a cluster id, found by running k-means on the features of a self-supervised speech model such as HuBERT (§35.5), so each unit stands for one kind of short sound.

The design question is how text tokens and speech tokens share one autoregressive model. There are four answers, and each buys something different.

Speech units in the vocabulary, with a chain of modality. SpeechGPT (Zhang et al. 2023) adds HuBERT units to the vocabulary of a 13B LLaMA text model. Its last training stage teaches a chain of modality. Given the units of a spoken question, the model writes the transcript, then a text answer, then the units of the spoken answer. The text steps let the model reason in the modality it was pretrained on. The cost is that every step is serial. The whole text answer exists before the first unit of speech, so the latency grows with the length of the answer.

Interleaving at word boundaries. Spirit LM (Nguyen et al. 2025) continues pretraining a 7B Llama 2 text model on text, on speech, and on aligned speech and text in which the modality switches at random word boundaries. A training sequence looks like [TEXT]the cat [SPEECH][Hu3][Hu7]…[TEXT]the mat, where [Hu3] is HuBERT unit 3. A model that has seen "the cat" continue as speech learns that the two modalities say the same thing. It then transfers skills between them, for example few-shot recognition and synthesis.

Parallel heads. Here text and speech are produced in the same step. Mini-Omni (Xie and Wu 2024) has eight output heads, one for text and seven for the seven codes its codec writes per frame. Each of these seven layers runs one step behind the one above it, with the text first, so the text of each step guides the audio that follows it. LLaMA-Omni (Fang et al. 2025) puts a small speech decoder on top of the language model. The decoder reads the model's hidden states as the text is written and emits speech units in chunks, so speech for a text prefix can play as soon as the prefix exists. It reaches 236 ms from the end of the spoken instruction to the start of the spoken answer.

Thinker and Talker. Qwen2.5-Omni (Xu et al. 2025) splits the model in two. The Thinker is a language model with audio and vision encoders, and it writes text. The Talker is a second autoregressive model that writes speech tokens as a stream. It reads the Thinker's hidden states together with the embeddings of the text tokens the Thinker sampled. Both inputs matter. The hidden states carry tone and intent before the words are written. Words that sound different can have similar hidden states, and the sampled tokens tell them apart. In practice this design keeps a strong text model in charge of content and lets the speech side run alongside it.

🔬 The systems in more detail

Mini-Omni. Its codec is SNAC (Siuzdak et al. 2024), a residual codec whose quantizers run at different frame rates. In the speech version the second and third quantizers run two and four times faster than the first, so one frame of the first holds $1 + 2 + 4 = 7$ codes, which Mini-Omni treats as seven layers. To protect reasoning, Mini-Omni decodes a batch of two, one sample that writes text only and one that speaks, and feeds the text-only sample's tokens to the speaking one.

LLaMA-Omni. Llama-3.1-8B-Instruct sits in the middle, fine-tuned to answer spoken instructions. A Whisper encoder and an adapter that downsamples 5 times feed it speech. The speech decoder has two Transformer layers and 425 million parameters, and it is non-autoregressive. It upsamples each hidden state of the language model 25 times. It is trained with the CTC loss (§36.2), which sums over all alignments, to emit the deduplicated sequence of HuBERT units that a unit vocoder turns into sound. The decoder is causal, so units are voiced in chunks of $n_{\text{chunk}}$ units (the paper's $\Omega$ ), and the 236 ms is measured with $n_{\text{chunk}} = 10$ .

Spirit LM's units. It uses 501 deduplicated HuBERT units. An expressive version adds pitch and style tokens.

GLM-4-Voice (Zeng et al. 2024) scales word-level interleaving. Its tokenizer has one codebook at 12.5 Hz and 175 bits per second, made by putting a vector-quantized bottleneck inside a speech recognizer's encoder. It synthesizes interleaved speech and text from ordinary text corpora with a text-to-token model, and continues pretraining a 9B text model on up to 1 trillion tokens.

Qwen3-Omni (Xu et al. 2025) makes both halves mixture-of-experts models (§13.1). The Thinker has 30 billion parameters, of which 3 billion are active per token, and the Talker has 3 billion with 0.3 billion active. Its Talker no longer reads the Thinker's hidden states for text. It reads the text itself plus the audio and visual features, which lets outside modules (retrieval, function calls, safety filters) change the text before it is spoken. The Talker predicts the first code of each 12.5 Hz frame, a small module predicts the frame's remaining codes, and a causal convolutional decoder emits audio from the first frame on.

Qwen3.5-Omni (Qwen Team 2026) scales the pattern to hundreds of billions of parameters, with mixture-of-experts layers and a hybrid attention design in both halves, a 256k context, and more than 100 million hours of audio-visual data. It adds a method that aligns text and speech units as they stream, for more stable streaming speech.

Freeze-Omni never updates its text model. It trains a streaming speech encoder and a speech decoder around a frozen Qwen2-7B-Instruct, with speech recognition and synthesis data and 60,000 multi-turn text dialogues.

Kimi-Audio (KimiTeam 2025) reads both continuous acoustic features and discrete semantic tokens at 12.5 Hz, writes discrete semantic tokens alongside text, and turns them into audio chunk by chunk with a streaming decoder trained by flow matching, which learns a velocity field that moves noise to data and generates by integrating it (§33.6). It was pretrained on more than 13 million hours of audio.

How much of the text model's knowledge survives the trip through speech? Spoken question answering measures it directly. The questions are synthesized as speech, and an answer counts if it is correct. The scores below are percentages of questions answered correctly.

model	how speech is attached	Web Questions	LLaMA Questions	Audio TriviaQA
SpeechGPT (13B)	units in the vocabulary	6.5	21.6	14.8
Moshi without its text stream	full duplex, audio only	9.2	21.0	7.3
Moshi (7B)	full duplex with a text stream	26.6	62.3	22.8
Helium, Moshi's backbone, given text	text	32.3	75.0	56.4
Freeze-Omni (7B)	frozen text model	44.73	72	53.88
Qwen2-7B-Instruct, Freeze-Omni's backbone, given text	text	45.13	77.67	63.93

The first four rows come from Moshi's Table 8 and the last two from Freeze-Omni's Table 3. Both tables label SpeechGPT as 7B, but its paper builds it on LLaMA-13B. The two papers synthesized their questions with different voices and scored differently (Freeze-Omni scores its output text), so compare rows within one paper. The pattern is still clear. Moshi, trained end to end, keeps most of its backbone's skill on the two easier sets and less than half of it on TriviaQA. Freeze-Omni, whose text model never changes, stays within six points of its backbone on two of the three sets. That gap is the cost in knowledge of training end to end, and avoiding it is why Freeze-Omni freezes its text model.

Moshi, a model that listens while it speaks

Moshi (Défossez et al. 2024), which its authors describe as the first real-time full-duplex spoken large language model, puts every idea of this section in one place. It works on streams, rows of tokens that each advance by one token per 80 ms frame, such as the user's first codec level or Moshi's text. It has three parts, each a kind of model you have met before.

Mimi, the causal codec of §37.3, turns 24 kHz audio into 12.5 frames per second at 1.1 kbps. Each frame has $Q = 8$ codes, one per level, each drawn from a codebook of 2,048 entries. Its first level is trained to match a self-supervised speech model (WavLM), so it carries mostly what is said. This is the semantic level. Levels 2 to 8 carry how it sounds, the acoustic levels. One frame lasts $\Delta_{\text{fr}} = 1/f_r = 80$ ms, where $f_r = 12.5$ Hz is the frame rate.
Helium, a 7B text language model pretrained on 2.1 trillion tokens of public English text, becomes the temporal Transformer of an RQ-Transformer. An RQ-Transformer (§37.4) is a pair of models, a large one that steps through time and a small one that fills in the codes of each step. The temporal Transformer takes one step per frame. At step $s$ it reads one input vector, the sum of one learned embedding per stream for the tokens of step $s-1$ . It outputs a context vector $\mathbf{z}_s \in \R^{d}$ with $d = 4096$ .
A small depth Transformer (6 layers, width 1,024, 16 heads, with separate weights for each stream it predicts) reads $\mathbf{z}_s$ and predicts the step's audio codes one at a time. Each code is conditioned on those already predicted in the step. A linear layer on $\mathbf{z}_s$ predicts the step's text token.

The new idea is what goes into the streams. Moshi models two audio streams side by side, its own and the user's.

The multistream layout

Picture each 80 ms of the conversation as one column of tokens. The top token is the text Moshi is saying in that frame. Below it come Moshi's 8 codes for the frame, then the user's 8 codes. That is 17 tokens per column, and the temporal Transformer takes one column per step. Moshi writes the top 9 tokens of each column, and the microphone, through Mimi's encoder, writes the bottom 8. One twist makes it work well. The acoustic codes are shifted one frame late, so each column holds the semantic code of the current frame and the acoustic codes of the previous one.

The cell prints the layout for a toy case with three levels per speaker, five frames, and a shift of one frame. You can see which frame each stream refers to at each step.

Python · cell 3

Read one column. At step 2 the model writes its text token $W_2$ and the codes $A_{2,1}$ , $A_{1,2}$ , and $A_{1,3}$ , where $A_{t,q}$ is Moshi's code at frame $t$ and level $q$ . It finishes frame 1 of its own audio while it starts frame 2. The layout needs $T + 1$ steps for $T$ frames, because the last frame's acoustic codes arrive one step after its semantic code. Here is the general statement.

📐 Moshi's multistream layout

Number the codec frames $t = 0, 1, \dots, T-1$ . Let $A_{t,q}$ be Moshi's code at frame $t$ and level $q \in \{1, \dots, Q\}$ , with level 1 semantic, and let $A'_{t,q}$ be the user's codes. Let $W_t$ be Moshi's text token aligned to frame $t$ (the inner monologue, defined below). Fix an acoustic delay $\tau_{\text{ac}} \in \{0, 1, 2, \dots\}$ frames. The model's sequence has $K_{\text{str}} = 2Q + 1$ streams and $S = T + \tau_{\text{ac}}$ steps, with entries $V_{s,k}$ for step $s$ and stream $k$ ,

V_{s,1} = W_s, \qquad V_{s,1+q} = \begin{cases} A_{s,1} & q = 1 \\ A_{s-\tau_{\text{ac}},\,q} & 2 \le q \le Q \end{cases}, \qquad V_{s,1+Q+q} = \begin{cases} A'_{s,1} & q = 1 \\ A'_{s-\tau_{\text{ac}},\,q} & 2 \le q \le Q \end{cases}.

Any entry whose frame index lies outside $0, \dots, T-1$ is a padding token. The model predicts each entry from all earlier steps and the earlier entries of its own step, $p_\theta(V_{s,k} \mid V_{0}, \dots, V_{s-1}, V_{s,1}, \dots, V_{s,k-1})$ . Training minimizes a weighted cross-entropy,

\mathcal{L} = \frac{1}{S}\sum_{s}\Big( \mathrm{CE}_{s,1} + \frac{\sum_{k\ge 2}\alpha_k\,\mathrm{CE}_{s,k}}{\sum_{k\ge 2}\alpha_k}\Big), \qquad \mathrm{CE}_{s,k} = -\log p_\theta(V_{s,k} \mid \cdot),

with $\alpha_k = 100$ for the two semantic streams and $\alpha_k = 1$ for the acoustic ones.

At inference the model samples its own streams, $k = 1, \dots, Q+1$ . The user's streams, $k = Q+2, \dots, 2Q+1$ , are filled with the codes Mimi's encoder produces from the microphone. There is no text stream for the user.

In plain words, each step is one 80 ms frame of the conversation, written as a column of 17 tokens. The top token is Moshi's text, then come Moshi's semantic code and its acoustic codes from $\tau_{\text{ac}}$ frames earlier, then the same for the user. Moshi writes its half of the column and the microphone writes the other half. The loss gives the text token of a step as much weight as all 16 audio codes together, and inside the audio a semantic code counts 100 times as much as an acoustic one. In the paper's ablation, raising that weight from 1 to 100 improved the fluency score of generated speech from 4.09 to 3.65 (the score is defined below, lower is better).

Training also predicts the user's codes, which lets the model generate whole simulated dialogues. At inference those predictions are ignored, and the released model's depth Transformer predicts only Moshi's 8 levels. (The paper numbers steps and frames from 1, so its condition $s \ge \tau + 1$ is our $s - \tau_{\text{ac}} \ge 0$ . Its $K$ is our $K_{\text{str}}$ , renamed because $K$ is a codebook size in §37.3.)

Why delay the acoustic levels at all? In an RQ-Transformer, the order of the codes decides which codes can be predicted from which (§37.4). With $\tau_{\text{ac}} = 0$ , the small depth Transformer must predict a frame's semantic code and all its acoustic codes from the same context vector. The large temporal Transformer never sees the semantic code of the frame whose sound it is shaping. With $\tau_{\text{ac}} = 1$ , the acoustic codes of frame $t$ are predicted at step $t+1$ , after the temporal Transformer has read the semantic code of frame $t$ . The large model then carries the link between what is said and how it sounds, which the paper reports "greatly improves" generation.

The widget draws the layout as columns of 80 ms steps, with Moshi's text, semantic, and acoustic rows above the user's semantic and acoustic rows. Levels 2 to 8 share one row per speaker, since they share one delay, and the numbers in the chosen column say which frame each row holds there. Choose the interruption scenario, set the acoustic delay to 2, and predict two things before you look. At the step where the user starts talking, which of Moshi's frames is the model still finishing? And how many steps pass before Moshi's text and semantic streams can react by falling silent?

Interview

Moshi's streams

Write build_streams(text, moshi_codes, user_codes, tau, pad=0). It returns the integer matrix $\mathbf{V}$ of the layout, with $S = T + \tau$ rows and $2Q + 1$ columns. The input text has shape (T,). The two code arrays have shape (T, Q), with column q - 1 holding level $q$ . Column 0 of $\mathbf{V}$ is the text, columns 1 to $Q$ are Moshi's levels 1 to $Q$ , and columns $Q+1$ to $2Q$ are the user's levels. Every entry whose frame falls outside $0, \dots, T-1$ is pad.

Then write split_streams(V, tau), which undoes the layout and returns (text, moshi_codes, user_codes) with their original shapes.

import numpy as np

def build_streams(text, moshi_codes, user_codes, tau, pad=0):
    raise NotImplementedError

def split_streams(V, tau):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The latency floor of a frame-based model

Where does Moshi's 160 ms come from? In plain words, the model needs one frame to hear the user, then $\tau_{\text{ac}}$ more frames before the audio of its reply is complete, then one step of compute. We can prove this from the layout, using three facts about timing.

A causal codec encoder can emit the codes of the user's frame $t$ only once that frame's audio has fully arrived, at time $(t+1)\Delta_{\text{fr}}$ . Frame $t$ covers the interval from $t\Delta_{\text{fr}}$ to $(t+1)\Delta_{\text{fr}}$ .
Step $s$ can run only once every input it reads exists.
The decoder can play Moshi's frame $t$ only once all $Q$ of its codes have been generated.

∴ The latency floor of a frame-based duplex model

Take the layout above with frame duration $\Delta_{\text{fr}}$ and acoustic delay $\tau_{\text{ac}}$ . Let $t_{\text{step}}$ be the time to compute one step (one temporal step, the depth steps, and decoding one frame), with $t_{\text{step}} \le \Delta_{\text{fr}}$ so the model keeps up. Suppose the user makes a sound at time $u$ inside frame $j$ . The first audio of Moshi's whose semantic code or text can depend on that sound starts playing at $(j + 1 + \tau_{\text{ac}})\Delta_{\text{fr}} + t_{\text{step}}$ , so the reaction time satisfies

\tau_{\text{ac}}\Delta_{\text{fr}} + t_{\text{step}} \;<\; \text{reaction time} \;\le\; (1 + \tau_{\text{ac}})\Delta_{\text{fr}} + t_{\text{step}} .

The worst case without compute, $(1 + \tau_{\text{ac}})\Delta_{\text{fr}}$ , is the model's theoretical latency.

∂ Proof of the latency floor

The user's sound first enters the layout in step $j$ , as $A'_{j,1}$ , and its acoustic codes enter later, at step $j + \tau_{\text{ac}}$ . The temporal Transformer reads steps $0, \dots, s-1$ to predict step $s$ . So the first of Moshi's predictions that can depend on step $j$ are those of step $j+1$ , namely $W_{j+1}$ and $A_{j+1,1}$ , the text and semantic code of Moshi's frame $j+1$ .

That frame is complete only when its acoustic codes are generated, at step $j + 1 + \tau_{\text{ac}}$ . Step $s$ reads $A'_{s-1,1}$ , which exists at time $s\Delta_{\text{fr}}$ . So step $j + 1 + \tau_{\text{ac}}$ starts at $(j + 1 + \tau_{\text{ac}})\Delta_{\text{fr}}$ , and its frame plays from $(j + 1 + \tau_{\text{ac}})\Delta_{\text{fr}} + t_{\text{step}}$ . Subtracting $u$ , which lies in $[j\Delta_{\text{fr}}, (j+1)\Delta_{\text{fr}})$ , gives the two bounds. ∎

One detail sits outside the theorem. The delayed acoustic codes of Moshi's frame $j$ are generated at step $j + \tau_{\text{ac}}$ , so they can already depend on the user's frame $j$ . The texture of the current sound can react a little sooner. What is said cannot.

Put Mimi's frame into the theorem and Moshi's published latency comes out, along with every latency in the paper's ablation of the delay.

✏️ Moshi's 160 ms, and the ablation behind it

Mimi's frame is $\Delta_{\text{fr}} = 80$ ms. With $\tau_{\text{ac}} = 1$ the floor is $(1 + 1)\times 80 = 160$ ms, Moshi's stated theoretical latency, "80ms for the frame size of Mimi + 80ms of acoustic delay" in the words of the Moshi repository. The same formula reproduces every number in the paper's ablation. No delay gives 80 ms and a delay of 2 gives 240 ms. The delay pattern of §37.4 shifts level $q$ by $q - 1$ steps, so the last of 8 levels is 7 steps late. It gives $(1+7)\times 80 = 640$ ms, which the authors call incompatible with real-time dialogue.

How much quality does the delay buy? The ablation transcribes the generated speech with Whisper (§36.4) and scores the transcripts with a small text model by their negative log-likelihood, where lower means more fluent text. Delays of 0, 1, and 2 gave 4.36, 4.12, and 4.09 (with the semantic weight still at 1), so one frame of delay bought most of the gain. Moshi was pretrained with a delay of 2 and trained with a delay of 1 in every later stage.

The practical latency of about 200 ms on an L4 GPU is this floor plus compute and audio input and output. The cost estimate later in this section shows that on that GPU, reading the weights once per frame alone takes about 49 ms, so a $t_{\text{step}}$ of a few tens of milliseconds is what to expect.

The formula also shows the main design trade-off of a frame-based model. A codec at 50 Hz with the same delay would have a floor of 40 ms. But the temporal Transformer would then have to finish a full forward pass every 20 ms instead of every 80 ms, four times as many steps per second with a quarter of the time for each. A lower frame rate buys the time to run a 7B model per frame, at the price of a coarser clock. That is the answer an interviewer wants when they ask why Moshi uses 12.5 Hz.

Inner monologue

A model that generates only audio tokens must decide what to say and how to say it in the same codes. Models trained on speech alone are weak at the first, as Moshi's audio-only ablation shows below. So Moshi adds a text stream for its own speech, placed in each step before the semantic code. Within a step the order is now text, then semantic, then acoustic. The paper calls this text stream an inner monologue.

The text must run on the 12.5 Hz clock of the audio, which raises a concrete question. A word lasts several frames but is only one or two tokens, so what goes in the other frames? The answer is two special tokens that never occur inside words. PAD fills frames with nothing new to write. EPAD (end of padding) marks the frame just before a word starts. Here is a small case worked by hand before the rules.

✏️ Aligning three words by hand

At 12.5 Hz, align "Hello" starting at 0.00 s (one token, 21), "I'm" at 0.30 s (three tokens, 31, 32, 33), and "Moshi" at 0.45 s (two tokens, 41, 42) into 12 frames. A word's start frame is the frame that contains its start time, $\lfloor \text{start} \times 12.5\rfloor$ .

"Hello" starts in frame 0. There is no frame before it for EPAD, so EPAD goes in frame 0 and the token 21 in frame 1. "I'm" starts in frame $\lfloor 0.30\times 12.5\rfloor = \lfloor 3.75\rfloor = 3$ , so EPAD goes in frame 2 and its tokens in frames 3, 4, and 5. "Moshi" starts in frame $\lfloor 5.625\rfloor = 5$ . But frame 5 holds the 33 of "I'm", so its tokens start at the next free frame, 6. Frame 5 keeps its text token, so "Moshi" gets no EPAD. The stream is

W = (\text{EPAD}, 21, \text{EPAD}, 31, 32, 33, 41, 42, \text{PAD}, \text{PAD}, \text{PAD}, \text{PAD}),

with 4 of 12 frames PAD. Rounding instead of flooring would have put "I'm" at frame 4.

📐 Inner monologue alignment

Let Moshi's speech have word-level timestamps (from the Whisper recognizer in the paper). Word $i$ starts at $o_i$ seconds, its onset, and has text tokens $w_{i,1}, \dots, w_{i,n_i}$ , with the words sorted by onset. Its start frame is $t_i = \lfloor o_i f_r \rfloor$ , the frame that contains $o_i$ . Initialize every $W_t$ to PAD, then process the words in order.

Write the tokens of word $i$ in consecutive frames starting at $t_i$ , so that frame $t_i + j - 1$ holds $w_{i,j}$ .
Put EPAD in frame $t_i - 1$ , just before the word, unless that frame holds a text token of an earlier word, in which case put nothing.
If $t_i = 0$ , put EPAD in frame 0 and start the word's tokens at frame 1.

Two conventions of ours complete the rules for cases the paper leaves open. If a word's first frame is still occupied by the previous word's tokens, its tokens start at the first free frame (and rule 2 then puts nothing). Anything that would land past the last frame, a token or an EPAD, is dropped.

In plain words, each word's tokens are written where the word starts, PAD fills the silence and the rest of each word's duration, and EPAD is a "here comes a word" signal one frame early. The paper writes the start index as the timestamp "divided by the framerate". It means divided by the frame duration of 80 ms, which is the same as multiplying by 12.5 Hz. It does not say how to round, so the floor is our choice. The authors give the reason for EPAD. It splits two decisions into two steps, the decision that a word is over and the choice of the next word.

How much of the stream is padding in real speech? A rough estimate lands close to the paper's measurement. Read English runs at about 2.6 words per second (the LJSpeech corpus, §37.1). At roughly 1.3 subword tokens per word, that is about 3.4 tokens per second, in line with Moshi's "3 to 4 text tokens per second". While Moshi speaks, each second then holds about $3.4 + 2.6 = 6$ text and EPAD tokens out of 12.5 frames, so about half the frames are PAD. If Moshi is silent a third of the time, the PAD share is about $1 - \tfrac23\times\tfrac{6}{12.5} \approx 0.68$ . The paper measures about 65 percent in English conversation. Because PAD dominates, Moshi's pretraining halves the weight of PAD tokens in the loss.

Core

Inner monologue alignment

Write align_text(words, n_frames, frame_rate=12.5, pad=0, epad=1), which follows the definition, including our two conventions. Each word is a pair (start_seconds, tokens), and the words are sorted by start time. Use $t_i = \lfloor o_i f_r\rfloor$ for the start frame. Return an integer array of length n_frames. The starter is a common wrong answer, so expect it to fail.

import numpy as np

def align_text(words, n_frames, frame_rate=12.5, pad=0, epad=1):
    W = np.full(n_frames, pad, dtype=np.int64)
    for start_s, toks in words:
        t = round(start_s * frame_rate)
        W[t - 1] = epad
        W[t:t + len(toks)] = toks

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Why does the inner monologue help so much? In the ablation, adding it lowered the transcript negative log-likelihood of generated speech from 3.65 to 2.77, and the average transcript length from 602 to 1,920 characters. It was the largest single improvement in the paper's generative ablations. (Weak speech models tend to collapse into silence, so length is a proxy for quality.) On spoken question answering, the table above shows Moshi at 9.2, 21.0, and 7.3 without it and at 26.6, 62.3, and 22.8 with it.

The authors describe the text as a scaffold. Text is the modality Helium was pretrained on, so a text token per frame lets the model choose words with its text knowledge and then voice them. Why is there no text stream for the user? The paper gives two reasons. Transcribing the user's speech in real time is hard, and an external recognizer would bring back the cascade the model was built to remove. The text stream also gives a control handle. Moshi's text stream is all PAD while it listens, and forcing an EPAD makes it start talking at once.

One spoken answer, frame by frame

The latency floor counts frames until Moshi can react. The inner monologue adds one more detail, because a word's sound starts one frame after its EPAD. Here is a whole answer with real times, which is also how you would explain Moshi's latency at a whiteboard.

✏️ From the last sound of a question to the first sound of the answer

The user asks "What is the capital of France?", and the last sound of the question ends at 1.62 s. Moshi answers "Paris". Take $\Delta_{\text{fr}} = 80$ ms, $\tau_{\text{ac}} = 1$ , and an assumed compute time of $t_{\text{step}} = 40$ ms per step.

The last sound lies in frame $j = \lfloor 1.62 / 0.08 \rfloor = 20$ , which covers 1.60 to 1.68 s. Mimi's encoder emits that frame's codes at 1.68 s, when the frame ends.
Step 21 is the first step that reads frame 20, so it starts at 1.68 s. Suppose Moshi decides to answer here. It writes EPAD in its text stream and a semantic code for frame 21, which is still silence.
Step 22 starts at 1.76 s. Moshi writes the text token of "Paris" and the semantic code of frame 22, the first frame with the word's sound.
Step 23 starts at 1.84 s and writes the acoustic codes of frame 22. Frame 22 is now complete, and after 40 ms of compute it starts playing at 1.88 s.

The first sound of "Paris" leaves the model 260 ms after the user's last sound. That is 60 ms to finish the user's frame, 80 ms for the EPAD frame, 80 ms of acoustic delay, and 40 ms of compute. Nothing in the layout forces the EPAD frame to come after the question. If Moshi has already predicted the end of the question from the frames before, it can write EPAD at step 20, and "Paris" then plays from 1.80 s, 180 ms after the user stopped. No silence threshold appears anywhere in either count.

This is the practical meaning of "no endpoint". A cascade has to wait for silence before it may start. A full-duplex model can start answering at any frame, so it can answer the way people do, by predicting the end of a turn before it arrives (§36.6). Whether it predicts well is a separate question, and the benchmark later in this section shows that it often does not.

Delayed streams turn one model into a recognizer and a synthesizer

The text stream can also be shifted against the audio. If the text runs behind the audio, the words are decided after the sound is heard, which is streaming recognition. If the text runs ahead of the audio, the sound is decided after the words are known, which is streaming synthesis. Moshi trained both versions with a 2-second shift and no other change to the loss, the architecture, or the data.

Its streaming synthesizer reached a 4.7 percent word error rate (WER) on LibriSpeech test-clean, a standard set of read audiobook speech, scored by running a recognizer over the output. For comparison, VALL-E (Wang et al. 2023, §37.4) has 5.9 and NaturalSpeech 3 (Ju et al. 2024) has 1.81, both offline, meaning they see the whole text first. Its streaming recognizer reached 5.7 percent, against 3.6 for a streaming FastConformer recognizer with similar lookahead (Noroozi et al. 2024), and it gives word timestamps at the 80 ms precision of the frames.

This is delayed streams modeling, from §36.6. The alignment is fixed before training, every stream advances one step per frame in lockstep, and the delay is the latency, an explicit number. Zeghidour et al. (2025) develop it into Kyutai's streaming recognizer and synthesizer.

How Moshi was trained

The recipe has four stages after Helium's text pretraining, and each one supplies something the others cannot. First comes audio pretraining on 7 million hours of mostly English speech, treated as one stream. Then the same audio is split by speaker into two streams, which teaches the two-stream layout. Then 2,000 hours of real telephone conversations, with each side on its own channel, teach overlap and interruptions. Last, more than 20,000 hours of synthetic dialogues teach Moshi to be a helpful assistant with one consistent voice.

🔬 Moshi's training recipe in detail

Audio pretraining. 7 million hours of audio, mostly English speech, are transcribed by Whisper large-v3 and modeled as a single audio stream for all speakers. The text is shifted randomly by up to 0.6 s either way, and 30 percent of text tokens are masked. Half the batches are pure text from Helium's data, with a separate optimizer state, so the model does not forget its text skills. The acoustic delay is 2.
Simulated multistream. The same audio is diarized, split by who is speaking when (§36.6). One speaker is chosen as "Moshi", and the waveform is split into that speaker and everything else. The result is two streams with no overlap and perfect silence in the inactive one. From this stage on the acoustic delay is 1, and a tenth of the batches are still pure text.
Fisher. The Fisher corpus holds 2,000 hours of telephone conversations between strangers, each side recorded on its own channel and upsampled here from 8 kHz to 24 kHz. It gives real overlap, short acknowledgments, and interruptions.
Instruction tuning. More than 20,000 hours of synthetic dialogues are scripted by Helium, fine-tuned on transcripts of real conversation, and voiced by Kyutai's multistream synthesizer. Moshi always has one voice actor's voice, recorded in more than 70 speaking styles, and the user has random voices. The user's stream is augmented with random gain from $-24$ to $+15$ dB, noise, reverberation, and simulated echo. The echo is a copy of Moshi's own stream scaled by up to 0.2 and delayed by 100 to 500 ms, so the model learns not to mistake its own voice for the user's.

One result is worth knowing for interviews. Training on a single voice for Moshi was enough for it to keep that voice, with no control at inference.

The reference code below runs the whole training step of a two-stream model at toy size. Each step's input is the sum of one embedding per stream for the previous step. A GRU, a small recurrent network (§3.3), stands in for the temporal Transformer. Independent output heads stand in for the depth Transformer and predict the model's own streams of a step in parallel. In the synthetic data the model's speech copies the user's two frames later, so the loss can fall below a uniform guess only if the model reads the user's streams.

🔥 In PyTorchfor a machine with PyTorch

🔬 Before and around Moshi

Moshi was not the first attempt. dGSLM (Nguyen et al. 2023) was the textless ancestor. It has two Transformer towers with shared weights, one per channel, that exchange information through cross-attention, and it models HuBERT units (500 clusters) of the 2,000 hours of Fisher. It generated laughter and more natural turn-taking than a cascade. But it did not run online, had no text knowledge, and modeled only semantic units, so it could not produce rich acoustics.

SyncLLM (Veluri et al. 2024) teaches Llama-3-8B a sense of time. It interleaves chunks of 160 to 240 ms of both speakers' deduplicated HuBERT units with periodic synchronization tokens. Before writing its own next chunk, it predicts an estimate of the user's next chunk, which lets it tolerate network latency of up to 240 ms. It trained on 212,000 hours of synthetic dialogue and 2,000 hours of real dialogue.

The listening-while-speaking model of Ma et al. (2024) adds a streaming listening channel to a token-based synthesizer. It compares fusing the two channels early, in the middle, and late in the network for detecting interruptions, and finds middle fusion the best balance.

Turn-taking, backchannels, and interruptions

To train a conversation model, or to evaluate one, we need the events of a conversation defined on what a machine can observe, the voice activity of each speaker over time. A backchannel is a short sound a listener makes without taking the floor, such as "mm-hmm" or "yeah". An interruption is starting to talk while the other person is speaking and outlasting them. The definitions below make these and the other events exact. IPUs, overlaps, pauses, and gaps are the events dGSLM measures (Nguyen et al. 2023), and Moshi's paper reuses them to compare generated dialogues with real ones. dGSLM also sketches backchannels and interruptions but does not count them. We make them exact with the 1-second bound on backchannels that Voice Activity Projection and Full-Duplex-Bench use.

📐 Turn-taking events

Let $a_t, b_t \in \{0, 1\}$ be the voice activity of speakers A and B in frame $t$ of a two-channel recording.

Inter-pausal units. In each channel separately, fill every silence of at most $\theta_{\text{ipu}} = 200$ ms that has speech on both sides. The maximal runs of activity that result are the speaker's inter-pausal units (IPUs). From here on, a speaker counts as active in a frame if the frame lies inside one of their IPUs.
Overlap. A maximal run of frames in which both speakers are active.
Pause and gap. A maximal run of frames in which neither speaker is active, not touching the start or end of the recording, is a silence. It is a pause if the frame just before it and the frame just after it each have the same single active speaker. Otherwise it is a gap, which normally means the speaker changed.
Backchannel. An IPU of one speaker that lasts at most $\theta_{\text{bc}} = 1$ s and lies entirely within an IPU of the other speaker.
Interruption. An IPU of one speaker that starts strictly inside an IPU of the other speaker and ends after that IPU ends, so the other speaker stopped first.

A turn is a maximal sequence of one speaker's IPUs separated only by pauses.

In plain words, pauses are silences inside one person's speech, gaps are silences between two people, and overlaps are both at once. A backchannel is a short "mm-hmm" over the other person's speech, and an interruption is starting while the other person talks and outlasting them.

✏️ Reading events off two tracks

Take 8 seconds of dialogue. Speaker A talks from 0.5 to 1.2 s, from 1.3 to 2.0 s, from 2.5 to 4.0 s, and from 5.5 to 7.0 s. Speaker B talks from 3.0 to 3.3 s and from 4.3 to 6.0 s.

The 100 ms dip in A's speech at 1.2 s is shorter than 200 ms, so it is bridged, and A's first IPU runs from 0.5 to 2.0 s. The silence from 2.0 to 2.5 s has A on both sides, so it is a pause. B's 300 ms from 3.0 to 3.3 s lies inside A's IPU from 2.5 to 4.0 s and is under 1 s, so it is both an overlap and a backchannel. The silence from 4.0 to 4.3 s has A before it and B after it, so it is a gap. A starts again at 5.5 s, inside B's IPU from 4.3 to 6.0 s, and talks until 7.0 s, after B has stopped. So A's IPU from 5.5 to 7.0 s is an interruption, and the overlap from 5.5 to 6.0 s is where it happens. The silences at the two edges, before 0.5 s and after 7.0 s, count as neither pauses nor gaps.

Interview

Turn-taking events from two tracks

Write turn_events(vad_a, vad_b, frame_ms=10, ipu_ms=200, bc_ms=1000), which follows the definition. It bridges silences of at most ipu_ms and counts backchannels of at most bc_ms. The two tracks are boolean arrays of equal length, one entry per frame of frame_ms milliseconds.

Return a dict with five keys. The keys "pauses", "gaps", and "overlaps" each hold a list of (start_ms, end_ms) pairs. The keys "backchannels" and "interruptions" each hold a list of (speaker, start_ms, end_ms) triples, with speaker either "A" or "B". Intervals are half-open. They start at the first frame of the event, end at the frame after its last, and are sorted by start time. The tests include the example above.

import numpy as np

def turn_events(vad_a, vad_b, frame_ms=10, ipu_ms=200, bc_ms=1000):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Measuring full-duplex behavior

How do you know whether a full-duplex model behaves well? Listening tests (§37.6) measure how speech sounds. Turn-taking needs scripted situations with a right answer, and an automatic record of what the model did. Full-Duplex-Bench (Lin et al. 2025) was the first benchmark built this way. It plays a prepared user stream to the model, records the model's output stream, transcribes it with word timestamps, and scores four behaviors.

The central quantity is whether the model takes over the turn, meaning it starts real speech rather than staying quiet or only saying "mm-hmm". Taking over is right when the user has finished and wrong when the user is only pausing.

📐 Takeover rate and the Full-Duplex-Bench metrics

In a test sample, the model takes over ( $\mathrm{TO} = 1$ ) if it produces any speech that is not a backchannel. It does not take over ( $\mathrm{TO} = 0$ ) if it stays silent or only backchannels. Here a backchannel is an output segment shorter than 1 s with fewer than two words. Over $n$ samples the takeover rate is $\mathrm{TOR} = \frac{1}{n}\sum_{i=1}^{n}\mathrm{TO}_i$ .

Pause handling. The user pauses for 0.4 to 1.0 s inside a turn. Lower TOR is better.
Backchanneling. The user speaks at length. Lower TOR is better. The number of backchannels per second should be human-like. Their timing is scored by the Jensen-Shannon divergence (JSD) between the model's backchannel times and those of human listeners, both binned into 200 ms windows. A model that makes no backchannel in a sample is scored as if its times were uniform. JSD measures how different two distributions are. It is the average of the KL divergences (§0.5) of each distribution from their mixture, so it is symmetric, and in bits it lies between 0 (identical) and 1.
Smooth turn-taking. The user finishes a turn. Higher TOR is better. The latency from the end of the user's speech to the start of the model's response is averaged only over the samples with $\mathrm{TO} = 1$ .
User interruption. The user cuts in while the model speaks. Higher TOR is better. A language model (GPT-4o) scores the response from 0 to 5 for how coherent and relevant it is and how well it adapts to the interruption, and the latency after the interruption is averaged over takeovers.

The paper's results for the four systems it could test show how far "full duplex in principle" is from good behavior.

system	pause TOR, real speech (lower better)	backchannel TOR (lower better)	backchannel JSD (lower better)	turn-taking TOR (higher better)	turn-taking latency (s)	interruption score (0 to 5)	latency after interruption (s)
dGSLM	0.935	0.691	0.934	0.975	0.352	0.201	2.531
Moshi	0.980	1.000	0.957	0.941	0.265	0.765	0.257
Freeze-Omni	0.481	0.636	0.997	0.336	0.953	3.615	1.409
Gemini Live	0.310	0.091	0.896	0.655	1.301	3.376	1.183

Read across a row and each system has a personality. Moshi answers fast (265 ms) and almost always takes the turn. But it also talks into 98 percent of the user's pauses, takes the turn in every backchannel test instead of only acknowledging, and its answers after an interruption score below 1 out of 5. Freeze-Omni, whose separate classifier decides when to speak, talks over about half of the user's pauses rather than nearly all. But it misses two thirds of real turn ends and answers in about a second. The commercial Gemini Live is the most polite and the slowest to take a turn. All four take over after most interruptions (TOR from 0.87 to 1.00, not shown), so the 0 to 5 score is what separates them there. No system is good at everything, so the metrics must be read together. The cell shows why on synthetic data.

Python · cell 4

The "shy" system takes over only about a third of the time, yet it reports the same latency as the "eager" one, because latency is averaged over takeovers only. A latency number without its TOR means little. The divergence has a subtlety too. A model's backchannels in one sample are one or two events, a spike, while the human distribution pools many listeners (59 per stimulus on average in the human data the benchmark uses) and is spread out. So even a perfectly timed single backchannel scores 0.52 here, not far from the 0.60 of a silent model scored as if uniform. The metric stays far from zero even for good timing, and the differences between systems are small, with every system in the table between 0.89 and 1.00.

🔬 Where the field stands

Two surveys put these numbers in context. Chen and Yu (2025) separate engineered synchronization, modular systems with a turn-taking controller like Freeze-Omni, from learned synchronization, end-to-end models like Moshi. They name the scarcity of real two-channel dialogue data as one of three open challenges, with diverging architectures and gaps in evaluation. Lu et al. (2026) audit more than a dozen systems that call themselves full duplex and find what they call a realization gap. Architectures that could act in every duplex state in principle behave only in the patterns their training and evaluation data contain. The field is moving fast. An arXiv search finds 48 papers submitted in September 2026 whose abstracts mention full duplex together with speech, spoken language, or dialogue, against 9 a year earlier, so treat any ranking as provisional.

The practical lesson for an interview is to never quote one duplex metric alone. Ask for the takeover rate in each situation, the latency over takeovers next to it, and the quality of answers after interruptions.

Building a production voice agent

A deployed agent wraps one of these models in a loop that also calls tools, keeps state, and follows policy (§24.1). Voice is one more interface to that loop, and the protocols that connect an agent to tools and to other agents (§24.6) apply unchanged. Voice adds three engineering problems that text agents do not have. The agent gets interrupted, it hears itself, and it hears things it should not obey.

Barge-in and echo

Barge-in is the user starting to talk while the agent is speaking. A half-duplex agent must decide three things within a fraction of a second. Is that sound the user at all? Is it a backchannel (keep talking) or an interruption (stop)? And how fast should it stop?

The first decision needs acoustic echo cancellation (AEC). The microphone hears the loudspeaker, so without AEC the VAD fires on the agent's own voice and the agent interrupts itself. An AEC filter keeps an adaptive estimate of how the loudspeaker's signal arrives at the microphone. It subtracts that estimate from the microphone signal and flags frames where what remains is only echo. A full-duplex model can also learn some robustness, which is why Moshi's instruction tuning added simulated echo of its own voice to the user's stream.

The second decision is a classification under time pressure. Backchannels are short, under a second by the definitions above. So a rule that stops only after the user has spoken continuously for $\theta_{\text{barge}}$ ignores most "mm-hmm"s, at the price of a stop latency of $\theta_{\text{barge}}$ for real interruptions. Cues from the streaming transcript ("uh-huh", "yeah", "right") and a turn-taking predictor such as VAP can stop faster on real interruptions and slower on acknowledgments. After stopping, the agent waits until the user has finished, using the endpointing described above, and answers the new turn. It never resumes the reply it abandoned.

Core

A barge-in controller

Write barge_in(user_vad, echo, response, barge_frames, resume_frames). The inputs are per-frame arrays of equal length. user_vad[t] is the VAD's output. echo[t] is true when AEC flags that frame's activity as the agent's own echo. response[t] is the id of the reply the agent has ready to play at frame $t$ , with 0 for none and positive integers as ids.

A frame has user speech if user_vad[t] is true and echo[t] is false. Keep two counters that include frame $t$ , the length of the current run of consecutive user-speech frames and the length of the current run of frames without user speech. The controller then applies three rules in each frame, in this order.

If it is playing reply $k$ and response[t] is no longer $k$ , it stops, because the reply ended.
If it is playing reply $k$ and the user-speech run has reached barge_frames, it stops, records the interruption as (first frame of the run, t), and cancels $k$ for good.
If it is not playing (including after a stop in this frame), it starts reply response[t] when that id is positive, not cancelled, and the run without user speech has reached resume_frames.

Return (play, stops). Here play is a boolean array that is true in the frames where the agent plays audio, and stops is the list of recorded interruptions.

import numpy as np

def barge_in(user_vad, echo, response, barge_frames, resume_frames):
    play = np.asarray(response) > 0
    return play, []

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

With 10 ms frames, the tests' barge_frames = 30 means the agent stops 300 ms into an interruption and talks through any shorter burst. A full-duplex model has no such rule. It decides from the content of both streams, which is more natural when it works and hard to tune when it does not, as Moshi's pause-handling score shows.

A cascaded agent loop

In production, a streaming cascade is an event loop. The microphone pushes frames, and a streaming recognizer with an endpointer emits events. A reply task streams tokens from the language model and synthesizes them clause by clause. Barge-in cancels the reply task and flushes the audio already queued. The sketch below shows the control flow with stand-in clients.

One detail matters more than it looks. After an interruption, the conversation history must contain only what the user actually heard. Otherwise the model will later refer to sentences it never finished saying.

Three more techniques hide latency in production. Speculative starts run the language model on the partial transcript before the endpoint fires and discard the result if the transcript changes. Fillers such as "let me check that" are synthesized at once while a slow tool call runs, the voice version of a loading spinner. And a tool call in the middle of an answer (§24.2) must be scheduled so that speech already playing covers its latency.

Spoken prompt injection and voice safety

A voice agent hears everything near its microphone, such as a television, a voicemail it was asked to summarize, or the audio of a web page or video its tools opened. Instructions inside that audio are prompt injection through a new channel (§19.3). They are harder to spot than injected text, because no one reads the transcript. The defenses carry over from text agents. Audio from tools and third parties is data, never instructions. Consequential actions need confirmation. And the agent's permissions are the smallest that do the job (§19.7). GPT-4o's system card also reports anecdotal evidence that safety robustness drops with low-quality or noisy input audio, with echoes, and with interruptions while the model is speaking, so these audio conditions belong in the red-teaming set.

The output side has its own risk. A model that can speak in any voice can speak in yours. OpenAI's approach for GPT-4o allows only preset voices, which are used as the target outputs during post-training. A separate classifier also checks the generated audio as it streams and blocks the output if the speaker does not match the chosen voice. The system card reports a precision of 0.96 and a recall of 1.0 for English. Moshi gets voice consistency from training on one voice. §37.6 covers voice cloning, spoofing, and watermarks in depth.

What a duplex model costs to serve

Running a 7B model per frame sounds expensive. Is it? We count it the way §5.6 and §6.5 taught. A forward pass costs about $2N$ FLOPs per token for a model with $N$ weights in its matrix products. The KV cache, the keys and values a Transformer stores for every past position, holds $M_{\text{KV}} = 2Ln_{kv}d_kT$ numbers for $L$ layers, $n_{kv}$ key-value heads of width $d_k$ , and $T$ cached positions.

At batch size 1, decoding is limited by memory bandwidth rather than arithmetic, because every step must read all the weights once (§20.1). The cell first counts Moshi's parameters from its published configuration (Table 1 of the paper). It then asks how long one read of the weights takes per frame.

Python · cell 5

On an L4, with 300 GB/s of memory bandwidth, a single conversation spends about 49 ms of every 80 ms frame reading weights. That leaves room, and it fits the 200 ms latency reported on that GPU. On an H100, with 3.35 TB/s, it takes 4.4 ms. Batching many conversations shares each weight read across them, exactly as for text decoding (§20.6). Every conversation advances one step per frame in lockstep, so the batch never has ragged lengths. What limits the batch is then the KV cache, which the exercise computes.

Research

The cost of listening and speaking at once

Write duplex_cost(cfg, minutes) for a multistream model built on an RQ-Transformer. Each frame, the model writes its own $Q$ levels with the depth model and reads the user's $Q$ levels. The dict cfg has these keys.

n_temporal and n_depth, the weights in matrix products of the temporal model and of the depth model used for one level.
frame_rate, and Q, the codec levels per speaker.
The temporal model's L (layers), h (query heads), n_kv (key-value heads), and d_k (head width).
bytes per cached number, and context, the attention window in steps, or None for unlimited.

Return a dict with these keys.

streams, the number of streams per step, and steps_per_s, the temporal steps per second.
flops_per_s, the forward FLOPs per second for one conversation, by the $2N$ rule for the temporal model and for every depth step.
kv_bytes, the temporal model's KV cache after minutes of conversation, capped by context.
flat_tokens_per_s, flat_flops_per_s, and flat_kv_bytes, the same three numbers for a single Transformer that reads all streams flattened into one sequence and has no depth model. It has n_temporal weights, the same layers and heads as the temporal model, and the same attention window, counted in its own positions.
context_seconds and flat_context_seconds, the seconds of conversation the attention window covers in each case (inf without a window).

def duplex_cost(cfg, minutes):
    f = cfg["frame_rate"]
    return {"streams": cfg["Q"] + 1, "steps_per_s": f,
            "flops_per_s": 2 * (cfg["n_temporal"] + cfg["n_depth"]) * f,
            "kv_bytes": 2 * cfg["L"] * cfg["h"] * cfg["d_k"] * cfg["bytes"] * minutes * 60 * f}

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

✏️ Ten minutes of full duplex, at real scale

Put in Moshi's numbers, about 6.6 billion weights in the temporal blocks and 0.1 billion per depth level. One conversation costs

2\times 6.6\times10^9\times 12.5 + 2\times 10^8\times 8\times 12.5 \approx 1.85\times 10^{11}

FLOPs per second, a rounding error on a GPU that does $10^{15}$ . Helium has 32 layers and 32 key-value heads of width 128, with no grouped-query sharing (§12.5). So each step caches $2\cdot 32\cdot 32\cdot 128\cdot 2 = 524{,}288$ bytes, half a mebibyte. Ten minutes is 7,500 steps, or 3.9 GB of cache.

Moshi's attention window is 3,000 steps, exactly 4 minutes. So the cache stops growing at 1.57 GB, and anything older than 4 minutes is forgotten (§25.1 is about what to do with longer conversations). On an 80 GB H100, after 15.6 GB of weights, about 40 full-window conversations fit. A flattened model would need 212.5 steps per second, each within 4.7 ms, and 17 times the cache per second, and its 3,000-token window would hold only 14 seconds of conversation. The split into a slow temporal model and a fast depth model is what makes the frame rate affordable.

Choosing a design

No design wins on every axis, and the right one depends on what the product must guarantee. The latencies below are the papers' own numbers or our worked example, each measured from a different clock, as the pitfall on comparing latency numbers warns.

design	example	first sound after the user stops	what survives of the user's voice	overlap, backchannels, barge-in	text model reused	control and audit
cascade, serial	early voice assistants	seconds (2.5 s in our example)	words only	none, barge-in by a rule	any, unchanged	best, with transcripts and swappable parts
streaming cascade	streaming recognizer, text model, and synthesizer	1.25 s in our example	words only	barge-in by a rule	any, unchanged	best
chain of modality	SpeechGPT	grows with the answer, the whole text comes first	units, mostly content	none	fine-tuned	text is visible
parallel heads	Mini-Omni, LLaMA-Omni	236 ms after the instruction ends (LLaMA-Omni)	what the speech encoder keeps	none built in	fine-tuned	text is produced alongside
Thinker and Talker	Qwen2.5-Omni to Qwen3.5-Omni	234 ms theoretical first packet, plus the endpoint	encoder features, including tone	through a turn controller	trained jointly, scores close to text-only models	Qwen3-Omni's text can be filtered before it is spoken
full-duplex multistream	Moshi	160 ms floor, about 200 ms in practice, no endpoint	all of it, in both directions	learned, native	fine-tuned, loses some knowledge	hardest, behavior is learned

Three examples show how to use the table. A bank's phone agent must follow policy, call tools, and keep an auditable transcript. A streaming cascade, or a Thinker-Talker model whose text can be checked before it is spoken, is the safe choice, and its latency can be bought down with speculation and fillers. A language-learning tutor needs to hear pronunciation and hesitation, which text discards, so an end-to-end model that reads audio is worth its loss of control. A companion app lives on natural timing, backchannels, and laughter. That is the case for full duplex, provided the team can evaluate turn-taking as carefully as Full-Duplex-Bench does.

📌 The duplex frontier, 2024 to 2026

September 2024, Moshi, a 7B full-duplex model with an inner monologue, a 160 ms theoretical latency, and about 200 ms in practice.
October 2024, the GPT-4o system card, with audio answers in 232 ms at best and 320 ms on average, and preset voices enforced by a streaming output classifier.
March 2025, Full-Duplex-Bench, the first benchmark of pause handling, backchanneling, turn-taking, and interruption, in which no tested system does well on all four.
March and September 2025, Qwen2.5-Omni and Qwen3-Omni, the Thinker-Talker models, the second with a 234 ms theoretical first packet.
September 2025, delayed streams modeling, one recipe for streaming recognition and synthesis by shifting one stream against another.
April 2026, Qwen3.5-Omni, with hundreds of billions of parameters and more than 100 million hours of audio-visual data.
June 2026, a survey of full-duplex systems finds a realization gap between what architectures allow and how they behave.
September 2026, 48 new arXiv papers on full-duplex speech in one month.

What comes after full duplex? A model that predicts how a conversation will unfold, who will speak next and what they will say, is learning the dynamics of an interaction. It is the same idea that world models apply to scenes and to agents acting in them.

🧭 What to remember

A voice turn's latency is the sum of endpointing, final recognition, the language model's first tokens, the synthesizer's first packet, and the network. Streaming hides generation time at each hand-off, but nothing can overlap the endpoint, so $t_{\text{end}} + t_{\text{net}}$ is the floor of every turn-based design. A silence endpointer trades cut-offs against delay, and only content-aware endpointing improves both.
A cascade loses everything that is not words and all of conversation's timing, and it keeps an unchanged text model, transcripts, and control. End-to-end designs put speech into one model in four ways, as vocabulary units with a chain of modality, by word-level interleaving, with parallel text and speech heads, or with a Thinker and a Talker. They pay in control and in some of the text model's knowledge.
Moshi models 17 streams at 12.5 Hz, its text, its 8 Mimi levels, and the user's 8, with acoustic levels delayed by $\tau_{\text{ac}}$ frames. Its reaction time is at most $(1 + \tau_{\text{ac}})\Delta_{\text{fr}}$ plus compute, 160 ms for one 80 ms frame of delay, and a spoken word starts one frame after its EPAD.
The inner monologue aligns Moshi's own words to frames with PAD and EPAD and gives the largest quality gain of any design choice in its ablations. Shifting text against audio turns the same model into a streaming recognizer or synthesizer, the idea of delayed streams.
Turn-taking events (IPUs, pauses, gaps, overlaps, backchannels, interruptions) have exact definitions on two voice-activity tracks. Full-duplex behavior is measured by takeover rates, latency over takeovers, and backchannel timing, and no system tested in 2025 did well on all four behaviors.
A production agent adds echo cancellation, a barge-in policy that ignores backchannels, a history that records only what was heard, defenses against spoken prompt injection, and preset voices with an output classifier.
Next, §37.6 asks whether all this generated speech is any good, with listening tests and their statistics, and how to protect people from voice cloning.

Défossez et al. (2024). Moshi: a speech-text foundation model for real-time dialogue. The architecture, the multistream layout (Section 3.4), the inner monologue, the training recipe, and the ablations behind the 160 ms floor, with code in the Moshi repository.
Lin et al. (2025). Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities. The takeover-rate metrics and the first comparison of duplex systems on pauses, backchannels, turn ends, and interruptions.
Nguyen et al. (2023). Generative Spoken Dialogue Language Modeling. dGSLM, the textless two-channel ancestor of Moshi, and the turn-taking statistics (IPUs, pauses, gaps, overlaps) used to compare generated and real dialogue.
Xu et al. (2025). Qwen3-Omni Technical Report. The Thinker-Talker design at scale, with a component-by-component first-packet budget in its Table 2.
Fang et al. (2025). LLaMA-Omni: Seamless Speech Interaction with Large Language Models. A CTC-trained streaming speech decoder on top of a Llama model, and a clean study of quality against latency.
Wang et al. (2024). Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM. How far a frozen text model can go, a measured latency breakdown, and a learned turn-state classifier.
Ekstedt & Skantze (2022). Voice Activity Projection: Self-supervised Learning of Turn-taking Events. Predictive turn-taking from voice activity, the model behind content-aware endpointing and backchannel prediction.
OpenAI (2024). GPT-4o System Card. Response latency, preset voices with a streaming voice classifier, and the audio-specific risks found in red teaming.

Made for larger screens

Speech-to-speech and full duplex

The cascade, and three changes that shorten the wait

Where the time goes in a voice turn

Time to first sound

Real budgets, read from the papers

Deciding that the user has finished

What a cascade loses besides time

Putting speech and text in one model

Moshi, a model that listens while it speaks

The multistream layout

Moshi's streams

The latency floor of a frame-based model

Inner monologue

Inner monologue alignment

One spoken answer, frame by frame

Delayed streams turn one model into a recognizer and a synthesizer

How Moshi was trained

Turn-taking, backchannels, and interruptions

Turn-taking events from two tracks

Measuring full-duplex behavior

Building a production voice agent

Barge-in and echo

A barge-in controller

A cascaded agent loop

Spoken prompt injection and voice safety

What a duplex model costs to serve

The cost of listening and speaking at once

Choosing a design

Read more

Watch next