This section is about models that connect video with language. They find the clip that matches a sentence, answer questions about a video, and say when something in it happened. These three skills sit behind video search, video assistants, and the video understanding of every frontier model, and each is a standard interview topic.

Here are three requests a video product receives every day. Find the clip in which "a person pours water into a glass" among 10,000 clips. Watch a 40-minute cooking video and answer "what did she do right after pouring the water?" Then say when it happened, to the second. The first request is retrieval, finding the item that matches a query. The second is question answering. The third is temporal grounding, finding the stretch of time that a sentence describes.

All three run into one budget. A language model reads a video as tokens, the vectors it reads one after another, and §29.1 counted the bill. Each sampled frame costs a few dozen to a few hundred tokens. The Gemini API, for example, samples 1 frame per second and spends 66 tokens per frame at its default setting and 258 at higher settings (70 and 280 on the Gemini 3 and 4 models). The 40-minute video is 2,400 seconds, so it gives 2,400 frames. At 258 tokens each it costs $2400\cdot258 = 619{,}200$ tokens before a single word of the question, and even the default gives $2400\cdot66 = 158{,}400$. A model's context window is the most tokens it can read at once, and 158,400 tokens is more than a 128K-token window holds.

We take the requests in order. Retrieval comes first. There, averaging the frames of an image-text model already works well, which is both good news and a warning. Then we learn from narrated videos, whose speech is often out of step with the picture. Next come question answering and the tests that show whether a benchmark needs time at all, and then temporal grounding and its metrics. Last, we design systems for long videos and live streams under a fixed token budget, and close with the frontier as of October 2026.

Retrieval starts from an image-text model

The simplest way to search video with text is to reuse a model that already matches images with text. Such a model is a dual encoder (§7.2). It has two networks. An image encoder $f$ maps an image $x$ to a vector, and a text encoder $g$ maps a caption $y$ to a vector in the same space $\R^d$. Both outputs are unit vectors, of length 1, so their dot product is the cosine of the angle between them. Training gives a matching image and caption a high cosine. CLIP, previewed in §1.8, is the best-known dual encoder.

A dual encoder is trained with a contrastive loss called InfoNCE. Take a batch of $B$ image-caption pairs. Each image scores every caption in the batch, and the loss asks it to pick out its own caption, like a multiple-choice question with $B$ options. The scores are called logits. With a temperature $\tau > 0$ they are $s_{ij} = f(x_i)^\top g(y_j)/\tau$, and the loss for image $i$ is the cross-entropy of choosing caption $i$,

CLIP averages this loss with the same loss taken down the columns, where each caption picks out its image. The gradient of $\mathcal{L}_i$ with respect to row $i$ of the logits is the row's softmax minus the one-hot vector of the right answer. In plain words, a gradient step raises the right logit in proportion to one minus its probability, and lowers every other logit in proportion to its probability. §7.2 calls this pattern "expected minus observed". How CLIP is trained at scale is the subject of §30.2. Here we need only that $f$ and $g$ exist and that their cosine ranks image-caption pairs well.

A video is not one image, so the first question is how to turn the $n$ sampled frames of a clip into one score against a caption. Write $x^{(1)}, \dots, x^{(n)}$ for the frames, $\mathbf{z}_k = f(x^{(k)}) \in \R^d$ for the unit embedding of frame $k$, and $\mathbf{t} = g(y)$ for the unit embedding of the caption. Before scoring clips, we fix how retrieval is measured.

import numpy as np

def retrieval_metrics(S):
    raise NotImplementedError

Averaging the frames

The most direct way to score a clip is to average its frame embeddings and compare the average with the caption. Luo et al. (2021) tested this against two ways of adding temporal modeling on top of CLIP, in a study they called CLIP4Clip.

The parameter-free score deserves a close look, because one exact identity explains both why it works and what it cannot do. Two frames show it.

In plain words, a mean-pooled clip score is the average of the per-frame scores, scaled up by a factor that depends only on how alike the frames are. A clip whose frames all look the same has $\lVert\bar{\mathbf{v}}\rVert$ near 1, and its score is the average frame score. And because the score ignores order, a clip of a man opening a door and the same clip played backward, which shows him closing it, get exactly the same score against every caption. Any pooling that treats the frames as an unordered set (mean, max, or attention pooling without positions) has the same blind spot.

That blind spot matters only if the captions are about order, and many are not. A caption such as "a band plays on a stage" describes every frame of its clip. Averaging is then the right thing to do, because it averages away the noise of each frame. A caption about a short event is different. Averaging dilutes the few frames that show it.

The cell below makes this concrete with synthetic embeddings. Each of 300 clips has 12 noisy frames. The frames that show the event point toward the caption, and the rest point toward the clip's background. We compare three ways of scoring a clip, mean pooling, the best single frame, and the mean of the best three frames.

When the caption describes the whole clip, mean pooling retrieves all 300 clips. The best single frame gets only 0.640, because the maximum of 12 noisy scores rewards whichever clip had a lucky frame. As the event shrinks, the order reverses. With 3 of 12 frames, mean pooling falls to 0.087 while the top-3 mean reaches 0.403. With one frame, the maximum does best (0.113) and the other two nearly vanish. Mean pooling is excellent for captions that describe a scene and poor for captions about a moment. Moments need the grounding methods later in this section.

The standard benchmark for this task is MSR-VTT, 10,000 web clips of 10 to 32 seconds with 20 captions each. Most papers test on a fixed set of 1,000 clip-caption pairs and train on about 7,000 or 9,000 other clips. A result is zero-shot when the model had no training on the benchmark's own data.

Mean pooling winning with less data is not a quirk. Lei et al. (2022) trained a model on one randomly chosen frame per video. With a good way of combining 12 frames at test time, it matched or beat multi-frame models pretrained on similar amounts of data, on standard retrieval and question-answering benchmarks. They call this the static appearance bias of the benchmarks. It is the language-side twin of what §29.3 found in action classification, where a sport is usually recognizable from one frame by its field and equipment.

To expose the bias, Lei et al. built retrieval tasks from Something-Something v2, a dataset of short clips of hands doing things to objects. Its captions are templates such as "Throwing [something] in the air and catching it", with the objects removed so that only the motion is described. There the single-frame model got 42.0 R@1. The same model with a 2-layer temporal Transformer over 4 frames, after a second stage of pretraining on video, got 77.0. In practice, check whether your queries are about motion before you add temporal machinery. An interviewer will ask why mean pooling is so strong, and the identity above is the answer.

Clips that win too many queries

Rank a gallery by cosine and count, for each clip, how many queries rank it first. You might expect each clip to win about one query. In high-dimensional embedding spaces the counts are very uneven instead. A few clips, usually generic ones near the center of the data, are the top match for many queries, and many clips are the top match for none. This is called hubness, and the generic clips are hubs. Radovanović et al. (2010) studied it in high-dimensional data in general, and Bogolin et al. (2022) showed that it is pervasive in text-video retrieval, MSR-VTT included. A hub steals the top rank from queries that belong to other clips.

The fix is to discount a clip that scores highly against many queries. For each clip, take a softmax of its scores over a set of queries. A hub spreads that softmax over many queries, so each of its weights is small. A specific clip keeps almost all of its weight on the one query it matches.

Here is the idea with three queries and a temperature of 0.1. Clip A is a hub and scores 0.30 against each query. Clip B scores 0.30 against query 1 and 0.10 against the other two. By raw cosine, A and B tie for query 1. A's softmax over the queries gives each query weight 1/3. B's gives query 1 weight $e^{3}/(e^{3} + 2e^{1}) = 0.787$. Multiplying each score by its weight, query 1 now scores B at $0.30\cdot0.787 = 0.236$ and A at $0.30/3 = 0.100$, so B wins.

The cell below builds a gallery with hubs. Every caption leans toward one shared direction, and a few clips are generic and lean the same way. We compare raw cosine, dual softmax over the test queries, and querybank normalization with a bank of 1,000 other captions. The helper lse is the log-sum-exp, $\log\sum_j e^{s_j}$, computed stably by subtracting the maximum first.

Raw cosine gets R@1 of 0.380. Its biggest hub wins 13 queries, and 46 percent of the clips win none. Querybank normalization lifts R@1 to 0.435 and dual softmax to 0.529, with the biggest hub down to 5 and 4 queries. Real gains are smaller, because real hubness is milder. On MSR-VTT text-to-video R@1, dual softmax raised the sequential CLIP model from 44.5 to 47.0 and zero-shot CLIP from 31.2 to 35.6 (Cheng et al. 2021, Table 6). Querybank normalization raised another CLIP-based retriever, CLIP2Video, from 45.6 to 47.2 (Bogolin et al. 2022).

Since CLIP4Clip, video-text encoders have been trained for video from the start. They combine contrastive learning on hundreds of millions of clips with masked video modeling, which hides most of a clip and trains the model to fill it in (§29.4).

Learning from narrated video

Captions written by people are expensive, so the first large video-text datasets used what people say. HowTo100M (Miech et al. 2019) took 1.22 million narrated instructional YouTube videos covering over 23,000 tasks. It cut them into 136 million clips, one per segment of the automatic speech transcript, and paired each clip with the words spoken during it.

The trouble is that people do not narrate in step with their hands. "Now we add the salt" comes before the salt, and "that's the dough done" comes after. The authors of the loss below cite an estimate that about half of HowTo100M's clip-narration pairs are not visually aligned (Miech et al. 2020). Here is what one misaligned pair does to the InfoNCE gradient.

The cell reproduces these numbers.

The bag idea has a name. In multiple instance learning, a label belongs to a bag of examples without saying which one. In noise-contrastive estimation, a model learns by telling true pairs from sampled negatives, which is what InfoNCE does. Joining the two gives MIL-NCE. To write it, let $\operatorname{LSE}_{j\in J}(s_j) = \log\sum_{j\in J}e^{s_j}$ be the log-sum-exp over a set of indices $J$.

In plain words, the softmax over all candidates gives each narration a probability of being the match. The events "the match is $y_j$" exclude one another, so the probability that the match is somewhere in the bag is the sum of their probabilities. MIL-NCE is the cross-entropy of that one event. With a bag of one, it is exactly InfoNCE.

In plain words, the total upward push on the bag equals the probability the model still gives to negatives, $1 - m$. The bag divides that push among its members in proportion to how plausible the model already finds each one. A misaligned candidate that the model scores low gets almost nothing. In the example, the push of 0.100 on the negative was split 0.018 and 0.081 between $y_1$ and $y_2$. This is the "expected minus observed" gradient of §7.2, with the one-hot target replaced by the bag's own posterior $a^{\mathcal{P}}$.

Two alternatives are worth knowing, because an interviewer will offer them. The first sums the per-candidate losses, $-\sum_{p\in\mathcal{P}}\log a_p$. Its gradient on each positive is $|\mathcal{P}|\,a_p - 1$, so the upward push on a candidate is $1 - |\mathcal{P}|\,a_p$. That push is largest for the candidates the model finds least plausible, which are usually the wrong ones. The second keeps only the best-scoring candidate, a hard maximum over the bag, which can lock onto an early mistake. MIL-NCE is the smooth choice between the two.

A simulation shows the robustness directly. Each of 32 videos has 8 steps. Each clip's narration describes its own step, except that with probability $p$ it describes the step before or after. We freeze the clip embeddings and learn a linear text encoder $g(y) = \operatorname{unit}(\mathbf{W}y)$, where $\operatorname{unit}$ scales a vector to length 1. We train it once with InfoNCE on the timestamp pairs and once with MIL-NCE on bags of the clip's own step and its two neighbors. The test uses 256 fresh, correctly aligned pairs. The simulation is kind to MIL-NCE, since every narration that describes a clip's step lies in that clip's bag. The training loop uses the gradient theorem above, and it gets the softmax within the bag from the full one by $a^{\mathcal{P}}_p = a_p/m$.

With clean narrations the two losses nearly tie (0.977 and 0.996). At 30 percent misalignment InfoNCE drops to 0.918 while MIL-NCE stays at 0.984. At 50 percent, roughly HowTo100M's estimated rate, InfoNCE falls to 0.863 and MIL-NCE holds 0.965.

import numpy as np

def mil_nce(S, pos_mask):
    raise NotImplementedError

Narration was the data of 2019 and 2020. Since then the text has shifted from what people said to what models write. In dense recaptioning, the 2024 to 2026 recipe, a strong captioning model describes every few seconds of video in detail. That moves the alignment problem out of the loss and into the data pipeline. The bag idea of MIL-NCE survives wherever text is only loosely attached to the picture, as with transcripts, comments, or chapter titles.

Video question answering

A video language model answers free-form questions about a video, which retrieval cannot do. Almost all of them share one design, which §31.1 builds for images and §31.5 extends to video. A vision encoder turns frames into vectors. A small network called a projector maps those vectors into the language model's embedding space. The language model then reads them as if they were word tokens.

Following one question through a real model makes the pattern concrete.

The frontier systems of 2025 and 2026, such as Qwen2.5-VL, Qwen3-VL, and Gemini, follow the same pattern at a larger scale.

The benchmarks differ in what they test, and an interviewer will expect you to know which is which. The numbers are from each benchmark's paper.

Does a benchmark need time at all

A single frame does well on many benchmarks. So ask of any video question-answering score how much of it needs the video, and how much of that needs more than one frame. The three baselines answer this. The blind score measures language priors, such as options that are implausible on their face. The single-frame score measures appearance. Full-video accuracy minus single-frame accuracy is the part of a score that can be credited to time. The shuffled baseline then checks whether that part depends on the order of the frames.

A second tool measures how much video a question needs, as opposed to how long the video is.

EgoSchema's median certificate is about 100 seconds. By the authors' measure that is 5.7 times longer than the next-longest dataset they measured and 10 to 100 times longer than the rest. A long video does not guarantee a long certificate. The paper's own example is a grounding dataset of 110-minute movies whose moments last about 4 seconds, so its certificates are a few seconds long. Yet two findings show how much of even EgoSchema is language. A pipeline that captions short clips and lets a text-only language model answer from the captions reached 52.2 percent (Zhang et al. 2023a, a system called LLoVi, whose first version reported 50.3), when end-to-end models were below 33. A language model that decides which frames to look at reached 54.1 percent while looking at 8.4 frames per question on average (Wang et al. 2024d, called VideoAgent). Long-video understanding, as these benchmarks measure it, is often a search problem followed by a language problem.

Benchmark scores are also less precise than they look. The standard error of an accuracy is the typical gap between the measured accuracy and the accuracy the model would get on unlimited questions of the same kind (§0.11). An accuracy $\hat p$ over $n$ independent questions has standard error $\sqrt{\hat p(1 - \hat p)/n}$. Questions often come in groups of $m$ that share a video, and answers in a group are correlated, with correlation $\rho$. Then the variance grows by the design effect $1 + (m - 1)\rho$.

import numpy as np

def temporal_necessity(blind, single, full, n_boot, rng):
    raise NotImplementedError

Reinforcement learning on video questions

Since 2025 the reasoning-style post-training of language models has been applied to video. GRPO (§18.4) samples a group of answers to each question, rewards each one, and pushes up the answers that scored above the group's average. It needs only a verifiable reward, one a program can compute (§18.3). Multiple-choice and numeric video questions have one.

Plain GRPO has a weakness that the diagnostics above predict. If a question can be answered from one frame, the policy is rewarded for answering it from one frame, and nothing pushes it to use time. Video-R1 (Feng et al. 2025) turns the shuffled baseline into a training signal, in a variant it calls T-GRPO. For each question it samples one group of answers from the frames in order and a second group from the same frames shuffled. Correct answers in the ordered group earn a bonus only on questions where the ordered group does at least as well as the shuffled one.

Temporal grounding

The third request was "say when it happened". Temporal grounding, also called moment retrieval, takes a video and a sentence and returns a time interval, a moment $[t_s, t_e]$ from a start time to an end time in seconds. It is judged by how much the predicted interval overlaps the true one. The measure is the one-dimensional version of the box IoU of §27.6, the intersection over union, the length the two intervals share divided by the length they cover together.

When the moments overlap, the union is also the span from the earliest start to the latest end, $\max(a_2, b_2) - \min(a_1, b_1)$, the form many papers write. Gao et al. (2017) defined the recall with tIoU "larger than" $\mu$, while most code uses "at least" $\mu$. The two differ on exact ties, which whole-second moments often produce.

How demanding is a threshold? A short calculation turns it into seconds, and it is the kind of estimate an interviewer likes.

In plain words, IoU 0.5 tolerates a shift of a third of the moment's length, and IoU 0.7 only $0.3/1.7 = 17.6$ percent. For an 8-second moment that is 2.7 seconds and 1.4 seconds. A model that sees one frame per second cannot place a boundary more precisely than the gap between frames. So R@1 at IoU 0.7 on short moments is partly a test of the sampling rate. Qwen3-VL's report, for instance, evaluates grounding on Charades-STA at 4 frames per second rather than the 2 it uses for other video benchmarks (Bai et al. 2025b).

import numpy as np

def tiou(a, b):
    a, b = np.asarray(a, float), np.asarray(b, float)
    inter = np.minimum(a[..., 1], b[..., 1]) - np.maximum(a[..., 0], b[..., 0])
    return inter / ((a[..., 1] - a[..., 0]) + (b[..., 1] - b[..., 0]))

def recall_at_iou(preds, gts, n, thr):

Temporal IoU is itself a verifiable reward. VideoChat-R1 (Li et al. 2025) trained Qwen2.5-VL-7B with GRPO on several video tasks at once. On grounding it rewarded each sampled answer with the tIoU of its predicted moment against the ground truth, plus a reward for the output format. Charades-STA mIoU rose from 29.0 to 60.8 (Table 1). Unlike a 0/1 answer reward, tIoU gives partial credit. A group of samples that all miss the moment slightly still has reward differences for GRPO to learn from.

How a language model writes a time

A language model says "from 12.5 to 17.0 seconds" by writing the numbers as text, so it must know the time of every frame it sees. §29.2 describes three ways to tell it.

1. Frame-index positions. Each token's time position says which frame it came from. The same index means different times at different sampling rates, so the model cannot report seconds unless the rate is fixed.
2. Absolute-time position IDs. A rotary position embedding rotates each query and key by angles proportional to its position (§5.5). Qwen2.5-VL ties the time part of that position to seconds. This works at any rate, but the positions become large and sparse for long videos.
3. Text timestamps. Qwen3-VL puts a string such as <3.0 seconds> before each group of frame tokens, so the model copies times it can read. TimeChat (Ren et al. 2023) did an earlier version inside its frame encoder, binding each frame's features to a description of its timestamp.

In every case the times must be the frames' presentation timestamps, the display times stored in the file. Frame index divided by frame rate is wrong for files recorded at a variable frame rate, as phones often do (§29.1).

Removing duplicate moments

Grounding models, like object detectors, often output several overlapping proposals for the same moment. One moment could then fill all five slots of R@5. The fix is greedy non-maximum suppression (NMS), as in §27.6 (Exercise 27.6b), with tIoU in place of box IoU. Visit the proposals from the highest score down. Keep each one unless its tIoU with an already kept proposal exceeds a threshold $\theta_{\text{nms}}$. A suppressed proposal never suppresses anything. One limit remains. A single threshold cannot tell two detections of one moment from two true moments that overlap.

import numpy as np

def temporal_nms(moments, scores, thr):
    raise NotImplementedError

The first widget below shows tIoU for one predicted moment. Give the prediction the ground truth's length and drag it to the right. Before you do, use the boundary theorem to predict the shift, as a fraction of the length, at which it stops counting at IoU 0.7 and at IoU 0.5.

The second widget ranks three scored predictions for one query, with and without NMS, and shows R@1 and R@2 at IoU 0.5. Two near-duplicates of a wrong moment outscore the right one. Switch NMS on and off and ask which recall changes, and why. Then drag the lowest-scored prediction until it partly overlaps the top one and lower the NMS threshold, to find where NMS starts to hurt.

Long video under a context budget

Now return to the 40-minute video, or a 2-hour one. Two facts from §29.1 set the problem.

• The token bill. A video of $\ell$ seconds, sampled at $f_{\text{smp}}$ frames per second with $c_f$ tokens per frame, costs about $\ell f_{\text{smp}} c_f$ tokens.
• The capture probability. A short event of $\delta$ seconds appears in at least one sampled frame with probability $\min(1, \delta f_{\text{smp}})$, when the sampling grid starts at a random time.

Put the two together, and a token budget becomes a probability of seeing the event.

In plain words, under uniform sampling the chance of seeing a short event is the number of frames the budget buys, times the event's length, divided by the video's length. Halving the tokens per frame doubles it, and doubling the video halves it. Resolution and coverage trade one for one. That is why systems for long video use low-resolution frames. It is also why "sample uniformly" is almost never the right answer for a question about a short event.

The widget runs this arithmetic for any video length, frame cost, and context, with frames spread evenly to fill the budget, at most one a second. The curve shows the chance of seeing a 3-second event at every video length.

Picking frames for the question

Uniform sampling spends the budget evenly, but a question is not about every second equally. Suppose we can score frames against the question cheaply, with a small image-text encoder run over the whole video at 1 frame per second. Then we can spend the language model's budget only on the frames that matter. This is query-aware frame selection.

The cell below simulates it for a 2-hour video. Frame embeddings come from 40 scenes plus noise. Five percent of frames are loosely related to the question, and a 3-second moment matches it more strongly. Uniform sampling gets the 465 frames that fit at 258 tokens per frame, and its result is checked against the coverage theorem. Query-aware selection takes the 64 or the 465 frames with the highest cosine to the question.

Uniform sampling sees the moment 19.2 percent of the time, as the theorem predicts (0.194). Query-aware selection with only 64 frames, about a seventh of the tokens, sees it 62.5 percent of the time, and with the same 465 frames 97.5 percent. It still fails sometimes, because loosely related frames outscore the moment. So real systems keep a window around each selected frame, cap the frames taken from one scene, and can look again. The exercise builds the selection step with one guard, a minimum gap between selected frames.

import numpy as np

def select_frames(frame_emb, query_emb, k, min_gap):
    scores = np.asarray(frame_emb) @ np.asarray(query_emb)
    return np.argsort(-scores)[:k]

Designing a long-video system

Suppose you must answer questions about 2-hour videos with a 128K-token model. Here is a design that follows the arithmetic, with the alternatives at each step.

1. Index once per video. Decode at 1 frame per second (7,200 frames) and embed each frame with a small image-text encoder. Detect scene changes, and optionally caption each scene and transcribe the audio. This costs encoder compute, not language-model context, and it is reused for every question about the video.
2. Retrieve per question. Score frames and captions against the question. Select candidates with a minimum gap and a cap per scene. Expand each into a short window, because a question such as "what did she do right after pouring" is answered by the frames after the match.
3. Read densely where it matters. Give the language model the selected windows at a higher resolution and frame rate. Add a coarse uniform pass at the lowest resolution, so that it sees the whole story, plus the transcript, all with text timestamps. At the default 66 tokens per frame, 600 coarse frames (one every 12 seconds) cost 39,600 tokens. Sixteen windows of 8 frames at the high 258 tokens cost 33,024. That leaves room for a long transcript.
4. Let it look again. If the answer is uncertain, let the model ask for another window, as agent systems do (Wang et al. 2024d).

The table compares this design with the other strategies in use.

The long-context row rests on one finding. Extending a language model's context alone, with no long-video training, lets a video model take in 2,000 frames, more than 200K visual tokens (Zhang et al. 2024b, a model called LongVA). The authors measured this with a visual needle-in-a-haystack test, which hides one distinct frame in a long video and asks about it, like the text tests of §12.7. Needle tests are easy for video in the same way they are for text, though. One attention head can match a distinct frame, so passing a needle test says little about harder uses of the context, and a model's effective video context is usually shorter than its advertised one.

Streaming video

Everything so far assumed an offline video, one we can index before the question arrives. A live assistant on smart glasses, a sports commentator, or a robot's camera sees a stream instead. Frames arrive without end, a question can come at any moment, and the answer may use only the frames seen so far. Sometimes the model should speak without being asked ("you left the stove on"). Memory is the first problem.

A model that keeps every frame adds $f_{\text{smp}}c_f$ tokens per second to its context. Each token adds its keys and values to the KV cache (§6.5). That is $2Ln_{kv}d_k s$ bytes for $L$ layers, $n_{kv}$ key-value heads of width $d_k$, and $s$ bytes per number, where the 2 counts keys and values. Each new frame's tokens also attend over everything cached, so the work per frame grows linearly with the age of the stream.

In plain words, a streaming model must forget on purpose, because it cannot keep everything. It has four levers.

1. A recent window plus a compressed memory. Keep recent frames in full and compress older ones into a fixed number of memory tokens by pooling, merging, or clustering (Flash-VStream, Zhang et al. 2024c).
2. Evicting from the cache. Drop keys and values directly. One rule keeps a sliding window of recent tokens plus the first few tokens, which absorb much of the attention (Xiao et al. 2023, §12.6). Another keeps the tokens that received the most attention (Zhang et al. 2023b, §25.2). Either way, what a future question will need is unknown when tokens are evicted.
3. Deciding when to speak. After every frame the model either predicts a special token that means "stay silent" or starts a reply, and it is trained on both (VideoLLM-online, Chen et al. 2024b).
4. Causal encoders. Nothing in the visual encoder may look ahead. A temporal module that attends in both directions must wait for future frames before it can encode the present one. So streaming models use per-frame encoders, causal attention over time, or causal operations. One example is the online temporal shift of §29.3 (Lin et al. 2019), which mixes in channels from the previous frame only. Two-frame tubelets (§29.2) add a fixed delay of one frame interval.

Latency is the other constraint. Encoding must keep up with the stream, one frame in at most $1/f_{\text{smp}}$ seconds on average. Answers must arrive while they are still relevant. So streaming systems use few tokens per frame and pipelines whose stages run at the same time. Agents face the same keep-or-forget problem over text, and the chapter on memory treats it there.

Where the frontier stands in October 2026

The frontier models now handle hour-long video, and the open models are close behind. The numbers below come from each lab's own reports, with each lab's own frame budget and settings, so compare them with care.

A near-human score is not yet a reliable one. LVBench's 1,549 questions come from only 103 videos, so its intervals are wide.

Even the frontier has known weak spots, and each has a way to measure it.

• Counting events or objects across a video is weak. The best open models score in the 30s on Molmo2's video counting evaluation.
• Fine temporal order and direction are weak when single frames cannot settle them, as TempCompass and the template-only Something-Something retrieval show.
• Grounding is imprecise at high IoU thresholds, partly because of sampling, as the boundary theorem shows.
• Timestamps can be hallucinated or shifted when the frame rate or the timestamp format differs from training.
• Length still costs accuracy, as Video-MME's long split shows. In long videos models sometimes describe events that never happened, which §31.6 discusses as hallucination in multimodal models.

Two neighboring problems go further. Models that predict what happens next in a video, given actions, are the subject of generative world models. Generating video itself is the subject of §34.5.

In PyTorch

The block below checks the MIL-NCE gradient theorem against autograd with torch.logsumexp. Then it scores a batch of clips by mean-pooled frame embeddings, as in CLIP4Clip, with the symmetric contrastive loss.

To run a real video language model, the Hugging Face interface for Qwen2.5-VL takes a video path, a sampling rate, and a pixel cap per frame, which together set the token bill. This needs a GPU and a model download, so it is shown and not run.

Read more

• Miech et al. (2020). End-to-End Learning of Visual Representations from Uncurated Instructional Videos. MIL-NCE, its derivation, and the ablations on bag size and negatives. The data come from Miech et al. (2019), HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.
• Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval, and Lei et al. (2022), Revealing Single Frame Bias for Video-and-Language Learning. Mean, sequential, and tight similarity on top of CLIP, when temporal parameters pay off, and retrieval tasks that force temporal modeling.
• Cheng et al. (2021). Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss, and Bogolin et al. (2022), Cross Modal Retrieval with Querybank Normalisation. Hubness and the two ways of correcting it. For the encoders that followed CLIP4Clip, see Zhao et al. (2024), VideoPrism, and Wang et al. (2024b), InternVideo2.
• Mangalam et al. (2023). EgoSchema, Fu et al. (2024), Video-MME, and Wang et al. (2024c), LVBench. Temporal certificates, the standard broad benchmark with its length splits and subtitle settings, and an extreme-length benchmark with a human baseline.
• Feng et al. (2025). Video-R1: Reinforcing Video Reasoning in MLLMs, and Li et al. (2025), VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning. T-GRPO with its ordered-versus-shuffled reward, and tIoU as a GRPO reward for grounding.
• Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries and Gao et al. (2017), TALL: Temporal Activity Localization via Language Query. Moment retrieval data, the R@$n$ at IoU and mAP metrics, and Charades-STA.
• Zhang et al. (2024b). Long Context Transfer from Language to Vision, Zhang et al. (2024c), Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams, and Chen et al. (2024b), VideoLLM-online: Online Video Large Language Model for Streaming Video. Long context with the visual needle test, and the two ideas of streaming, fixed memory and learning when to speak.
• Bai et al. (2025b). Qwen3-VL Technical Report, Clark et al. (2026), Molmo2, Google DeepMind (2026), Gemini 4 Argon model card and evaluation note, the Qwen3.5-397B-A17B and Qwen3.8-Flash-Next model cards, and Du et al. (2026), CASE: Cost-Aware Stopping for Efficient Long-Video Agents. The frontier as of October 2026, its frame budgets, and the evaluation settings behind its numbers.

Watch next

▶

▶

▶

▶