Chapter 27 · Section 27.6 · 95 min

Detection, segmentation, and SAM

This section is about vision models that say where things are, not only what an image shows. A classifier reads an image and returns one label. Most real uses of vision need more. A driving system needs a box around every car and pedestrian. A radiology tool needs the outline of a tumor, pixel by pixel. A photo editor needs the exact shape of the object the user just clicked. These are dense prediction tasks, so called because the model makes a prediction at every location of the image rather than one for the whole image. They run most deployed vision systems, and interviews for vision roles probe them in detail.

The input side is already built. The last section ended with backbones that turn an image into feature maps. A feature map is a grid of vectors, and each vector describes one small region of the image. Its stride is the number of pixels that one step on the grid covers. A plain ViT gives one map at stride 16, one vector per 16 by 16 patch. Hierarchical backbones, which shrink the grid stage by stage as a CNN does, give a pyramid of maps at strides 4, 8, 16, and 32. A classifier averages such a map into one vector and throws the layout away. This section keeps the map and puts small networks on it, called heads, that output boxes and masks.

Three things change when the output stops being one label. First, the output becomes a set whose size depends on the image, zero pedestrians or forty. Second, the set has no natural order, so "the third box" means nothing, and a loss must not care which output slot predicted which object. Third, a metric must decide which prediction belongs to which object before it can score anything. That matching step is most of the difficulty of measuring a detector.

We take these in order. First we measure how much two boxes overlap. Then we build the classical detector piece by piece, from fixed reference boxes to the rule that deletes duplicate boxes. Next comes DETR, a Transformer that predicts the whole set of objects at once and needs none of those hand-made pieces. Then we compute COCO average precision, the standard detection score, exactly. Last come segmentation heads, their metrics, and SAM, SAM 2, and SAM 3, models that segment whatever you click on or name.

Four tasks, four kinds of output

The four standard dense tasks differ only in what they output. Picture a street photo with two touching cars, a road, and the sky. Detection draws a scored box around each car. Semantic segmentation colors every pixel by its class, so the two touching cars become one blob of car pixels. Instance segmentation gives each car its own mask. Panoptic segmentation does both at once. Every pixel gets a class, and every car pixel also says which car it belongs to.

📐 The four dense prediction tasks

Fix an image of height $H$ and width $W$ in pixels and a list of $K$ object classes.

Object detection. Output a set of triples $\{(c_k, \mathbf{b}_k, s_k)\}_{k}$ , each a class $c_k\in\{1,\dots,K\}$ , a box $\mathbf{b}_k$ , and a confidence score $s_k\in[0,1]$ . The number of triples is not fixed.
Semantic segmentation. Output one class per pixel, a map in $\{1,\dots,K\}^{H\times W}$ . Two touching cars are one region of "car" pixels.
Instance segmentation. Output a set of triples $\{(c_k, \mathbf{Y}_k, s_k)\}_k$ with a binary mask $\mathbf{Y}_k\in\{0,1\}^{H\times W}$ in place of a box. Masks may overlap, and shapeless regions such as sky and road are usually not predicted at all.
Panoptic segmentation. Assign every pixel a class and, for the countable thing classes (cars, people), an instance id, so the masks partition the image. Each uncountable stuff class (sky, grass) gets one segment (Kirillov et al. 2019).

The standard benchmark for all four tasks is COCO, Common Objects in Context (Lin et al. 2014). It has about 118,000 training images of everyday scenes, labeled with 80 object categories, plus stuff classes for panoptic segmentation.

A fifth kind of dense output is a real number per pixel instead of a class. The main example is monocular depth estimation, the distance from the camera at every pixel, predicted from a single image. Surface normals are another. These tasks use the same backbones and upsampling heads as semantic segmentation, with a regression loss in place of the per-pixel classifier.

A box is four numbers, and three conventions for those numbers are in common use. Mixing them up is one of the most common bugs in detection code, because every mix-up still produces plausible-looking numbers.

✏️ One box, written three ways

Take a 640 by 480 image and a box whose top-left corner is at pixel $(68, 12)$ and bottom-right corner at $(164, 172)$ . Here $x$ grows to the right and $y$ grows downward, as in image arrays.

Corner form, $(x_1, y_1, x_2, y_2) = (68, 12, 164, 172)$ . Torchvision and most loss code use it.
Center form, $(x_c, y_c, w_b, h_b)$ , the center plus the width and height, here $(116, 92, 96, 160)$ . DETR predicts it normalized by the image size, $(116/640, 92/480, 96/640, 160/480) = (0.181, 0.192, 0.150, 0.333)$ , so every number lies in $[0, 1]$ .
COCO form, $(x_1, y_1, w_b, h_b) = (68, 12, 96, 160)$ , the top-left corner plus the size. COCO's annotation files use it.

We write $w_b$ and $h_b$ with subscripts because $h$ is the number of attention heads in this book.

Measuring overlap, IoU and GIoU

To score a predicted box against a true one, we need a number that is 1 when the boxes coincide, 0 when they do not touch, and the same for a small object and a large one. A pixel error is not that number. Five pixels off is terrible for a 10-pixel bird and irrelevant for a 500-pixel bus. The shared area divided by the total area covered has all three properties.

Here it is with numbers. Take two 2 by 2 squares, one with its corner at the origin and one shifted by 1 in both directions. They share a 1 by 1 square. Together they cover $4 + 4 - 1 = 7$ square units, so the ratio is $1/7$ . This ratio is called intersection over union.

📐 Intersection over union and generalized IoU

For two boxes $A$ and $B$ with areas $|A|$ and $|B|$ , let $C$ be the smallest axis-aligned box that contains both. Then

\operatorname{IoU}(A, B) = \frac{|A\cap B|}{|A\cup B|} = \frac{|A\cap B|}{|A| + |B| - |A\cap B|}, \qquad \operatorname{GIoU}(A, B) = \operatorname{IoU}(A, B) - \frac{|C\setminus(A\cup B)|}{|C|}.

In corner form, $|A\cap B| = \max(0, \min(x_2^A, x_2^B) - \max(x_1^A, x_1^B))\cdot\max(0, \min(y_2^A, y_2^B) - \max(y_1^A, y_1^B))$ , and $C = (\min x_1, \min y_1, \max x_2, \max y_2)$ . IoU lies in $[0, 1]$ and GIoU in $(-1, 1]$ (Rezatofighi et al. 2019).

In plain words, IoU is the fraction of the combined area that both boxes cover. GIoU, the generalized IoU, subtracts a penalty, the fraction of the enclosing box that neither box covers. The penalty is large when the boxes are far apart.

The penalty is what makes GIoU useful as a training loss. Suppose a predicted box misses its target entirely. Then $|A\cap B| = 0$ and $|A\cup B| = |A| + |B|$ , so

\operatorname{IoU} = 0, \qquad \operatorname{GIoU} = 0 - \frac{|C| - |A| - |B|}{|C|} = \frac{|A| + |B|}{|C|} - 1 .

Slide the prediction toward the target and the IoU stays at 0 the whole way. Its gradient with respect to the box coordinates is exactly zero, so a loss of $1 - \operatorname{IoU}$ gives a missing prediction no signal at all. The GIoU rises as the enclosing box shrinks, so $1 - \operatorname{GIoU}$ pulls the prediction toward its target. As the boxes move infinitely far apart, $|C|\to\infty$ and the GIoU tends to $-1$ , which is why its range is open at $-1$ .

Both IoU and GIoU are scale invariant. Scale both boxes by the same factor and every area scales by the square of that factor, so the ratios do not change. An L1 distance between coordinates, the sum of the absolute differences, does not. It charges five pixels of error the same on the 10-pixel bird and on the 500-pixel bus. DETR, later in this section, uses both kinds of term in its loss.

✏️ IoU and GIoU by hand

Take $A = (0, 0, 2, 2)$ and $B = (1, 1, 3, 3)$ in corner form. The overlap is the unit square $(1, 1, 2, 2)$ , so $|A\cap B| = 1$ , and $|A\cup B| = 4 + 4 - 1 = 7$ , giving $\operatorname{IoU} = 1/7 \approx 0.143$ . The enclosing box is $C = (0, 0, 3, 3)$ with area 9, of which $9 - 7 = 2$ is covered by neither box, so

\operatorname{GIoU} = \frac17 - \frac29 = \frac{9 - 14}{63} = -\frac{5}{63} \approx -0.079 .

Now move $B$ to $(3, 0, 5, 2)$ , one unit to the right of $A$ . The IoU is 0, $C = (0, 0, 5, 2)$ has area 10, and $\operatorname{GIoU} = 8/10 - 1 = -0.2$ . Move it further, to $(6, 0, 8, 2)$ , and the GIoU falls to $8/16 - 1 = -0.5$ . The IoU cannot tell these two misses apart. The GIoU can.

The vectorized version below compares an $(n, 4)$ array of boxes with an $(m, 4)$ array and returns all $nm$ pairs at once, by broadcasting. Each line is one piece of the definition. The clamp at zero is the line people forget.

Python · cell 1

In practice, IoU is the measure behind every later step of this section. It labels training examples, decides which duplicate boxes to delete, and decides whether a detection counts as correct. An interviewer who asks you to write it expects the clamp, and often asks next why a box loss needs GIoU.

Core

Pairwise IoU and GIoU

Write box_iou(A, B) and box_giou(A, B). A has shape (n, 4) and B has shape (m, 4), both in corner form $(x_1, y_1, x_2, y_2)$ with $x_2 > x_1$ and $y_2 > y_1$ . Each function returns an (n, m) array, with entry $(i, j)$ comparing box $i$ of A with box $j$ of B, and uses no Python loops over boxes. The tests check the worked example, boxes that do not overlap, boxes that are diagonal neighbors (where a missing clamp gives a positive overlap), and random boxes.

import numpy as np

def box_iou(A, B):
    """A: (n, 4), B: (m, 4) boxes as (x1, y1, x2, y2). Return the (n, m) IoU matrix."""
    raise NotImplementedError

def box_giou(A, B):
    """Same inputs. Return the (n, m) generalized IoU matrix."""

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Classical detectors, from anchors to RoIAlign and focal loss

Before 2020, nearly every detector turned detection into a very large number of small questions, each asked at a fixed place in the image. Is there an object near this place? If so, how should a reference box at this place move to fit it? The vocabulary of that design is still everywhere, including inside the detectors that replaced it, and interviewers still ask about every piece.

Anchors and box deltas

An anchor is a fixed reference box placed at a location of the feature map. The network does not predict a box from nothing. For each anchor it predicts a score, "is there an object here", and a small correction that moves the anchor onto the object. The classic two-stage detector, Faster R-CNN (Ren et al. 2015), which we assemble below, places 9 anchors at every position. They are 3 sizes (areas of $128^2$ , $256^2$ , and $512^2$ pixels) times 3 aspect ratios (height over width of 1, 2, and 1/2).

The correction is measured relative to the anchor. Take an anchor 128 pixels wide centered at $x = 100$ , and an object 96 pixels wide centered at $x = 116$ . The network is asked for the shift as a fraction of the anchor's width, $16/128 = 0.125$ , and for the logarithm of the width ratio, $\log(96/128) = -0.288$ . In general, with the superscript $a$ marking the anchor and $(x_c, y_c, w_b, h_b)$ the object's box in center form,

\Delta_x = \frac{x_c - x_c^a}{w_b^a}, \qquad \Delta_y = \frac{y_c - y_c^a}{h_b^a}, \qquad \Delta_w = \log\frac{w_b}{w_b^a}, \qquad \Delta_h = \log\frac{h_b}{h_b^a}.

The vector $\boldsymbol{\Delta} = (\Delta_x, \Delta_y, \Delta_w, \Delta_h)$ , the box deltas, is what the network regresses. Dividing by the anchor's size makes the targets the same for a small anchor and a large one. The logarithm makes doubling and halving equally large errors, since $\log 2 = -\log\tfrac12$ . Inverting each equation gives the decoder that turns predicted deltas back into a box,

x_c = x_c^a + \Delta_x w_b^a, \qquad y_c = y_c^a + \Delta_y h_b^a, \qquad w_b = w_b^a e^{\Delta_w}, \qquad h_b = h_b^a e^{\Delta_h},

and the exponential keeps every decoded width and height positive.

✏️ Box deltas by hand

An anchor of size $128\times128$ centered at $(100, 100)$ has corners $(36, 36, 164, 164)$ . The object has corners $(68, 12, 164, 172)$ , so its center is $(116, 92)$ and its size is $96\times160$ . Then

\Delta_x = \frac{116 - 100}{128} = 0.125, \quad \Delta_y = \frac{92 - 100}{128} = -0.0625, \quad \Delta_w = \log\frac{96}{128} = -0.288, \quad \Delta_h = \log\frac{160}{128} = 0.223 .

Decoding reverses it. The center is $100 + 0.125\cdot128 = 116$ and the width is $128\,e^{\log 0.75} = 96$ .

🔬 What production box coders add

Real code adds three details. Torchvision's Faster R-CNN multiplies the second-stage targets by the weights $(10, 10, 5, 5)$ , the reciprocals of typical standard deviations of the deltas $(0.1, 0.1, 0.2, 0.2)$ , so the targets have roughly unit spread, and it divides by the weights again when decoding (Fast R-CNN normalized its targets the same way). The regression loss is smooth L1, which is quadratic for small errors and linear beyond a small constant, so its gradient shrinks to zero as the error vanishes and never grows past a fixed size for outliers. And the decoder clamps $\Delta_w$ and $\Delta_h$ at $\log(1000/16)\approx4.1$ before exponentiating, so one wild prediction cannot overflow.

Which anchors count as positives is decided by IoU. In Faster R-CNN's first stage, an anchor whose IoU with some object exceeds 0.7 is a positive. So is the anchor with the highest IoU for each object, so that no object is left without one. An anchor whose IoU with every object is below 0.3 is a negative, and the rest are ignored in training.

Detectors split into two kinds by what they do with these scores. A two-stage detector uses them only to propose regions, then looks at the best proposals again and classifies them, as we will see below. A one-stage detector classifies every anchor directly into the $K$ classes. RetinaNet (Lin et al. 2017b), the one-stage detector we meet with focal loss, labels anchors with IoU at least 0.5 as positives and below 0.4 as negatives, and ignores the band between.

In an interview, expect to write the encoder and decoder, and to say why the width delta is a logarithm.

Warm-up

Encoding and decoding box deltas

Write encode_deltas(boxes, anchors) and decode_deltas(deltas, anchors). Both boxes and anchors have shape (n, 4) in corner form $(x_1, y_1, x_2, y_2)$ , and row $i$ of boxes is the target for row $i$ of anchors. Widths are $x_2 - x_1$ and heights $y_2 - y_1$ . encode_deltas returns the (n, 4) array of deltas $(\Delta_x, \Delta_y, \Delta_w, \Delta_h)$ defined above. decode_deltas inverts it and returns corner boxes. The tests check the worked example, an exact round trip in both directions, and the usual slips (corners used as centers, a missing logarithm, dividing by the object's size instead of the anchor's).

import numpy as np

def encode_deltas(boxes, anchors):
    """boxes, anchors: (n, 4) corners (x1, y1, x2, y2). Return (n, 4) deltas (dx, dy, dw, dh)."""
    raise NotImplementedError

def decode_deltas(deltas, anchors):
    """deltas: (n, 4), anchors: (n, 4) corners. Return the (n, 4) decoded corner boxes."""

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Feature pyramids

Objects in one image range from a few pixels to the whole frame. So small anchors should read fine feature maps, where each vector covers a few pixels, and large anchors should read coarse ones. A CNN backbone already has maps at several strides. But the fine maps come from early layers, which know little about what objects are. The feature pyramid network, FPN (Lin et al. 2017a), passes that knowledge down from the coarse maps.

It works from the top down. Start from the coarsest map, at stride 32. Upsample it by 2 with nearest-neighbor interpolation, which copies each vector into a 2 by 2 block. Add a $1\times1$ convolution of the backbone's next finer map, called the lateral connection. Repeat down to stride 4, and smooth each sum with a $3\times3$ convolution. Every level has the same width, 256 channels, so one shared head can run on all of them. Each level now combines the fine resolution of its own stage with what the deepest stage knows about objects.

A proposed region, called a region of interest, reads one level, chosen by its size. FPN numbers its levels so that level $k$ has stride $2^k$ . It sends a region of width $w_b$ and height $h_b$ to level

k = \Big\lfloor 4 + \log_2\frac{\sqrt{w_bh_b}}{224}\Big\rfloor ,

clamped to the levels that exist. A region 224 pixels on a side reads level 4, stride 16. A 112-pixel region reads level 3, stride 8, and a 448-pixel region reads level 5, stride 32. Each halving of the side moves one level finer, so every region that the clamp does not touch spans between 14 and 28 cells on a side of the grid it reads.

In practice, the pyramid is what lets one detector find both a distant pedestrian and a nearby bus. Its gains are largest on small objects. An interviewer may ask why the fine levels need the top-down path at all, and the answer is that early layers see edges and textures but not yet objects.

A plain ViT has only one map, at stride 16. §27.5 showed how to build a pyramid from that one map anyway, the simple feature pyramid of ViTDet (Li et al. 2022a). Max pooling gives stride 32. Transposed convolutions, the learned upsampling of §0.8, give strides 8 and 4. Combined with masked-autoencoder (MAE) pretraining, which teaches a ViT to fill in hidden patches, that made plain ViTs competitive backbones for dense tasks again.

Non-maximum suppression

A dense detector fires many overlapping boxes on each object, because every anchor near the object learns to say "object". Non-maximum suppression, NMS, keeps the best box of each cluster and deletes the rest. It is greedy. Keep the highest-scoring box, delete everything that overlaps it too much, and repeat with what is left.

NMS is a heuristic, a hand-set rule rather than something the network learns, and it has two weaknesses. It has one global threshold, so it cannot tell two detections of one object from two detections of two overlapping objects. And it is not differentiable, so the network cannot learn to avoid the duplicates it creates. The cell below builds a small crowd. Two people stand close together, with true boxes that overlap at IoU 0.71, and a third person stands apart. Each person gets four jittered detections, and we run NMS at several thresholds.

Python · cell 2

At 0.3 and 0.5, the best box of the first person suppresses every box of the second, because the two people overlap more than the threshold allows. At 0.7 all three people survive with no duplicates, and at 0.8 and above duplicates leak through. No single threshold is right for every scene, and in a dense crowd the window where one works can vanish.

Soft-NMS (Bodla et al. 2017) lowers the scores of overlapping boxes instead of deleting them. Its linear form multiplies the score of a box $\mathbf{b}_i$ by $1 - \operatorname{IoU}(\mathbf{b}_i, \mathbf{b}_{\text{kept}})$ whenever that IoU exceeds $t$ . Its Gaussian form multiplies every score by $\exp(-\operatorname{IoU}^2/\sigma)$ for a width $\sigma$ . The second person survives with a lower score instead of vanishing. That helps in crowds, but the heuristic remains. DETR, below, removes it.

The widget puts two people side by side and lets you push them together. The strip beside the picture marks every threshold that finds both people with no duplicate, and from an overlap of 0.7 on there is none. Switching to soft-NMS replaces deletion by the linear score decay, and a box counts as kept while its score stays at least 0.3. It finds both people up to an overlap of about 0.68.

Interview

Non-maximum suppression, class-aware

Write nms(boxes, scores, iou_thresh, classes=None) for corner boxes of shape (n, 4) and scores of shape (n,). Return the indices of the kept boxes as an integer array, highest score first. Suppress a box only if its IoU with a kept box is strictly greater than iou_thresh, the rule torchvision uses. When classes is given, boxes of different classes never suppress each other. Break ties in score by input order, with a stable sort. Coordinates may be negative, as they are for decoded boxes before clipping, and an empty input returns an empty array.

import numpy as np

def nms(boxes, scores, iou_thresh, classes=None):
    """boxes (n, 4) corners, scores (n,), classes (n,) ints or None.
    Return the indices of the kept boxes, highest score first."""
    return np.argsort(-np.asarray(scores))          # placeholder that keeps everything

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Two-stage detectors and RoIAlign

A two-stage detector splits detection into two questions. Where might an object be? And what exactly is it? Faster R-CNN (Ren et al. 2015) answers both from one shared backbone feature map.

Stage one, the region proposal network (RPN). A $3\times3$ convolution slides over the feature map. Two $1\times1$ convolutions then predict, for each of the 9 anchors at each position, an objectness score (object or background, with no class) and four deltas. A $1000\times600$ image at stride 16 has about $60\times40\times9\approx20{,}000$ anchors. At inference the deltas are decoded, the boxes are clipped to the image, and NMS at IoU 0.7 thins them to about 2,000 proposals. The top 300 go to the second stage at test time.

Stage two, the RoI head. Each proposal is a region of interest, or RoI. For each one, the head extracts a small fixed-size grid of features, $7\times7$ for the box head, runs a few layers on it, and predicts $K + 1$ class scores (the $K$ classes plus background) and a box correction for each class. Class-wise NMS then removes duplicates among the refined boxes.

Two-stage detectors handle class imbalance by filtering. The RPN discards almost all the background before the second-stage classifier sees it. One-stage detectors have no such filter, and focal loss, in the next subsection, does that job for them.

🔬 How the two stages are trained

The RPN samples 256 anchors per image, up to half of them positive, and applies a two-class log loss to all of them and a box loss to the positives. During training, the top 2,000 proposals feed the second stage. At test time Faster R-CNN keeps 300, and Mask R-CNN with an FPN keeps 1,000. With an FPN the RPN runs on every level with one anchor size per level, 5 sizes and 3 aspect ratios in total, and each proposal later reads from the level chosen by the formula above. The RoI head samples one positive (IoU at least 0.5 with some object) for every three negatives, 512 RoIs per image with an FPN. Cascade R-CNN (Cai and Vasconcelos 2018) repeats the RoI head three times. Each stage refines the previous stage's boxes and trains with a stricter positive threshold (IoU 0.5, then 0.6, then 0.7), which mostly raises AP at high IoU thresholds.

The interesting part is how the head extracts features for a proposal. A proposal is a box in continuous image coordinates, and the feature map is a grid of cells at stride $s$ . The first method, RoIPool from Fast R-CNN, rounds twice. Take a proposal whose left edge is at $x = 57$ pixels, on a stride-16 map. Its grid coordinate is $57/16 = 3.56$ , and RoIPool rounds it to cell 4, which starts at pixel 64, 7 pixels away. Then it splits the rounded box into $7\times7$ bins, rounds the bin edges again (down for the start, up for the end), and takes the maximum over the cells of each bin.

The first rounding moves an edge by up to half a cell, 8 pixels at stride 16, and the second adds up to another cell. A classifier barely notices, since a few pixels of shift do not change what the object is. A mask does notice. A mask predicted from shifted features lands shifted on the image. Mask R-CNN (He et al. 2017), which extends Faster R-CNN to predict a mask for every region, removed the rounding with RoIAlign.

📐 RoIAlign (He et al. 2017)

Let $\mathbf{F}\in\R^{C\times H\times W}$ be a feature map at stride $s$ , whose spatial scale is $1/s$ , and let $(x_1, y_1, x_2, y_2)$ be a box in image pixels. Map it to the grid with no rounding, $\tilde x_k = x_k/s - o$ and $\tilde y_k = y_k/s - o$ for $k\in\{1,2\}$ , where the offset $o$ is $\tfrac12$ (the aligned convention, explained below) or 0 (the original). Split the box into $p_h\times p_w$ bins of size $\delta_h = (\tilde y_2 - \tilde y_1)/p_h$ by $\delta_w = (\tilde x_2 - \tilde x_1)/p_w$ . In bin $(i, j)$ , place an $n_s\times n_s$ grid of sample points, with $n_s$ the sampling ratio, at

y_{ia} = \tilde y_1 + \Big(i + \frac{a + \frac12}{n_s}\Big)\delta_h, \qquad x_{jc} = \tilde x_1 + \Big(j + \frac{c + \frac12}{n_s}\Big)\delta_w, \qquad a, c\in\{0,\dots,n_s-1\},

read $\mathbf{F}$ at each point by bilinear interpolation, and average the $n_s^2$ values. Here $\mathbf{F}(y, x)$ , at a point between cells, is the weighted average of the four nearest cells, with weights that fall off linearly with distance. The box below gives the exact formula and torchvision's rule at the border.

🔬 Bilinear interpolation and the border rule

Bilinear interpolation at $(y, x)$ , with $y_0 = \lfloor y\rfloor$ , $x_0 = \lfloor x\rfloor$ , and fractional parts $\phi_y = y - y_0$ and $\phi_x = x - x_0$ , is

\mathbf{F}(y, x) = (1-\phi_y)(1-\phi_x)\mathbf{F}_{:, y_0, x_0} + (1-\phi_y)\phi_x\mathbf{F}_{:, y_0, x_0+1} + \phi_y(1-\phi_x)\mathbf{F}_{:, y_0+1, x_0} + \phi_y\phi_x\mathbf{F}_{:, y_0+1, x_0+1},

a vector of $C$ channels. Rows are indexed by $y$ and columns by $x$ . At the border, torchvision's rule is that a point with $y < -1$ , $y > H$ , $x < -1$ , or $x > W$ reads 0, and any other point is first clamped into $[0, H-1]\times[0, W-1]$ , with the $+1$ neighbors capped at the last row and column.

In plain words, RoIAlign treats the feature map as a continuous function. It reads that function at evenly spaced points inside each bin and averages them. When a box moves by a tenth of a pixel, its sample points move by a tenth of a pixel too, and the features change smoothly instead of jumping by a whole cell. Mask R-CNN uses $n_s = 2$ , four points per bin, and found that taking the max or the average over those points scores almost the same (30.2 and 30.3 mask AP).

Here is a tiny case. A $7\times7$ grid of bins over a box 80 pixels wide on a stride-16 map makes each bin $80/16/7 = 0.71$ cells wide. RoIPool must round that to whole cells, so some bins read one cell and others two. RoIAlign samples each bin at two points 0.36 cells apart and reads between cells by interpolation.

The offset of one half is a classic interview probe. Feature cell $j$ summarizes the patch of the image centered at pixel $(j + \tfrac12)s$ . So image coordinate $x$ corresponds to grid coordinate $x/s - \tfrac12$ , the convention under which index $j$ means "the center of cell $j$ ". The original Mask R-CNN code left out the $-\tfrac12$ and shifted every feature by half a cell. The detection libraries Detectron2 and torchvision fixed this with the flag aligned=True and kept aligned=False for the old behavior.

The cell measures how far RoIPool's bins land from where they should be, along one axis. It compares the center of the cells each bin reads with the exact center of the bin.

Python · cell 3

On the 80-pixel box, the RoIPool bins sit 6 to 24 pixels to the right of where they belong. Over random boxes the error averages about half a stride and reaches a full stride at the 95th percentile, 17 pixels at stride 16 and 35 at stride 32. RoIAlign with aligned=True has no such error, because its sample points sit symmetrically inside each exact bin. With aligned=False, every bin is off by exactly half a cell, 8 pixels at stride 16. For a 50-pixel object, a 17-pixel shift is a third of its width.

With aligned features in hand, Mask R-CNN adds a third branch to the RoI head. It predicts a $28\times28$ mask for each region from a $14\times14$ RoIAlign grid. It predicts one mask per class, each through a per-pixel sigmoid, and the loss uses only the mask of the region's true class. The total loss per RoI is the sum of three terms, $\mathcal{L}_{\text{cls}} + \mathcal{L}_{\text{box}} + \mathcal{L}_{\text{mask}}$ , for the class, the box, and the mask.

🔬 Mask R-CNN's mask branch in detail

The branch applies four $3\times3$ convolutions of width 256 to the $14\times14$ grid, a stride-2 transposed convolution to $28\times28$ , and a $1\times1$ convolution to $K$ channels, one $28\times28$ mask per class. The mask loss is the average binary cross-entropy over the $28\times28$ pixels of the true class's mask, computed on positive RoIs against the ground-truth mask cropped to the RoI and resized to $28\times28$ . Because the class branch already decides the class, the masks of different classes do not compete. Replacing the per-class sigmoids by a softmax over classes costs 5.5 mask AP. At test time, the mask branch runs on the 100 highest-scoring boxes, and the mask of the predicted class is resized to the box and thresholded at 0.5.

Interview

RoIAlign from scratch

Write roi_align(feat, box, out_size, spatial_scale, sampling_ratio=2, aligned=True) exactly as defined above. feat has shape (C, H, W), and box is one box $(x_1, y_1, x_2, y_2)$ in image pixels. out_size is an integer or a pair (p_h, p_w), and the result has shape (C, p_h, p_w). Use the offset $o = \tfrac12$ when aligned is true and 0 otherwise. In each bin, sample sampling_ratio points along each axis at the centers of a regular subgrid, interpolate bilinearly with torchvision's border rule, and average. The tests compare against a reference loop that matches torchvision.ops.roi_align. They catch a rounded box (RoIPool's quantization), a wrong half-pixel offset, swapped $x$ and $y$ , a missing spatial scale, and sample points at the corners of the subgrid instead of its centers. Every test box spans at least one feature cell.

import numpy as np

def roi_align(feat, box, out_size, spatial_scale, sampling_ratio=2, aligned=True):
    """feat (C, H, W), box (x1, y1, x2, y2) in image pixels. Return (C, p_h, p_w)."""
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Focal loss, and why easy negatives matter

A one-stage detector faces an extreme class imbalance. RetinaNet scores about 100,000 anchors per image, and only a tiny fraction of them overlap an object. Almost every anchor is an easy negative, background that the network already rejects with high confidence.

Recall from §0.5 that cross-entropy, the standard classification loss, charges minus the logarithm of the probability the model gave to the right answer. An easy negative that the model calls background with probability 0.99 costs $-\log 0.99 = 0.010$ . That is tiny, but 99,900 of them add up to about 1,000. A hundred hard positives at probability 0.5 cost $100\cdot\log 2 = 69$ . The easy negatives carry 93.5 percent of the loss. They carry 95 percent of its gradient too, since each sends back a gradient of $1 - p_t$ , 0.01 against 0.5 for a hard positive. So training spends its effort on background it has already learned.

The fix is to shrink the loss of examples the model already gets right. To write it, take one anchor and one class, with label $y\in\{0, 1\}$ , logit $x$ , and predicted probability $p = \sigma(x)$ , where $\sigma$ is the sigmoid. Define $p_t = p$ if $y = 1$ and $p_t = 1 - p$ otherwise, the probability assigned to the correct answer. The subscript $t$ stands for "true", as in the focal loss paper, and is not the IoU threshold. Cross-entropy is then $-\log p_t$ .

In plain words, focal loss is cross-entropy multiplied by $(1 - p_t)^{\gamma_f}$ , a factor near 1 for a hard example and near 0 for an easy one. With $\gamma_f = 2$ , an example at $p_t = 0.9$ has its loss cut 100 times, and one at $p_t \approx 0.968$ about 1,000 times. An example at $p_t\le0.5$ loses at most a factor of 4. The bias initialization serves the same goal from the first step. If every anchor started at probability 0.5, the 100,000 background anchors would produce a huge loss and an unstable first update.

What matters for training is the gradient, the signal each example sends back to the network. Here is its exact form.

∴ The focal loss gradient

Let $x_t$ be the logit of the correct answer ( $x_t = x$ for a positive and $-x$ for a negative, since $1 - \sigma(x) = \sigma(-x)$ ), so $p_t = \sigma(x_t)$ and $\mathrm{d}p_t/\mathrm{d}x_t = p_t(1 - p_t)$ . Dropping the constant $\alpha_t$ ,

\frac{\mathrm{d}\operatorname{FL}}{\mathrm{d}x_t} = (1 - p_t)^{\gamma_f}\big(\gamma_f\,p_t\log p_t - (1 - p_t)\big), \qquad \frac{\mathrm{d}\operatorname{FL}/\mathrm{d}x_t}{\mathrm{d}\operatorname{CE}/\mathrm{d}x_t} = (1 - p_t)^{\gamma_f}\Big(1 + \gamma_f\,\frac{p_t\,(-\log p_t)}{1 - p_t}\Big)\xrightarrow[p_t\to1]{}(1+\gamma_f)(1 - p_t)^{\gamma_f}.

Here $\operatorname{CE} = -\log p_t$ is the cross-entropy. The derivation below proves both formulas.

∂ Proof of the focal loss gradient

By the product rule,

\frac{\mathrm{d}}{\mathrm{d}x_t}\Big[-(1 - p_t)^{\gamma_f}\log p_t\Big] = \gamma_f(1 - p_t)^{\gamma_f - 1}\,\frac{\mathrm{d}p_t}{\mathrm{d}x_t}\,\log p_t - (1 - p_t)^{\gamma_f}\,\frac{1}{p_t}\frac{\mathrm{d}p_t}{\mathrm{d}x_t}.

Substituting $\mathrm{d}p_t/\mathrm{d}x_t = p_t(1 - p_t)$ gives $\gamma_f(1 - p_t)^{\gamma_f}p_t\log p_t - (1 - p_t)^{\gamma_f + 1}$ , which factors as stated. Setting $\gamma_f = 0$ gives the cross-entropy gradient $-(1 - p_t)$ , and dividing gives the ratio. As $p_t\to1$ , $-\log p_t = (1 - p_t) + O((1 - p_t)^2)$ , so the fraction tends to 1 and the ratio to $(1 + \gamma_f)(1 - p_t)^{\gamma_f}$ . $\blacksquare$

In plain words, compared with cross-entropy, focal loss shrinks the gradient of an easy example by roughly $(1 + \gamma_f)(1 - p_t)^{\gamma_f}$ . With $\gamma_f = 2$ , an anchor the network already gets right at 99 percent contributes about 3,300 times less gradient, while a misclassified anchor keeps most of its gradient. The cell checks the formula against a finite difference. Then it splits the loss of one image into its easy-negative and hard-positive parts.

Python · cell 4

Under cross-entropy, the 99,900 easy negatives carry 93.5 percent of the loss of this image. Under focal loss they carry 1.7 percent, and the hundred hard positives get the attention. With focal loss, the one-stage RetinaNet beat the two-stage detectors of its time, reaching 39.1 AP on COCO test-dev with a ResNet-101-FPN backbone (Lin et al. 2017b). Focal loss outlived anchors. DETR's descendants use it for their class heads, and SAM uses it for its masks.

The plot shows the mechanism. The left panel is the loss and the right panel the size of its gradient with respect to the logit, both against $p_t$ , for four values of $\gamma_f$ . The gradient axis is logarithmic, so equal vertical gaps are equal factors.

Python · cell 5

In the shaded band of easy examples, the curves separate by orders of magnitude, about $(1 + \gamma_f)(1 - p_t)^{\gamma_f}$ as derived. Below $p_t = 0.5$ they stay within a small factor of each other. A larger $\gamma_f$ silences easy examples more, but it also starts to mute moderately hard ones. That is why $\gamma_f = 2$ won the paper's ablation, with 34.0 AP against 32.2 at $\gamma_f = 5$ and 31.1 for cross-entropy with the best $\alpha_f$ . An interviewer will ask what $\gamma_f$ does, why the bias starts at $-4.6$ , and how a two-stage detector solves the same problem without focal loss.

Anchor-free heads and label assignment

Anchors bring their own hyperparameters (sizes, ratios, IoU thresholds), and most anchors are wasted. An anchor-free detector drops them. In FCOS (Tian et al. 2019), every location of every pyramid level is mapped back to an image point $(x, y)$ . A point that falls inside a ground-truth box $(x_1, y_1, x_2, y_2)$ is a candidate positive. Instead of deltas from an anchor, the head regresses the distances from the point to the four sides of the box,

l^* = x - x_1, \qquad t^* = y - y_1, \qquad r^* = x_2 - x, \qquad b^* = y_2 - y,

all positive for a point inside the box. Each object goes to one pyramid level by its size, small objects to fine levels and large ones to coarse levels.

Points near a box's edge produce poor boxes. So a one-layer branch predicts center-ness,

\text{centerness}^* = \sqrt{\frac{\min(l^*, r^*)}{\max(l^*, r^*)}\cdot\frac{\min(t^*, b^*)}{\max(t^*, b^*)}}\in[0, 1].

In plain words, center-ness is 1 at the exact center of a box and falls toward 0 at an edge. A point at distances $(l^*, t^*, r^*, b^*) = (10, 30, 30, 10)$ scores $\sqrt{\tfrac13\cdot\tfrac13} = \tfrac13$ . The branch is trained with binary cross-entropy, and its output multiplies the class score at test time, so NMS ranks central boxes first. In the FCOS ablation it raised COCO AP from 33.5 to 37.1. Classification uses focal loss, so FCOS is a RetinaNet without anchors.

The lesson interviewers look for is that anchors were never the essential part. What decides accuracy is label assignment, the rule that picks which anchors or points are positives for each object.

🔬 Label assignment matters more than anchors

FCOS sends an object to a level by $\max(l^*, t^*, r^*, b^*)$ , from $[0, 64]$ pixels on stride 8 up to above 512 on stride 128, and a point inside two boxes on the same level takes the smaller box. Computing center-ness from the predicted distances instead of learning it as a separate output gives no gain (33.5 AP in both cases), so it must be learned. CenterNet (Zhou et al. 2019) goes further and treats each object as a single point. It predicts a heatmap of box centers for each class, plus a size and an offset at each peak, and finds the peaks with a $3\times3$ max-pool in place of NMS.

Zhang et al. (2020) showed that RetinaNet and FCOS reach nearly the same accuracy once they use the same positives and negatives. Their method, ATSS, adapts the IoU threshold to each object. For each object it takes the $k = 9$ anchors per level whose centers are closest to the object's center. It keeps those whose IoU exceeds the mean plus one standard deviation of these candidates' IoUs and whose centers lie inside the box. Recent YOLO detectors use task-aligned assignment from TOOD (Feng et al. 2021). It ranks the candidates for each object by the product of two powers, the predicted class score to a small power and the IoU of the predicted box to a larger one (powers 1 and 6 in TOOD), and takes the top few. The positives are then the predictions that are already good at both tasks.

DETR, detection as set prediction

The pipeline above has many hand-designed parts, the anchor sizes, the IoU thresholds that label anchors, and the NMS threshold. Carion et al. (2020) asked whether a Transformer could output the set of objects directly, so that none of these parts would be needed. Their model, DETR (DEtection TRansformer), is short to describe.

A ResNet-50 backbone maps the image to a feature map of $C = 2048$ channels at stride 32. A $1\times1$ convolution reduces it to width $d = 256$ , and the map is flattened into a sequence of tokens, one per grid location. A standard Transformer encoder with 6 layers and 8 heads mixes those tokens. It knows where each token sits through fixed 2-D sine-cosine position encodings, the idea of §27.1 applied to a CNN's grid, with half of the $d$ channels encoding the row and half the column.

The decoder produces the objects. Recall from §4.6 that in cross-attention a set of query vectors reads from another sequence, and that a model can learn its own query vectors, one per output slot. DETR's decoder has $N_q = 100$ such slots, whose learned vectors are called object queries. Over 6 layers, each slot reads the image through cross-attention and talks to the other slots through self-attention. It ends as a description of one object, or of nothing.

A shared head turns each of the $N_q$ outputs into two things. One is a distribution over $K + 1$ classes, the $K$ real ones plus a "no object" class written $\varnothing$ . The other is a box in normalized center form, from a 3-layer MLP and a sigmoid. Every query outputs a box. Most of them should say $\varnothing$ .

Here is one real image followed through every stage, with its shapes.

✏️ One image through DETR, shape by shape

Take the 640 by 480 photo of the box example at the start of the section, and a DETR with a ResNet-50 backbone trained on COCO.

Resize the image so its shorter side is 800 pixels. The input is $3\times800\times1066$ .
The backbone halves the resolution five times, to $2048\times25\times34$ , one vector per 32 by 32 pixel region.
A $1\times1$ convolution cuts the width to 256. Flattening gives 850 tokens of width 256.
Six encoder layers of self-attention mix the 850 tokens. Each head scores $850\times850 = 722{,}500$ pairs.
Six decoder layers update 100 object queries of width 256. Each cross-attention head scores $100\times850 = 85{,}000$ query-token pairs.
The heads give 100 rows of 92 class logits and 100 boxes. The 92 are 91 slots for COCO's category ids, which run up to 90 with gaps, plus no object.
To read the answer, take the softmax of each row and drop the no-object column. Keep the queries whose best class beats a threshold such as 0.7. No NMS runs.

Suppose query 17 says "cat" with probability 0.93 and box $(0.181, 0.192, 0.150, 0.333)$ . Multiplying by the original width and height gives center $(116, 92)$ and size $96\times160$ . That is the corner box $(68, 12, 164, 172)$ from the start of the section.

The cell checks the shapes and shows why DETR reads only the stride-32 map. Attention scores every pair of tokens, so its cost grows with the square of the token count.

Python · cell 6

At stride 32 an attention map per head takes about 3 MB. At stride 8 the same image has 13,400 tokens, and every head of every encoder layer would score 180 million pairs, about 720 MB in 32-bit floats. That is 248 times the stride-32 cost, which is why DETR gave up the fine map and, with it, much of its accuracy on small objects.

🔬 Where DETR puts its position information

DETR does not add the encoder's sine-cosine positions once to the input. It adds them to the queries and keys of every attention layer, while the values carry no position. The decoder's input is all zeros. What makes the 100 slots different is the object queries, which act as positional encodings for the slots. They are added to the queries and keys of every decoder self-attention layer and to the queries of every cross-attention layer. In cross-attention, the keys are the encoder's outputs plus the spatial encodings and the values are the encoder's outputs alone. Layer by layer, the zeros turn into object descriptions. So the learned vectors steer where each slot looks rather than being its content, and Carion et al. show that each slot learns to specialize in a few regions and box sizes. DAB-DETR, below, made this reading explicit by replacing each query with a box whose coordinates are refined at every layer.

The hard part is the loss. The model outputs 100 predictions in a fixed order of slots. The image contains $m$ objects in no order at all, on COCO usually a handful and rarely more than a few dozen. Nothing says which slot should predict which object. DETR's answer is to find, for each training image, the one-to-one assignment of predictions to objects that fits best. Then it trains each prediction toward the object it was assigned.

A tiny case shows why this works. Suppose two queries both sit on the same cat, one with a tight box at 80 percent confidence and one with a looser box at 60 percent. The best assignment gives the cat to the first query. The second has no object left, so it is trained to say "no object". The loss itself punishes the duplicate, so the model learns to output one box per object.

📐 DETR's set prediction loss (Carion et al. 2020, Eq. 1 and 2)

Let the predictions be $\hat y_i = (\hat p_i, \hat{\mathbf{b}}_i)$ for $i = 1,\dots,N_q$ , where $\hat p_i$ is a distribution over the $K + 1$ classes and $\hat{\mathbf{b}}_i\in[0,1]^4$ a box. Let the ground truth be $y_j = (c_j, \mathbf{b}_j)$ for $j = 1,\dots,m$ with $m\le N_q$ , padded to $N_q$ entries with $y_j = (\varnothing, \cdot)$ for $j > m$ . For a permutation $\pi$ of $\{1,\dots,N_q\}$ , the matching cost of the pair $(y_j, \hat y_{\pi(j)})$ is

\mathcal{L}_{\text{match}}(y_j, \hat y_{\pi(j)}) = \mathbb{1}[c_j\ne\varnothing]\Big(-\hat p_{\pi(j)}(c_j) + \lambda_{L1}\big\lVert\mathbf{b}_j - \hat{\mathbf{b}}_{\pi(j)}\big\rVert_1 + \lambda_{\text{giou}}\big(1 - \operatorname{GIoU}(\mathbf{b}_j, \hat{\mathbf{b}}_{\pi(j)})\big)\Big),

with GIoU computed on corner forms. The optimal matching and the Hungarian loss are

\hat\pi = \arg\min_{\pi}\sum_{j=1}^{N_q}\mathcal{L}_{\text{match}}(y_j, \hat y_{\pi(j)}), \qquad \mathcal{L}_{\text{Hungarian}} = \sum_{j=1}^{N_q}\Big[-w_{c_j}\log\hat p_{\hat\pi(j)}(c_j) + \mathbb{1}[c_j\ne\varnothing]\,\mathcal{L}_{\text{box}}(\mathbf{b}_j, \hat{\mathbf{b}}_{\hat\pi(j)})\Big],

where $\mathcal{L}_{\text{box}}$ is the same $\lambda_{L1}$ and $\lambda_{\text{giou}}$ combination, and $w_\varnothing = 0.1$ while $w_c = 1$ for real classes. DETR uses $\lambda_{L1} = 5$ and $\lambda_{\text{giou}} = 2$ . In the released code, the class term is divided by the sum of the weights $w$ and the box terms by the number of objects $m$ .

In plain words, first find the cheapest one-to-one assignment of predictions to objects, under a cost that mixes class probability and box distance. Then apply ordinary classification and box losses to the matched pairs, and train every unmatched prediction to say "no object".

Four details in that definition are deliberate, and interviewers ask about each.

Probabilities in the matching, log-probabilities in the loss. The matching cost uses $-\hat p$ , which lies in $[-1, 0]$ , the same order of size as the box terms. A $-\log\hat p$ term would be unbounded, and one confident class score could override the geometry. Carion et al. report that probabilities match better. The loss itself uses $-\log\hat p$ because that is cross-entropy.
The constant in the GIoU term. The paper writes the box cost as $1 - \operatorname{GIoU}$ and the code as $-\operatorname{GIoU}$ . Every matching pairs exactly $m$ objects, so the two costs differ by the constant $m\lambda_{\text{giou}}$ for every $\pi$ and have the same minimizer.
The weight 0.1 on $\varnothing$ . With 100 queries and a handful of objects, about 95 percent of the class terms are "no object". Down-weighting them by 10 is the simplest fix for that class imbalance, a fixed weight where focal loss uses one that depends on how easy each example is. Carion et al. compare it to the way Faster R-CNN subsamples background proposals. The padded entries cost nothing to match, since their cost does not depend on the prediction.
Auxiliary losses. The same prediction head and the same Hungarian loss are applied after every one of the 6 decoder layers, which helps the decoder output the right number of objects. At inference only the last layer is used.

∂ Why the Hungarian loss ignores the order of the queries

(1) Let $\rho$ be a permutation and define the permuted predictions $\hat y'_i = \hat y_{\rho(i)}$ . A matching $\pi$ of the permuted list pairs $y_j$ with $\hat y'_{\pi(j)} = \hat y_{\rho(\pi(j))}$ , so its cost equals the cost of the matching $\rho\circ\pi$ of the original list. The map $\pi\mapsto\rho\circ\pi$ is a bijection of the set of all permutations onto itself (its inverse is $\pi\mapsto\rho^{-1}\circ\pi$ ), so the two minimizations range over the same set of costs and have the same minimum. If the original minimizer $\hat\pi$ is unique, the permuted minimizer is $\rho^{-1}\circ\hat\pi$ , which pairs every object with the same prediction as before, so every term of the loss is unchanged. (2) Minimizing $\sum_j \mathbf{C}_{j,\pi(j)}$ over permutations for a square cost matrix $\mathbf{C}$ is the linear assignment problem, which the Hungarian method of Kuhn (1955), in its later $O(n^3)$ form, solves exactly. Here $n = N_q$ . $\blacksquare$

In plain words, shuffling the queries cannot change the loss, and the best matching takes one cubic-time solve per image, not a search over all $N_q!$ orderings. The padded columns make the square problem equivalent to a rectangular one. Choose $m$ distinct predictions for the $m$ real objects at minimum total cost, and send the remaining $N_q - m$ predictions to $\varnothing$ . scipy.optimize.linear_sum_assignment solves the rectangular problem directly, with a shortest augmenting path algorithm. For $N_q = 100$ and a few dozen objects it takes well under a millisecond on a CPU, which is why DETR runs it on the CPU for every image of every batch.

✏️ A three-query matching by hand

Take two objects, a cat $A$ at $(0.30, 0.40, 0.20, 0.30)$ and a dog $B$ at $(0.70, 0.55, 0.30, 0.40)$ , in normalized center form, and three queries. Query $q_0$ predicts $(0.31, 0.41, 0.20, 0.28)$ with probabilities 0.80 cat, 0.05 dog, 0.15 no object. Query $q_1$ is a near duplicate of $q_0$ , box $(0.29, 0.38, 0.24, 0.30)$ with 0.60, 0.10, 0.30. Query $q_2$ predicts $(0.68, 0.56, 0.30, 0.42)$ with 0.10, 0.70, 0.20.

The L1 distances to $A$ are $0.01 + 0.01 + 0 + 0.02 = 0.04$ for $q_0$ and $0.07$ for $q_1$ , and $q_2$ is $0.05$ from $B$ . The GIoUs are 0.844, 0.726, and 0.833 (computed on corners in the cell below). So the matching costs are

\mathbf{C}_{q_0A} = -0.80 + 5(0.04) + 2(1 - 0.844) = -0.288, \quad \mathbf{C}_{q_1A} = -0.60 + 5(0.07) + 2(1 - 0.726) = 0.297, \quad \mathbf{C}_{q_2B} = -0.70 + 5(0.05) + 2(1 - 0.833) = -0.116,

and every pair that puts a query on the far object costs more than 6. There are $3\cdot2 = 6$ ways to give the two objects two different queries. The cheapest is $A\leftarrow q_0$ , $B\leftarrow q_2$ at $-0.404$ , and the runner-up is $A\leftarrow q_1$ , $B\leftarrow q_2$ at $0.181$ . So $q_1$ is trained toward "no object".

Its class term is $-0.1\log0.30 = 0.120$ , against $-\log0.80 = 0.223$ and $-\log0.70 = 0.357$ for the matched queries. Dividing by the weight sum $1 + 1 + 0.1$ gives a class loss of $(0.223 + 0.357 + 0.120)/2.1 = 0.333$ . The box loss is $5\cdot(0.04 + 0.05)/2 + 2\cdot(0.156 + 0.167)/2 = 0.548$ , for a total of 0.882. The loss itself punishes the second box on the cat, so no NMS is needed afterward.

Python · cell 7

The widget lets you play the matcher. Unlike the example above, its confident query is the loose one. Its $q_0$ is a loose box on cat $A$ at confidence 0.8, and its $q_1$ a second box on $A$ at 0.6, the right size but off center. Drag $q_1$ onto $A$ and watch the moment the assignment flips. Watch the margin to the runner-up matching too. A small margin means a tiny change in one box can hand an object to a different query.

Interview

DETR's matching and Hungarian loss

Write detr_match_and_loss(logits, pred_boxes, gt_labels, gt_boxes, l1=5.0, giou=2.0, eos_coef=0.1) for one image, and return (pred_idx, gt_idx, loss).

The inputs are these. logits has shape (N_q, K + 1), and its last column is the no-object class. pred_boxes has shape (N_q, 4) and gt_boxes has shape (m, 4), both in normalized center form $(x_c, y_c, w_b, h_b)$ . gt_labels holds m integers in 0..K-1. Assume $1\le m\le N_q$ .

Build the (N_q, m) cost matrix $-\hat p_i(c_j) + \lambda_{L1}\lVert\cdot\rVert_1 + \lambda_{\text{giou}}(1 - \operatorname{GIoU})$ , with softmax probabilities (computed stably, so large logits cannot overflow) and with GIoU computed on corners, and match with scipy.optimize.linear_sum_assignment. The loss has three terms. The class term is $\sum_i w_i(-\log\hat p_i(t_i))/\sum_i w_i$ over all queries, where $t_i$ is the matched class or no object, and $w_i$ is eos_coef for no object and 1 otherwise. The L1 term is l1 times the summed L1 distance of the matched pairs, divided by $m$ . The GIoU term is giou times the summed $1 - \operatorname{GIoU}$ of the matched pairs, divided by $m$ .

import numpy as np
from scipy.optimize import linear_sum_assignment

def detr_match_and_loss(logits, pred_boxes, gt_labels, gt_boxes, l1=5.0, giou=2.0, eos_coef=0.1):
    """logits (N_q, K + 1), last column = no object. Boxes (cx, cy, w, h) in [0, 1].
    Return (pred_idx, gt_idx, loss)."""
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

In PyTorch, a DETR head is a set of learned queries, one decoder layer, and two small heads, and the whole training step fits on a page. The block below overfits one image with three objects. With so few objects, it finds the matching by brute force over all $10\cdot9\cdot8 = 720$ ordered choices, which gives the same answer as the Hungarian method. Real code calls linear_sum_assignment on the detached cost matrix, since the matching itself is not differentiated.

The block makes one more simplification. PyTorch's built-in decoder layer has no argument for positional encodings. So the block feeds the learned queries in as the decoder's input, as §4.6 introduced them, instead of starting from zeros and adding the queries in every layer as DETR does.

🔥 In PyTorchfor a machine with PyTorch

Three queries end up confident, with the right classes and boxes, and the other seven say "no object". On one image, the matching settles at the first step. Across a dataset it does not, and that is DETR's weakness.

Why DETR converges slowly, and what fixed it

DETR's slow training has two causes. First, every object query starts as an arbitrary vector, and its cross-attention spreads almost evenly over all the image tokens. Each query must learn from scratch where to look. Global attention over a large map is also expensive, as the walkthrough above measured, which is why DETR could afford only stride 32 and lost small objects. Second, the matching is unstable early in training. When predictions are poor, many queries have similar costs for the same object, so the assignment flips from one step to the next. A query is trained toward a cat in one step and toward "no object" in the next.

The descendants of DETR fix one cause or both, with three ideas. First, let each query attend to a few points instead of the whole map. Second, give each query an explicit box to refine, and add training queries whose targets are known without matching. Third, match each object to several queries during training only. Together they cut training from 500 epochs to 12, with higher accuracy, and made DETRs the strongest detectors. An interviewer who asks about DETR almost always asks this next.

🔬 How DETR's descendants fixed its training

Sparse attention to a few points. Deformable DETR (Zhu et al. 2021) replaces dense cross-attention. Each query attends to a few sampling points near a reference point, with offsets the network predicts, on several pyramid levels. The cost then grows with the number of queries and points rather than with the size of the map. Multi-scale features at stride 8 become affordable, small objects improve, and training needs 10 times fewer epochs.
Queries as boxes, and denoising. DAB-DETR (Liu et al. 2022) gives each query an explicit reference box $(x_c, y_c, w_b, h_b)$ and refines it layer by layer, so every cross-attention starts centered on a known region. DN-DETR (Li et al. 2022b) also feeds in noised copies of the true boxes as extra queries that must reconstruct the originals. Those queries have known targets, so they need no matching and give a stable signal. DINO (Zhang et al. 2023), a detector unrelated to the self-supervised method of §28.4, adds contrastive denoising, where noised copies far from the object must predict "no object". It reaches 49.4 AP in 12 epochs with ResNet-50, and 63.2 AP on COCO test-dev with a Swin-L backbone pretrained on Objects365, a large extra detection dataset (63.3 with test-time augmentation).
One-to-many matching in training. Hybrid matching (Jia et al. 2023) and its relatives add a training-only branch in which each object is matched to several queries. That gives many more positive examples per image, while the one-to-one branch used at test time keeps the no-NMS property.

The self-supervised backbones that power these detectors, trained on images without labels, are the subject of Chapter 28. The point here is structural. The strongest published detectors of recent years are set predictors trained with a matching loss. The pyramid they read comes either from a hierarchical backbone or from a simple pyramid built on a plain ViT.

COCO average precision, exactly

Detection is scored by average precision, AP. Recall the ranking version from §0.11. Sort the predictions by score. At each cutoff, precision is the fraction of the flagged items that are correct, and recall is the fraction of all the positives that have been found. AP summarizes the resulting precision-recall curve in one number. In ranking, and in retrieval (§7.5), every item is relevant or not before you rank anything. In detection, whether a prediction is correct depends on which object it is matched to, on an IoU threshold, and on which other predictions got there first.

So COCO defines its own procedure, and the official evaluation code, pycocotools, is the only authority on it. Here it is step by step, because nearly every detection number in papers and in this section comes from it.

📐 COCO average precision (Lin et al. 2014 and pycocotools)

For one class and one IoU threshold $t$ , the procedure has five steps.

In each image, keep the 100 highest-scoring detections. Sort them by descending score, and put the regular ground-truth objects before the crowd regions.
Match greedily in that order. Each detection takes, among the regular objects not yet matched, the one with the highest IoU, provided that IoU is at least $t$ . If none qualifies, it may match a crowd region. For a crowd region, IoU is computed as the intersection over the detection's own area, and one crowd region can absorb any number of detections. A detection matched to a regular object is a true positive, one matched to a crowd region is ignored, and the rest are false positives.
Pool all the detections that are not ignored over all images and sort them by score. At the $k$ -th one, precision is $\mathrm{TP}_k/k$ and recall is $\mathrm{TP}_k/n_{\text{gt}}$ , where $\mathrm{TP}_k$ counts the true positives among the first $k$ and $n_{\text{gt}}$ counts the regular ground-truth objects.
Replace precision by its envelope, $\tilde P_k = \max_{k'\ge k}P_{k'}$ , the best precision available at this recall or higher.
For each recall level $r\in\{0, 0.01, \dots, 1\}$ , take $\tilde P$ at the first operating point whose recall is at least $r$ , or 0 if recall never reaches $r$ . $\operatorname{AP}_t$ is the mean of these 101 numbers.

COCO's headline AP is the mean of $\operatorname{AP}_t$ over $t\in\{0.50, 0.55, \dots, 0.95\}$ , and then over the categories that have ground truth. $\operatorname{AP}_{50}$ and $\operatorname{AP}_{75}$ are single thresholds. AP small, medium, and large restrict the ground truth to areas below $32^2$ , between $32^2$ and $96^2$ , and above $96^2$ pixels. Objects outside the range are treated like crowd regions, and unmatched detections outside it are ignored rather than counted as false positives.

Mask AP is the same procedure with IoU computed between binary masks instead of boxes. It is what instance segmentation reports, and papers that report both write box AP as $\operatorname{AP}^{\text{bb}}$ .

In plain words, rank all detections by confidence, walk down the list counting hits, and summarize the precision-recall curve by its upper envelope, sampled at evenly spaced recalls. Averaging over thresholds from 0.50 to 0.95 rewards boxes that are not only in the right place but tight.

You will still meet AP numbers computed under older rules, in papers and in interviews. The main ones come from the PASCAL VOC benchmark, which COCO replaced.

✏️ AP of six detections, by hand

Two images hold four objects of one class. Image 1 has $G_1 = (0, 0, 10, 10)$ and $G_2 = (20, 0, 30, 10)$ , and image 2 has $G_3 = (0, 0, 10, 10)$ and $G_4 = (20, 20, 30, 30)$ . The six detections, by descending score, are these.

score	image	box	best IoU	at $t = 0.5$
0.95	1	$(0.5, 0, 10.5, 10)$	0.905 with $G_1$	TP
0.90	1	$(1, 0, 11, 10)$	0.818 with $G_1$ , taken	FP (duplicate)
0.80	2	$(2, 0, 12, 10)$	0.667 with $G_3$	TP
0.70	2	$(40, 40, 50, 50)$	0	FP
0.60	1	$(23, 0, 33, 10)$	0.538 with $G_2$	TP
0.50	2	$(40, 0, 50, 10)$	0	FP

Walking down the list, precision is $1, \tfrac12, \tfrac23, \tfrac12, \tfrac35, \tfrac12$ and recall is $\tfrac14, \tfrac14, \tfrac12, \tfrac12, \tfrac34, \tfrac34$ . The envelope, swept from the right, is $1, \tfrac23, \tfrac23, \tfrac35, \tfrac35, \tfrac12$ . The 26 recall points $0, 0.01, \dots, 0.25$ are first reached at the first detection and read 1. The 25 points $0.26, \dots, 0.50$ read $\tfrac23$ , the 25 points $0.51,\dots,0.75$ read $\tfrac35$ , and the 25 points above 0.75 read 0, because $G_4$ is never found. So

\operatorname{AP}_{50} = \frac{26\cdot1 + 25\cdot\tfrac23 + 25\cdot\tfrac35}{101} = \frac{57.67}{101} = 0.571 .

Now raise the threshold. From 0.55 to 0.65, the detection on $G_2$ (IoU 0.538) becomes a false positive, and $\operatorname{AP}_t = (26 + 25\cdot\tfrac23)/101 = 0.422$ . From 0.70 to 0.90, the detection on $G_3$ (IoU 0.667) fails too, leaving $26/101 = 0.257$ . At 0.95 nothing matches. The COCO AP is $(0.571 + 3\cdot0.422 + 5\cdot0.257 + 0)/10 = 0.313$ , little more than half of $\operatorname{AP}_{50}$ , because only one box is tight.

The cell reproduces the numbers at threshold 0.5, compares the three interpolation rules, and draws the curve with its envelope.

Python · cell 8

The three rules give 0.571, 0.564, and 0.567 on the same detections. Differences of this size are why papers state their protocol, and why the number from your own script may not match the leaderboard until you use pycocotools, COCO's official evaluation code.

The widget runs the whole procedure on a scene of five objects. It draws the precision-recall curve with its envelope at a chosen IoU threshold, and the AP at each of the ten thresholds that COCO averages.

Interview

COCO average precision for one class

Write coco_ap(dets, gts, iou_thresholds=np.linspace(0.5, 0.95, 10), max_dets=100) and return the mean of $\operatorname{AP}_t$ over the thresholds. dets is a list of (image_id, [x1, y1, x2, y2], score) and gts a list of (image_id, [x1, y1, x2, y2], is_crowd), all for one class.

Follow the COCO definition above exactly. That means the cap of max_dets detections per image, and greedy matching in descending score order to the unmatched regular object of highest IoU, provided it is at least $t$ . A crowd region is matched only when no regular object qualifies, with IoU equal to the intersection over the detection's area, and its matches are then ignored. Finish with the envelope and 101-point interpolation. Assume at least one regular object. The tests include the worked example, crowd regions, the cap, and random scenes checked against a reference that agrees with pycocotools.

import numpy as np

def coco_ap(dets, gts, iou_thresholds=np.linspace(0.5, 0.95, 10), max_dets=100):
    """dets: list of (image_id, [x1, y1, x2, y2], score). gts: list of (image_id, [x1, y1, x2, y2], is_crowd).
    Return COCO AP for one class, averaged over the IoU thresholds."""
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Two properties of AP matter in practice. First, it measures ranking, not calibration. Multiply every score by 0.1 and AP does not move. A detector with excellent AP can still output scores that are useless as probabilities, so a deployed system that thresholds scores needs a separate calibration step (§0.11). Second, low-scoring detections are free. A false positive ranked below every true positive adds an operating point at the final recall with lower precision, which the envelope ignores, so AP does not change at all. That is why detectors submit up to 100 boxes per image even when an image holds three objects.

Segmentation heads

A semantic segmentation model must output a class for every pixel, at the input resolution, from features at stride 16 or 32. A 512 by 512 image at stride 16 gives a 32 by 32 grid, so the head must turn 1,024 vectors into 262,144 pixel labels. The oldest answer is the fully convolutional network (Long et al. 2015). It applies a $1\times1$ convolution to the last feature map as a classifier for each location. Then it upsamples the class scores back to full resolution with bilinear interpolation or transposed convolutions (§0.8). Skip connections add in scores from finer, earlier stages to sharpen the boundaries. For Transformers there are three standard designs.

Reassembling a plain ViT. DPT (Ranftl et al. 2021) reads the tokens of a plain ViT at four depths and reshapes each set back into a grid. It resamples the four grids to strides 4, 8, 16, and 32 with convolutions and transposed convolutions, and fuses them from coarse to fine. It is an FPN built from a ViT's layers rather than its stages.
An all-MLP decoder. SegFormer (Xie et al. 2021) pairs a hierarchical Transformer encoder with a decoder that projects each stage to a common width, upsamples all of them to stride 4, concatenates them, and classifies each pixel.
Mask classification. Instead of asking each pixel "which class are you", ask "which masks are there, and what class is each one". This is DETR's set prediction applied to masks.

Mask classification works as follows in MaskFormer (Cheng et al. 2021) and its successor Mask2Former (Cheng et al. 2022). A pixel decoder, an FPN-like module on the backbone, produces an embedding for every pixel of a stride-4 grid with $H'$ rows and $W'$ columns, stacked as $\mathbf{F}\in\R^{H'W'\times d}$ . A Transformer decoder updates $N_q$ queries, as in DETR. Each query $j$ yields a class distribution $\hat p_j$ over $K + 1$ classes and a mask embedding $\mathbf{e}_j\in\R^d$ . Its mask is the dot product of the mask embedding with every pixel's embedding,

\hat{\mathbf{Y}}_j = \sigma(\mathbf{F}\mathbf{e}_j) \in [0,1]^{H'W'}.

In plain words, a pixel belongs to query $j$ 's mask when its embedding points the same way as the query's mask embedding. Training uses Hungarian matching, as in DETR, with a mask cost in place of the box cost. That cost is a binary cross-entropy or focal term plus a dice term, both computed on $112\times112 = 12{,}544$ sampled points rather than on the whole mask, which cuts training memory by 3 times in Mask2Former.

Dice ignores the pixels that are correctly left out, which is the point. Take a 100-pixel image whose object covers 4 pixels, and a model that predicts probability 0.01 everywhere, an empty mask. Its per-pixel cross-entropy averages 0.19, which looks small because 96 of the 100 pixels are right. Its dice loss is $1 - (2\cdot0.04 + 1)/(1.0 + 4 + 1) = 0.82$ , close to the maximum of 1. A small object cannot be drowned out by a large background.

One output format now serves all three segmentation tasks. For semantic segmentation, pixel $u$ gets the class $\arg\max_c\sum_j\hat p_j(c)\,\hat{\mathbf{Y}}_j[u]$ . For instance segmentation, each query is a scored mask. For panoptic segmentation, drop the queries whose most likely class $c_j$ is no object or whose confidence is low, and give pixel $u$ to the query $\arg\max_j \hat p_j(c_j)\,\hat{\mathbf{Y}}_j[u]$ .

Mask2Former's main change is in the cross-attention. A query that is meant to describe one object should not average features from the whole image. So Mask2Former restricts each query to the region its own current mask covers. It uses the additive attention mask of §4.4, which puts $-\infty$ on the scores of the positions a query must not see, so their softmax weights are exactly 0.

📐 Masked attention (Cheng et al. 2022)

In decoder layer $\ell$ , the cross-attention of query $j$ to pixel $u$ gets an additive mask computed from the previous layer's predicted mask, binarized at 0.5,

\mathbf{M}^{(\ell-1)}_{j,u} = \begin{cases} 0 & \text{if } \hat{\mathbf{Y}}^{(\ell-1)}_j[u] \ge 0.5,\\ -\infty & \text{otherwise,}\end{cases} \qquad \mathbf{o}_j = \sum_u \operatorname{softmax}_u\Big(\frac{\mathbf{q}_j^\top\mathbf{k}_u}{\sqrt{d_k}} + \mathbf{M}^{(\ell-1)}_{j,u}\Big)\,\mathbf{v}_u ,

where $\mathbf{q}_j$ is the query, $\mathbf{k}_u$ and $\mathbf{v}_u$ are the key and value of pixel $u$ , and $d_k$ is the key width. Each query attends only inside its own current foreground, so the mask is predicted rather than fixed. If a predicted mask is empty, the query attends everywhere instead.

Measuring segmentation, mIoU and panoptic quality

Semantic segmentation is scored per class, over pixels. A confusion matrix counts in entry $n_{ab}$ , for $a, b\in\{1,\dots,K\}$ , the pixels whose true class is $a$ and predicted class is $b$ , summed over the whole dataset. From it, $\mathrm{TP}_c = n_{cc}$ are the pixels correctly called $c$ , $\mathrm{FP}_c = \sum_a n_{ac} - n_{cc}$ the pixels wrongly called $c$ , and $\mathrm{FN}_c = \sum_b n_{cb} - n_{cc}$ the pixels of class $c$ that were missed. These are the counts of §0.11, with pixels as the examples.

📐 Mean intersection over union (mIoU)

With an ignore label (often 255) marking pixels that are unlabeled or ambiguous, drop every pixel whose target is the ignore label, accumulate one confusion matrix over all pixels of all images, and set

\operatorname{IoU}_c = \frac{\mathrm{TP}_c}{\mathrm{TP}_c + \mathrm{FP}_c + \mathrm{FN}_c}, \qquad \operatorname{mIoU} = \frac{1}{|\mathcal{C}|}\sum_{c\in\mathcal{C}}\operatorname{IoU}_c,

where $\mathcal{C}$ is the set of classes with $\mathrm{TP}_c + \mathrm{FP}_c + \mathrm{FN}_c > 0$ , those present in the prediction or the target. A class that never appears in either has an undefined IoU and is skipped, not counted as 0 or 1.

mIoU is the IoU from the start of this section, applied to the set of pixels of one class. Every class gets the same weight in the mean, so a class with few pixels counts as much as the sky. It has two blind spots. It does not know about instances, so merging two touching cars into one blob costs nothing. And accumulating over the dataset, the standard, gives a different number from averaging per-image scores, which weights small images and rare classes differently.

Panoptic quality, PQ (Kirillov et al. 2019), scores the segments themselves rather than the pixels, so it does see instances. A predicted segment and a true segment of the same class are matched if their IoU is strictly greater than 0.5. That threshold makes the matching unique without any search.

Because each true segment has at most one match, three counts follow with no search at all. They are the matched pairs, the predicted segments left over, and the true segments left over, and PQ is built from exactly those three.

📐 Panoptic quality

With the matched pairs as the true positives TP, the unmatched predicted segments as FP, and the unmatched true segments as FN, for one class,

\operatorname{PQ} = \frac{\sum_{(p,g)\in\mathrm{TP}}\operatorname{IoU}(p, g)}{|\mathrm{TP}| + \tfrac12|\mathrm{FP}| + \tfrac12|\mathrm{FN}|} = \underbrace{\frac{\sum_{\mathrm{TP}}\operatorname{IoU}}{|\mathrm{TP}|}}_{\text{segmentation quality}}\times\underbrace{\frac{|\mathrm{TP}|}{|\mathrm{TP}| + \tfrac12|\mathrm{FP}| + \tfrac12|\mathrm{FN}|}}_{\text{recognition quality}},

and the dataset's PQ is the average over classes. The second factor is the F1 score of the matching, and the first is the average mask quality of what was found.

Core

mIoU from a confusion matrix

Write miou(pred, target, n_classes, ignore_index=255) for integer label arrays of the same shape, with any number of images stacked in front. Return (miou, iou), where iou has shape (n_classes,) and holds np.nan for classes absent from both the prediction and the target. Drop the pixels whose target is ignore_index, then accumulate one confusion matrix over all the remaining pixels.

import numpy as np

def miou(pred, target, n_classes, ignore_index=255):
    """pred, target: integer label arrays of the same shape, any number of images.
    Return (mIoU, per-class IoU array with nan for classes absent from both)."""
    pred, target = np.asarray(pred), np.asarray(target)
    acc = float((pred == target).mean())                             # placeholder, pixel accuracy
    return acc, np.full(n_classes, acc)

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Promptable segmentation, from SAM to SAM 3

Every model so far was trained for a fixed list of classes. Kirillov et al. (2023) asked a different question, the vision version of prompting a language model. Can one model segment anything a user points at, in any image, without being told its class? Their answer is SAM, the Segment Anything Model, now the standard tool for labeling masks and for interactive cut-outs.

The ambiguity clause matters. A click on a person's shirt could mean the shirt or the person, and both answers are correct. A model trained with one output and an ordinary loss learns to output the average of the plausible masks, which is no mask at all. SAM's design follows from two requirements, a real-time response to each prompt and an honest handling of ambiguity. It has three parts.

Image encoder. The encoder is a ViT-H/16 (the encoder of §27.2, huge size, 16-pixel patches), pretrained without labels by masked autoencoding, which hides most patches and learns to reconstruct them. It reads the image resized so its longer side is 1024 and padded to $1024\times1024$ , a $64\times64$ grid of 4,096 tokens. Global attention over 4,096 tokens in all 32 blocks would be expensive. So 28 blocks attend only within $14\times14$ windows of tokens and four equally spaced blocks attend globally, the design that §27.5 described for plain-ViT detectors. Two convolutions then give a $64\times64\times256$ image embedding, computed once per image.

Prompt encoder. Each point becomes a 256-dimensional vector, a positional encoding of its location plus a learned embedding that says foreground or background. A box becomes two such vectors, one per corner. A mask prompt is downsampled by small convolutions and added to the image embedding.

Mask decoder. The decoder adds a few learned output tokens to the prompt vectors. An output token is a learned vector, like the class token of §27.1, whose final state becomes one predicted mask. All these tokens go through a two-layer Transformer of width 256 with 8 heads. Each layer does four things. The tokens attend to each other. The tokens cross-attend to the 4,096 image vectors. An MLP updates the tokens. And the image vectors cross-attend back to the tokens. That last step updates the image embedding with the prompt, which is why the design is called two-way. Each mask is then the dot product of an output token's vector with every pixel of an upsampled image embedding, the same mask-as-dot-product idea as Mask2Former.

🔬 Inside SAM's encoders and decoder

The point encoding uses random Fourier features (Tancik et al. 2020), $[\sin(2\pi\mathbf{G}^\top\mathbf{u}), \cos(2\pi\mathbf{G}^\top\mathbf{u})]$ , for the coordinate $\mathbf{u}$ rescaled to $[-1, 1]^2$ and a fixed Gaussian matrix $\mathbf{G}\in\R^{2\times128}$ . It is a continuous-coordinate cousin of the sinusoids of §5.4. Each box corner has its own learned type embedding. A mask prompt is given at a quarter of the input resolution and downsampled 4 more times by two $2\times2$ stride-2 convolutions (with 4 and 16 channels), then mapped to 256 channels by a $1\times1$ convolution. The image encoder's neck is a $1\times1$ convolution to 256 channels and a $3\times3$ convolution, each followed by LayerNorm. In the decoder, the image embedding is upsampled 4 times by two transposed convolutions to $256\times256$ , and a small MLP turns each output token into a 32-dimensional vector that is dotted with every upsampled pixel.

Ambiguity. For a single ambiguous prompt, the decoder outputs three masks, typically the whole object, a part, and a sub-part, plus a predicted IoU for each. Training computes the loss against every one of the three and back-propagates only the smallest. Each output is then free to specialize in one interpretation instead of averaging them. When several prompts are given, ambiguity is rare, and a fourth output token produces a single mask.

📌 SAM by the numbers (Kirillov et al. 2023)

The mask loss is focal plus dice in a 20 to 1 ratio, and the IoU head is trained with mean squared error against the actual IoU of each predicted mask. Training simulates interaction, 11 rounds per mask. The first prompt is a random point or a noisy box, 8 rounds each add a point in the error region of the previous prediction, and 2 rounds add nothing, so the model learns to refine its own mask. The prompt encoder and decoder together run in about 50 ms on a CPU in a web browser, and use less than 1 percent of the image encoder's compute. SA-1B, the dataset built with SAM in the loop, has 11M images and 1.1B masks, 99.1 percent of them generated fully automatically, about 400 times more masks than the largest earlier segmentation dataset. Training took 68 hours on 256 A100 GPUs.

The split between a heavy encoder and a light decoder is an amortization argument. The expensive part runs once and is shared by many cheap requests. It is worth checking the size of the gap yourself. The cell counts multiply-adds for SAM's ViT-H encoder at 1024 pixels, with the 4,096-token grid padded to $70\times70$ so that it tiles into 25 windows of $14\times14$ . Then it makes a rough count for the decoder.

Python · cell 9

The encoder costs about 2.8 trillion multiply-adds and the decoder about 1.8 billion, a ratio near 1,600. Windowing saves 1.1 trillion multiply-adds per image compared with all-global attention, by the cost formula of §27.2. The encoder runs once on a GPU when the image is loaded. Every click afterward costs only the decoder, which is why SAM feels instant in a browser. Interviewers like this question because the answer is a design principle, put the expensive computation where it can be shared.

In code, the split is visible in the API. The predictor below is from the released segment_anything package (shown, not run).

Research

SAM's minimum-over-masks loss and IoU targets

Write sam_mask_loss(pred_logits, gt_mask, focal_w=20.0, dice_w=1.0, alpha=0.25, gamma=2.0). pred_logits has shape (3, H, W), three candidate masks as logits, and gt_mask is a binary (H, W) array. For each candidate, compute two losses. The first is the sigmoid focal loss averaged over pixels, with $\alpha_t$ equal to alpha on foreground pixels and 1 - alpha on background pixels. The second is the dice loss on probabilities, with smoothing 1. Combine them as focal_w * focal + dice_w * dice, and return (loss, k), the smallest combined loss and the index of the mask that achieves it. Use a numerically stable log-sigmoid.

Also write iou_target(pred_logits, gt_mask), which returns the (3,) IoU of each mask, thresholded at logit 0, with the ground truth. When a mask and the ground truth are both empty, the prediction is perfect, so return 1. This is the target SAM's IoU head regresses toward.

import numpy as np

def sam_mask_loss(pred_logits, gt_mask, focal_w=20.0, dice_w=1.0, alpha=0.25, gamma=2.0):
    """pred_logits (3, H, W), gt_mask (H, W) of 0 and 1. Return (min combined loss, index of that mask)."""
    raise NotImplementedError

def iou_target(pred_logits, gt_mask):
    """Return the (3,) IoU of each thresholded mask (logit > 0) with gt_mask."""

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

SAM 2, memory for video

A video is a sequence of frames in which the object you clicked keeps moving, gets hidden, and comes back. Running SAM frame by frame would need a new prompt on every frame. SAM 2 (Ravi et al. 2024) turns the model into a streaming one that remembers the object.

Each frame passes once through an image encoder. SAM 2 uses Hiera (Ryali et al. 2023), a hierarchical ViT pretrained with MAE and described in §27.5. Its finer stages also feed the mask decoder directly, to sharpen mask edges. Before decoding, a memory attention module of 4 Transformer blocks lets the current frame's features attend to a memory bank of earlier frames.

The memory bank holds memories of recent frames and of every frame the user prompted, each built from the frame's features and its predicted mask. A new head predicts whether the object is visible at all in the current frame, since in video it may be fully hidden. The prompt encoder and decoder are SAM's.

Here is the loop on a concrete clip. You click a dog in frame 1. In frames 2 to 39 the memory of frame 1 and of the latest frames lets the model find the dog without a new click. In frame 40 the dog walks behind a tree, and the visibility head says the object is absent, so no mask is drawn. In frame 55 the dog reappears, and the kept memory of the prompted frame lets the model pick it up again. If it picks the wrong dog, one corrective click on that frame becomes a new prompted memory.

As with SAM, a data engine with the model in the loop built the training set. SA-V has 50.9K videos with 642.6K masklets (an object's masks across time), 35.5M masks in total. On video segmentation, SAM 2 is more accurate than earlier methods while needing 3 times fewer interactions. On images it is more accurate than SAM and 6 times faster, mostly because the Hiera encoder is smaller and more efficient than ViT-H.

From a fixed class list to text queries

Every detector so far ends in a $K$ -way classifier, one learned weight vector per class. So it can find only the $K$ classes it was trained on, and adding "forklift" means labeling new data and retraining. The fix comes from CLIP, which trains an image encoder and a text encoder so that a picture and its caption land near each other. CLIP's zero-shot classifier embeds each class name with the text encoder and picks the class whose embedding is closest to the image's. Do the same for regions instead of whole images. Replace the class weight vectors by text embeddings of the class names, and the list of classes becomes an input.

In plain words, a region is labeled with whichever phrase its embedding points toward, and the phrases can be anything the text encoder can read. A sigmoid per phrase, rather than a softmax over phrases, lets one region match "dog" and "terrier" at once, and lets an image contain none of the queried phrases. The bias matters because most region-phrase pairs are negatives, the same imbalance as in focal loss.

The simplest instance is OWL-ViT (Minderer et al. 2022). It takes a ViT image-text model pretrained like CLIP and removes the final pooling. To every output patch token it attaches a linear projection that gives $\mathbf{e}_i$ and a small MLP that gives a box. So every patch token, at least 576 of them, is a candidate object, and there is no decoder. It is fine-tuned on detection data with DETR's matching loss. Its ViT-L/14 reaches 34.6 AP on LVIS, a dataset with over a thousand categories. On LVIS's rare categories, which were removed from its detection training data, it reaches 31.2 AP.

A second family pretrains on phrase grounding, the task of finding the region that each phrase of a sentence describes, and fuses the text into the image features. The main examples are GLIP (Li et al. 2022c) and Grounding DINO (Liu et al. 2023), whose largest model reaches 52.5 AP on COCO with no COCO training data.

Read such "zero-shot COCO" numbers with care. Objects365, a large detection dataset in their training data, nearly covers COCO's 80 categories. So the test is new images, not new concepts, and the rare categories of LVIS are the harder test.

🔬 Grounded pretraining and real-time open vocabulary

Two details of OWL-ViT are worth knowing. Its released code predicts a scale and a shift of the logit from each token, so $\tau$ and $b$ vary per region. And its head accepts the image embedding of an example object in place of $\mathbf{t}_k$ , which gives one-shot detection by example. GLIP writes the class names as one sentence ("person. bicycle. car. ...") and scores each region against each word. Its image and text encoders cross-attend to each other in their last layers. Grounding data are just phrases paired with boxes, so GLIP could pretrain on 27M pairs, 3M annotated by people and 24M web image-caption pairs whose boxes a teacher GLIP proposed. GLIP-L reaches 49.8 AP on COCO without training on COCO. Grounding DINO builds the same idea into the DINO detector, fusing text into the backbone features, the choice of initial queries, and the decoder, and its Swin-L model is the one that reaches 52.5 AP. YOLO-World (Cheng et al. 2024) brings open vocabulary to real time. It gives a YOLO detector a CLIP text encoder and fuses text and image features in the middle of the network. A deployment usually has a fixed list of words, so their embeddings are computed once and folded into the weights, and the detector then runs at plain YOLO speed, with 35.4 AP zero-shot on LVIS minival at 52 FPS on a V100.

SAM 3, prompting with concepts

SAM and SAM 2 segment the one object you point at. Ask for "every yellow school bus" and they cannot help, because they have no notion of a category. SAM 3 (Carion et al. 2025) defines promptable concept segmentation. Given a short noun phrase ("striped cat"), image exemplars (boxes on examples, positive or negative), or both, it detects, segments, and tracks every instance of the concept in an image or a video.

In terms of the previous subsection, SAM 3 is an open-vocabulary detector built like DETR. It also accepts exemplar boxes as prompts, and it adds masks, a presence token, and tracking. Its architecture brings this whole section together. A shared backbone, the Perception Encoder (Bolya et al. 2025), is trained to align images with text as CLIP does, and it encodes both the image and the phrase. A fusion encoder lets the image features cross-attend to the prompt tokens. Then a DETR-style decoder of learned object queries predicts, for each query, whether it is a match and its box. Each box is refined layer by layer, as in DETR's descendants above, and a mask head dots query embeddings with pixel embeddings, as in Mask2Former.

The new piece is a learned global presence token, which predicts only whether the concept appears in the image at all. Each query then solves only localization, and its final score is a product,

p(\text{query } i \text{ is a match}) = p(\text{concept present in the image})\cdot p(\text{query } i \text{ matches}\mid\text{concept present}).

In plain words, deciding "is there a forklift in this picture" needs global context, while drawing a box needs local detail. Asking every query to do both at once makes the two objectives fight, and factoring the score separates recognition from localization. For video, a tracker in the style of SAM 2 propagates each detected instance with memory.

How text-aligned image features are trained is the subject of CLIP's contrastive training, and how the wording of a class prompt changes zero-shot accuracy is taken up with zero-shot classification. Finding the moment in a video that a phrase describes is taken up with video-language models. Robots use such masks too, taking the outline of the object to grasp from a SAM-style model instead of learning segmentation from scratch, a pattern that recurs in robot learning.

Choosing among the families

Family	Examples	Output and post-processing	Strengths	Weaknesses
Two-stage, anchor-based	Faster R-CNN, Mask R-CNN, Cascade R-CNN	RPN proposals, then RoIAlign features and per-region class, box, and mask; NMS	Mature, accurate on small objects, easy to add heads	Many hand-set hyperparameters, NMS, slower
One-stage, dense	RetinaNet, FCOS, CenterNet, YOLO family	A prediction at every location, from anchors or as distances to the sides; NMS	Fast, simple deployment	Duplicates handled only by NMS, crowds suffer, results depend on the label assignment rule
Set prediction	DETR, Deformable DETR, DINO, RT-DETR, RF-DETR	A fixed set of queries; no NMS	End to end, the strongest published accuracy, and the best real-time results since RT-DETR	Matching makes training slower and less stable, needs the fixes above
Mask classification	MaskFormer, Mask2Former	Query masks plus classes, one model for three segmentation tasks	Unifies semantic, instance, panoptic	Heavy pixel decoder, fixed class list
Open-vocabulary	OWL-ViT, GLIP, Grounding DINO, YOLO-World	Region or query embeddings scored against text embeddings of any phrase list	New classes without retraining, and they pre-label data for everything else	Below a closed-set detector trained on the target classes, and zero-shot numbers often share concepts with the training data
Promptable	SAM, SAM 2, SAM 3	Masks for a click, box, phrase, or exemplar	Zero-shot, interactive, labels data for everything else	SAM and SAM 2 have no semantics; quality is bounded by the prompt

A useful default in 2026, for a new detection or segmentation problem with modest labeled data, is a DETR-family detector or a Mask2Former-style head on a strong self-supervised ViT backbone, with SAM 2 or SAM 3 in the labeling loop to produce the training masks.

⚠️ Three silent bugs in detection code

The box format. COCO files store $(x_1, y_1, w_b, h_b)$ , torchvision expects corners, and DETR predicts normalized centers, and every combination produces plausible-looking numbers. Convert at one boundary and assert there. The evaluation protocol. A gain of 0.5 points can come from using 11-point instead of 101-point interpolation, more than 100 detections per image, or a different crowd rule, so compare numbers only from the same pycocotools call. The NMS threshold at test time. A detector tuned with NMS at 0.5 and evaluated at 0.7 (or the reverse) changes AP on crowded classes by points. Models in the DETR family should be evaluated with no NMS at all, since NMS slightly hurts them at the last decoder layer (Carion et al. 2020).

🧭 What to remember

Dense tasks output sets, boxes or masks with classes and scores, in no order. IoU measures overlap independently of scale, and GIoU adds an enclosing-box penalty so that $1 - \operatorname{GIoU}$ still has a gradient for boxes that do not overlap.
Classical detectors label anchors by IoU, regress scale-free box deltas, read features from an FPN, and remove duplicates with NMS, a greedy heuristic that fails in crowds. Two-stage detectors propose regions with an RPN and read them with RoIAlign, which samples bilinearly with no rounding. One-stage detectors fight the imbalance of about 100,000 easy negatives with focal loss, whose gradient shrinks by about $(1 + \gamma_f)(1 - p_t)^{\gamma_f}$ for easy examples.
DETR makes detection set prediction. Hungarian matching picks the cheapest one-to-one assignment of $N_q$ queries to objects, the loss does not depend on the order of the queries, and duplicates are trained toward no object, so no NMS is needed. Its global attention costs the square of the token count, which kept it on the stride-32 map. Deformable attention, reference boxes, denoising, and one-to-many matching fixed its slow convergence and its small objects, and DETRs now lead real-time detection.
COCO AP matches detections greedily by score at IoU thresholds from 0.50 to 0.95, makes precision monotone, samples it at 101 recalls, and ignores crowd regions. It measures ranking, not calibration, and protocol details move it by points.
Mask classification predicts masks as dot products of query and pixel embeddings, and Mask2Former's masked attention lets one model do semantic, instance, and panoptic segmentation. mIoU accumulates one confusion matrix over the dataset and skips absent classes, and PQ is segmentation quality times recognition quality.
Open-vocabulary detectors score regions against text embeddings, $\mathbf{e}_i^\top\mathbf{t}_k/\tau + b$ , so classes become an input. SAM shares one heavy image encoding among many cheap prompts and resolves ambiguity with three masks and a minimum loss. SAM 2 adds a streaming memory for video, and SAM 3 combines open-vocabulary detection with masks, a presence token, and tracking.
Next, Chapter 28 explains how the self-supervised backbones under these systems, DINO and MAE among them, learn their features without labels.

Carion et al. (2020). End-to-End Object Detection with Transformers. DETR. Read Section 3.1 for the set loss and Appendix A.2 and A.4 for the exact losses and training details.
Lin et al. (2017b). Focal Loss for Dense Object Detection. RetinaNet and focal loss, with the clearest account of why class imbalance hurts one-stage detectors.
Cheng et al. (2022). Masked-attention Mask Transformer for Universal Image Segmentation. Mask2Former, mask classification, and masked attention, with careful ablations.
He et al. (2017). Mask R-CNN. The mask branch, RoIAlign, and the ablations of Table 2 that show why alignment matters, the clearest account of two-stage detection.
Lin et al. (2014). Microsoft COCO: Common Objects in Context, with the official evaluation code, cocoeval.py. The dataset, and the only authoritative definition of COCO AP.
Kirillov et al. (2023). Segment Anything. The promptable task, the SAM architecture (Appendix A), and the SA-1B data engine.
Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos. Streaming memory attention and the SA-V dataset.
Carion et al. (2025). SAM 3: Segment Anything with Concepts. Promptable concept segmentation, the presence token, and the data engine with AI verifiers.

Made for larger screens

Detection, segmentation, and SAM

Four tasks, four kinds of output

Measuring overlap, IoU and GIoU

Pairwise IoU and GIoU

Classical detectors, from anchors to RoIAlign and focal loss

Anchors and box deltas

Encoding and decoding box deltas

Feature pyramids

Non-maximum suppression

Non-maximum suppression, class-aware

Two-stage detectors and RoIAlign

RoIAlign from scratch

Focal loss, and why easy negatives matter

Anchor-free heads and label assignment

DETR, detection as set prediction

DETR's matching and Hungarian loss

Why DETR converges slowly, and what fixed it

COCO average precision, exactly

COCO average precision for one class

Segmentation heads

Measuring segmentation, mIoU and panoptic quality

mIoU from a confusion matrix

Promptable segmentation, from SAM to SAM 3

SAM's minimum-over-masks loss and IoU targets

SAM 2, memory for video

From a fixed class list to text queries

SAM 3, prompting with concepts

Choosing among the families

Read more

Watch next