Chapter 34 · Section 34.7 · 105 min

Control, personalization, and editing

This section is about steering an image generator that is already trained. A text prompt describes a whole family of images, but people often want one particular image. They want this pose, this face, or this layout, or a real photo with one thing changed. Retraining a large generator for each wish is out of the question, so every method here leaves most of the trained model untouched.

Here is the problem in one concrete case. A designer has a pencil sketch of a car's outline, a product photo from the client, and a sentence. No sentence can say "this outline, this exact logo, and change only the sky". Retraining is not an option either, since training Stable Diffusion 1.5, an open text-to-image model, took about 150,000 A100 GPU hours (§34.1). So the question is practical. How do we give a trained generator that much control without retraining it, and without breaking what it already knows?

The section answers in four parts. First come new input paths that carry a sketch, a depth map, or a reference photo into the frozen model. We prove that the safest of them start by changing nothing at all. Then come ways to teach the model one new subject, such as your dog, from a few photos. Then come five ways to edit a real photo, by noising it, by inverting it to noise, by steering attention, by instruction, and by filling a masked region. Last, we ask how anyone can tell a generated image from a real one, and what watermarks and signed records can and cannot prove.

A few facts from earlier chapters run through the whole section, so here they are in brief. The generators work on a latent, a small grid of numbers into which an autoencoder compresses an image and from which it expands it back (§34.1). Training mixes a clean latent $\mathbf{z}_0$ with Gaussian noise $\boldsymbol\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ along a straight line, $\mathbf{z}_t = (1-t)\mathbf{z}_0 + t\boldsymbol\epsilon$ . The time $t$ runs from 0, clean data, to 1, pure noise. This straight path is the one a rectified flow uses (§33.6).

The network that removes noise is the denoiser. It reads the noisy latent $\mathbf{z}_t$ , the time $t$ , and a condition $\mathbf{c}$ such as a prompt. It predicts either the noise, written $\boldsymbol\epsilon_\theta$ , or the velocity $\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{c}) \approx \mathbb{E}[\boldsymbol\epsilon - \mathbf{z}_0 \mid \mathbf{z}_t]$ , the direction from data toward noise. A sampler makes an image by starting from pure noise at $t = 1$ and stepping against the velocity down to $t = 0$ .

One more tool appears everywhere. Classifier-free guidance makes the model follow its condition more strictly (§33.5). It runs the network twice per step, once with the condition and once with an empty prompt, and extrapolates from the second prediction through the first, $\mathbf{v}_\varnothing + \gamma(\mathbf{v}_c - \mathbf{v}_\varnothing)$ . Here $\mathbf{v}_c$ and $\mathbf{v}_\varnothing$ are the two predictions and $\gamma$ is the guidance scale. The value $\gamma = 1$ is plain conditional sampling, and larger values follow the prompt harder. Some papers write the scale as $w$ or $s$ , and we translate their numbers into $\gamma$ .

Three ways to steer a frozen generator

A frozen model is one whose weights we do not change. Every method in this section keeps the big generator frozen, or almost frozen, and does one of three things.

Add an input path. New layers carry a new condition, such as an edge map, a depth map, or a reference photo, into the frozen denoiser. The design question is how to add them without disturbing the model when training starts. The cleanest answer is a path whose output starts at exactly zero.
Change a few parameters. To teach the model a new subject, we train as little as one new word vector. Larger options are a low-rank update to some weight matrices, a chosen subset of the weights, or all of them with a penalty that protects the old behavior.
Change the sampling, not the model. Nothing is trained. We start sampling from a noised copy of a photo, or find the noise from which the model regenerates it. Or we rewrite attention while sampling, or overwrite the known part of an image at every step.

The first family rests on one trick, a new branch whose last layer starts at zero. Here it is with numbers, before any architecture.

✏️ A zero branch, by hand

Take a one-number "denoiser" $y = wx$ with $w = 2$ , frozen. To make it respond to a control signal $c$ , add a branch with a trainable copy $w_c$ of $w$ and two layers $z_1$ and $z_2$ that start at zero,

y = wx + z_2\,w_c\,(x + z_1c), \qquad w_c = 2,\quad z_1 = z_2 = 0 \text{ at the start.}

Train on one example, $x = 1$ and $c = 1$ , with target $y^\star = 4$ and loss $L = \tfrac12(y - y^\star)^2$ . The initial output is $y = 2$ , exactly the frozen model's, and the residual is $r = y - y^\star = -2$ . The chain rule gives

\frac{\partial L}{\partial z_2} = r\,w_c(x + z_1c) = -4, \qquad \frac{\partial L}{\partial w_c} = r\,z_2(x + z_1c) = 0, \qquad \frac{\partial L}{\partial z_1} = r\,z_2\,w_c\,c = 0 .

Only $z_2$ moves. With learning rate 0.1 it becomes $0.4$ , the output becomes $2 + 0.4\cdot2 = 2.8$ , and the residual becomes $-1.2$ . Now $\partial L/\partial z_2 = -1.2\cdot2 = -2.4$ , $\partial L/\partial w_c = -1.2\cdot0.4\cdot1 = -0.48$ , and $\partial L/\partial z_1 = -1.2\cdot0.4\cdot2\cdot1 = -0.96$ . All three learn from the second step on. At no point did the branch add a random perturbation to the output.

So a branch that ends in a zero layer starts as an exact no-op and still learns. Two facts make that work. The frozen output passes through unchanged, and the zero layer still sees a nonzero input, so it receives a gradient. The rest of this part scales the toy up to a real model.

ControlNet, a trainable copy behind two zero layers

ControlNet adds a spatial condition, such as an edge map, a depth map, or a human pose skeleton, to a frozen text-to-image model (Zhang, Rao, and Agrawala 2023). It is the toy above at full scale. To see where its branch goes, recall the diffusion U-Net of §34.2. Its encoder is a stack of blocks that shrinks the latent grid level by level. A middle block works on the smallest grid, and a decoder grows the grid back. Skip connections hand each encoder level's features straight to the decoder level of the same size. The encoder is where the layout of the image is analyzed, so that is where a spatial condition should enter.

Two more terms are needed. A $1\times1$ convolution applies the same linear map, plus a bias, to the channel vector at every pixel. A zero convolution is a $1\times1$ convolution whose weights and bias start at zero.

📐 ControlNet

Let $\mathcal{F}(\cdot\,;\Theta)$ be a block of a pretrained network that maps a feature map $\mathbf{x} \in \R^{h\times w\times c}$ to $\mathbf{y} = \mathcal{F}(\mathbf{x};\Theta)$ . ControlNet freezes $\Theta$ , makes a trainable copy $\Theta_c$ initialized to $\Theta$ , and adds two zero convolutions $\mathcal{Z}(\cdot\,;\Theta_{z1})$ and $\mathcal{Z}(\cdot\,;\Theta_{z2})$ . For a condition feature map $\mathbf{c}_f$ of the same shape as $\mathbf{x}$ , the controlled block computes

\mathbf{y}_c = \mathcal{F}(\mathbf{x};\Theta) + \mathcal{Z}\Big(\mathcal{F}\big(\mathbf{x} + \mathcal{Z}(\mathbf{c}_f;\Theta_{z1});\,\Theta_c\big);\,\Theta_{z2}\Big).

For Stable Diffusion the copy is the whole encoder plus the middle block. The encoder has 12 blocks, three at each latent resolution 64, 32, 16, and 8. The copy runs on the same noisy latent, timestep, and prompt as the locked model. A small convolutional encoder maps the $512\times512$ condition image to the $64\times64$ latent grid. The condition can be edges from the Canny edge detector, depth, surface normals, line drawings, a segmentation, or a human pose. The copy's 13 outputs, each through its own zero convolution, are added to the U-Net's 12 skip connections and to the output of its middle block. In the released code, the input zero convolution is the last layer of the small condition encoder, initialized to zero.

In plain words, the locked model keeps doing what it did. The copy reads the condition and computes corrections, with a backbone that already understands images. The zero layers decide how much of each correction reaches the locked model, and at the start they let none of it through.

Why copy the encoder instead of training a small new network? A fresh network would have to learn to read images from the condition data alone. The paper tested this. Replacing each copied block by a single convolution gave a variant the authors call ControlNet-lite, and it failed whenever the prompt did not already describe the scene. Why zero layers instead of ordinary ones? The same study replaced the zero convolutions by convolutions with Gaussian random weights, and quality fell to the level of ControlNet-lite. The authors attribute this to the random layers damaging the copy's pretrained weights early in fine-tuning.

The copy is not free. The cell builds the Stable Diffusion 1.5 U-Net layer by layer from its released configuration, checks the total against the known size, and then counts the ControlNet.

Python · cell 1

Why the zero layers do no harm and still learn

The hand example generalizes. The argument is the one that showed a DiT block starts as the identity when its modulation layer starts at zero, the adaLN-Zero initialization of §34.2. A $1\times1$ convolution acts at each position $(i, j)$ as an affine map, $\mathcal{Z}(\mathbf{h})_{ij} = \mathbf{W}\mathbf{h}_{ij} + \mathbf{b}$ . So its gradients are sums over positions of outer products, exactly as for a linear layer applied to each token (§0.2).

∴ ControlNet starts as the base model and opens from the output side

Take a controlled block at initialization ( $\Theta_c = \Theta$ , $\Theta_{z1} = \Theta_{z2} = \mathbf{0}$ ), any condition $\mathbf{c}_f$ , and let $\mathrm{d}\mathbf{y}_c$ be the gradient of the loss arriving at the block's output. Write $\mathbf{h} = \mathcal{F}(\mathbf{x} + \mathcal{Z}(\mathbf{c}_f;\Theta_{z1});\Theta_c)$ for the copy's features. Then

the block equals the frozen block, $\mathbf{y}_c = \mathcal{F}(\mathbf{x};\Theta)$ , for every input and condition;
the output zero layer receives $\mathrm{d}\mathbf{W}_{z2} = \sum_{ij}\mathrm{d}\mathbf{y}_{c,ij}\,\mathbf{h}_{ij}^\top$ and $\mathrm{d}\mathbf{b}_{z2} = \sum_{ij}\mathrm{d}\mathbf{y}_{c,ij}$ , which are nonzero in general, since $\mathbf{h} = \mathcal{F}(\mathbf{x};\Theta)$ is a nonzero feature map;
the copy $\Theta_c$ , the input zero layer $\Theta_{z1}$ , and the condition encoder receive exactly zero gradient;
after one gradient step in which $\mathrm{d}\mathbf{W}_{z2} \ne \mathbf{0}$ , the gradient reaching $\mathbf{h}$ is $\mathbf{W}_{z2}^\top\mathrm{d}\mathbf{y}_c \ne \mathbf{0}$ , and from then on the copy and the input zero layer learn too.

The proof is in the box below.

∂ Proof of the zero-start theorem

With $\mathbf{W}_{z1} = \mathbf{0}$ and $\mathbf{b}_{z1} = \mathbf{0}$ the copy's input is $\mathbf{x}$ , so $\mathbf{h} = \mathcal{F}(\mathbf{x};\Theta_c) = \mathcal{F}(\mathbf{x};\Theta)$ . With $\mathbf{W}_{z2} = \mathbf{0}$ and $\mathbf{b}_{z2} = \mathbf{0}$ the second term is zero, which is item 1. At each position the output zero layer computes $\mathbf{W}_{z2}\mathbf{h}_{ij} + \mathbf{b}_{z2}$ , and the linear-layer gradients give item 2 and $\mathrm{d}\mathbf{h}_{ij} = \mathbf{W}_{z2}^\top\mathrm{d}\mathbf{y}_{c,ij} = \mathbf{0}$ . Everything that affects the loss only through $\mathbf{h}$ (the copy's weights, the input zero layer, and the condition encoder that produces $\mathbf{c}_f$ ) therefore gets zero gradient, which is item 3. After a step with learning rate $\eta$ , $\mathbf{W}_{z2} = -\eta\,\mathrm{d}\mathbf{W}_{z2} \ne \mathbf{0}$ , so $\mathrm{d}\mathbf{h} \ne \mathbf{0}$ in general. The input zero layer then receives $\mathrm{d}\mathbf{W}_{z1} = \sum_{ij}\mathrm{d}\mathbf{u}_{ij}\,\mathbf{c}_{f,ij}^\top$ , where $\mathrm{d}\mathbf{u} = \mathbf{J}_{\mathcal{F}}^\top\mathrm{d}\mathbf{h}$ is the gradient at the copy's input. That is item 4. ∎

In plain words, at the start the output zero layer blocks everything behind it, but it still sees the copy's features. So it is the only part that learns on the first step, and its gradient measures how well those features line up with what the loss wants. The copy and the input layer start to learn only once the output layer is nonzero. Two consequences follow. The first steps of training cannot inject random features into the frozen model. And the copy's pretrained weights are never pushed around by gradients that passed through a random layer.

The cell checks the theorem on a toy with one tanh block of width 8. The branch must learn to add an effect of the control signal to the frozen block's output.

Python · cell 2

At step 0 only the output layer has a gradient, and from step 1 on all three do. The widget runs the same toy and lets you replace the zero layers by random ones.

Small random starts behave almost like zero, because the branch's output and its noisy gradients both scale with the initial weights. Zero is the limit of that. It is the one choice that is exactly harmless at any learning rate and any depth. An interviewer who asks "why zero and not small random?" wants this answer.

Core

A ControlNet block and its gradients

Build the toy ControlNet of the theorem and check its gradients. The frozen block is $F(\mathbf{U};\mathbf{W}) = \tanh(\mathbf{U}\mathbf{W}^\top)$ . It acts on a batch of rows $\mathbf{U} \in \R^{n\times d}$ , with weights $\mathbf{W} \in \R^{d\times d}$ . For inputs $\mathbf{X} \in \R^{n\times d}$ and controls $\mathbf{C} \in \R^{n\times d_c}$ , the controlled block and its loss are

\mathbf{Y} = \tanh(\mathbf{X}\mathbf{W}^\top) + \tanh\big((\mathbf{X} + \mathbf{C}\mathbf{Z}_1^\top)\mathbf{W}_c^\top\big)\mathbf{Z}_2^\top, \qquad L = \frac{1}{2n}\sum_{i,j}(Y_{ij} - Y^{\text{tgt}}_{ij})^2.

Implement controlnet_init(W, d_c). It returns a dict with the frozen W, a trainable copy Wc that is a separate array, and two zero layers, Z1 of shape $(d, d_c)$ and Z2 of shape $(d, d)$ .
Implement controlnet_forward_and_grads(X, C, Y_tgt, p). It returns (Y, L, grads), where the dict grads holds the gradients of $L$ with respect to Z1, Z2, and Wc, under those keys.

The tests check the theorem at initialization, compare your gradients with finite differences away from it, and check that one step opens the branch.

import numpy as np

def controlnet_init(W, d_c):
    raise NotImplementedError

def controlnet_forward_and_grads(X, C, Y_tgt, p):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Training and using a ControlNet

Training uses the base model's ordinary denoising loss, with the condition image as an extra input. Only the copy, the zero layers, and the small condition encoder are trained. One detail matters a lot. The authors replaced half of the training prompts by the empty string. That forces the copy to read the condition image itself instead of leaning on the text.

At sampling time a conditioning scale multiplies the ControlNet's outputs before they are added. It trades how strictly the image follows the condition against how freely it follows the prompt. Several ControlNets compose by adding their outputs, for example pose from one and depth from another, and the paper found that this needs no extra weighting.

Guidance needs one decision, because the control can join either prediction or both. Suppose it joins both the conditional and the unconditional prediction. Then with an empty prompt the two predictions are identical, and guidance does nothing. Suppose instead it joins only the conditional prediction. Then the whole control effect sits inside $\mathbf{v}_c - \mathbf{v}_\varnothing$ , guidance multiplies it by $\gamma$ , and the control becomes very strong. The authors add it to the conditional prediction only and then weight each level by its resolution, as the box below explains.

🔬 Sudden convergence, data size, and resolution weighting

The model does not learn the condition gradually. Its samples stay good throughout, since the base model is intact. Then, usually within 10,000 optimization steps, it suddenly starts following the condition. The authors call this the sudden convergence phenomenon.

Training was robust from under 50,000 to over a million image pairs. One comparison makes the point. Stability's own depth-to-image model for Stable Diffusion 2 was trained on more than 12 million images on a cluster of A100s. A depth ControlNet for the same base was trained on 200,000 samples on one RTX 3090 Ti in five days. Users asked to tell their outputs apart did no better than chance (precision $0.52 \pm 0.17$ ).

For guidance, the paper multiplies the connection of a block at resolution $h_i$ by $w_i = 64/h_i$ . So a connection on the $8\times8$ grid weighs eight times as much as one on the $64\times64$ grid, and the finest levels get the weakest push. The authors call this CFG resolution weighting. The released code applies the same ordering as a scale that falls from 1 at the middle block to about 0.1 at the finest level.

Lighter adapters for structure and for image prompts

A ControlNet is heavy because it copies the encoder. Two lighter designs keep the base model fully frozen and add far fewer weights.

The first is the T2I-Adapter (Mou et al. 2023), a small separate network for structure. It rearranges the $512\times512$ condition image onto the $64\times64$ latent grid by moving blocks of pixels into channels. Four feature blocks, with three downsamplings between them, turn the result into four feature maps. Their shapes match the four levels of the U-Net encoder, and each is added to the encoder's features at its level, $\hat{\mathbf{F}}_{\text{enc}}^i = \mathbf{F}_{\text{enc}}^i + \mathbf{F}_c^i$ . The adapter has about 77 million parameters.

The adapter does not read the time $t$ . So its features are computed once per image and reused at every sampling step, while a ControlNet runs at every step. The price is that the adapter cannot adapt its features to the noise level. Several adapters compose with weights, $\mathbf{F}_c = \sum_k\omega_k\mathbf{F}_{\text{AD}}^k(\mathbf{C}_k)$ , where $\mathbf{F}_{\text{AD}}^k$ is the $k$ -th adapter and $\mathbf{C}_k$ its condition image. A sketch for structure plus a color palette is a typical pair.

The second, IP-Adapter (Ye et al. 2023), solves a different problem. It lets an image act as a prompt, as in "in the style of this" or "this object". Text enters the U-Net through cross-attention. There the image tokens supply the queries, and the text tokens supply the keys and values (§4.6, §34.3). The obvious move is to append image tokens to the text tokens. The authors found that this works poorly, and gave the image its own attention instead.

📐 Decoupled cross-attention (IP-Adapter)

At a cross-attention layer let $\mathbf{X} \in \R^{T\times d}$ be the image tokens and $\mathbf{Q} = \mathbf{X}\mathbf{W}_Q$ their queries. Let $\mathbf{K}_t = \mathbf{C}_{\text{ctx}}\mathbf{W}_K$ and $\mathbf{V}_t = \mathbf{C}_{\text{ctx}}\mathbf{W}_V$ be the keys and values of the $T_c$ text tokens $\mathbf{C}_{\text{ctx}} \in \R^{T_c\times d_c}$ , all frozen. A frozen CLIP image encoder embeds the reference image, and a trainable projection (a linear layer and a LayerNorm) turns that embedding into $N_i$ image tokens $\mathbf{C}_{\text{img}} \in \R^{N_i\times d_c}$ . New trainable matrices $\mathbf{W}_K', \mathbf{W}_V' \in \R^{d_c\times d_k}$ , initialized from $\mathbf{W}_K$ and $\mathbf{W}_V$ , give $\mathbf{K}_i = \mathbf{C}_{\text{img}}\mathbf{W}_K'$ and $\mathbf{V}_i = \mathbf{C}_{\text{img}}\mathbf{W}_V'$ . The layer outputs

\operatorname{Attn}(\mathbf{Q}, \mathbf{K}_t, \mathbf{V}_t) + \lambda_{\text{ip}}\operatorname{Attn}(\mathbf{Q}, \mathbf{K}_i, \mathbf{V}_i), \qquad \operatorname{Attn}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \operatorname{softmax}\!\big(\mathbf{Q}\mathbf{K}^\top/\sqrt{d_k}\big)\mathbf{V},

with the same queries in both terms and a weight $\lambda_{\text{ip}}$ chosen at inference ( $\lambda_{\text{ip}} = 1$ in training). Only the projection and the new $\mathbf{W}_K', \mathbf{W}_V'$ of every cross-attention layer are trained. The formula shows one attention head. A multi-head layer splits the $d_k$ columns into heads and scales each head by the square root of its own width.

Why keep two softmaxes when one longer softmax over all the tokens would be simpler? The theorem below says what each choice does to the text path.

∴ Separate softmaxes keep the text path intact

With $\lambda_{\text{ip}} = 0$ , a decoupled layer computes exactly the original text cross-attention.
If instead the image tokens are appended to the text tokens in one softmax, the output for a query $\mathbf{q}$ is the convex blend $\pi(\mathbf{q})\operatorname{Attn}(\mathbf{q}, \mathbf{K}_t, \mathbf{V}_t) + (1 - \pi(\mathbf{q}))\operatorname{Attn}(\mathbf{q}, \mathbf{K}_i, \mathbf{V}_i)$ . Here $\pi(\mathbf{q}) = Z_t/(Z_t + Z_i)$ , and $Z_t = \sum_j e^{\mathbf{q}^\top\mathbf{k}_{t,j}/\sqrt{d_k}}$ and $Z_i = \sum_j e^{\mathbf{q}^\top\mathbf{k}_{i,j}/\sqrt{d_k}}$ are the two partial normalizers.

∂ Proof that one softmax blends the two attentions

Item 1 is the definition. For item 2, write $s_{t,j}$ for the query's scaled score on text key $j$ . The joint softmax's denominator is $Z_t + Z_i$ . Split the weighted sum of values into its text part and its image part, and multiply and divide each by its own partial normalizer,

\frac{1}{Z_t + Z_i}\sum_j e^{s_{t,j}}\mathbf{v}_{t,j} = \frac{Z_t}{Z_t + Z_i}\cdot\frac{1}{Z_t}\sum_j e^{s_{t,j}}\mathbf{v}_{t,j},

and the same for the image part. This is the identity that §34.2 proved for joint attention, where image and text tokens share one softmax. ∎

In plain words, concatenation makes the image compete with the text for each query's attention. How much the image takes is set by the query's own scores, and the user cannot control it. Concatenation also forces the frozen text projections to handle image tokens they never saw. Decoupling adds the image's contribution on top of an untouched text path, with a dial $\lambda_{\text{ip}}$ that returns exactly the base model at zero.

One caveat is worth stating. IP-Adapter does not start as the identity during training. Its new projections are copies of $\mathbf{W}_K$ and $\mathbf{W}_V$ , which speeds up training, and its image tokens come from a newly initialized projection.

Python · cell 3

The total, 22.3 million, matches the paper's "about 22M". That is 2.6 percent of the 860 million parameters of the U-Net it steers. Because the base model stays frozen, the same adapter works with fine-tuned versions of that base model and alongside ControlNets.

🔬 How IP-Adapter was trained, and what came after it

IP-Adapter was trained on about 10 million image-text pairs from two web-scraped datasets, LAION-2B and COYO-700M, for a million steps on 8 V100 GPUs. The text was dropped alone with probability 0.05, the image alone with probability 0.05, and both together with probability 0.05, so that guidance works for either condition. Identity encoders work the same way, with no optimization per subject. PhotoMaker, for example, stacks the embeddings of several photos of one person into one identity embedding (Li et al. 2023).

The frontier systems of late 2025 treat reference images as ordinary context instead, as the editing part below explains. Black Forest Labs' FLUX.2 (November 25, 2025) accepts up to 10 reference images. Google's Nano Banana Pro (November 20, 2025) blends up to 14 and keeps up to 5 people consistent.

Teaching the model one new subject

You have four photos of your dog and want "your dog on the moon". Teaching a trained model one new subject from a few images is called personalization. An image encoder like IP-Adapter needs no training per subject, but it generalizes only as far as its own training data. Fine details such as a particular coat pattern often do not survive one embedding. The alternative is to optimize something for this subject alone, which costs minutes per subject. The methods below differ in what they optimize, from one word vector up to every weight.

Textual inversion, one new word

The smallest thing we can train is the vector for one new word. The text encoder starts by looking each token up in a table of embeddings, one learned vector per token (§1.2). We add one row to that table and train only that row.

📐 Textual inversion

Pick a placeholder token $S_\ast$ and give it a new embedding $\mathbf{v}_\ast \in \R^{d_e}$ in the text encoder's input table ( $d_e = 768$ for the CLIP text encoder of Stable Diffusion 1). Keep the denoiser $\theta$ , the text encoder $\phi$ , and every other embedding frozen. Train only $\mathbf{v}_\ast$ on 3 to 5 images $\mathbf{x}$ of the concept, captioned by templates $y$ such as "a photo of $S_\ast$ ", with the ordinary denoising loss,

\mathbf{v}_\ast = \arg\min_{\mathbf{v}}\ \mathbb{E}_{\mathbf{x}, y, t, \boldsymbol\epsilon}\big\lVert\boldsymbol\epsilon - \boldsymbol\epsilon_\theta\big(\mathbf{z}_t, t, \phi(y;\mathbf{v})\big)\big\rVert^2, \qquad \mathbf{z}_0 = \mathcal{E}(\mathbf{x}),

where $\mathcal{E}$ is the autoencoder's encoder. The new word then works in any prompt, as in "an oil painting of $S_\ast$ ".

Textual inversion comes from Gal et al. 2022. They initialized the new embedding from a one-word description such as "cat" or "sculpture", and ran 5,000 steps at batch size 4 on two V100 GPUs. For Stable Diffusion the result is 768 numbers, 3 KB, and it works inside any prompt.

Its ceiling is the frozen model. One embedding can only steer the model toward images it can already make from some input. Fine identity details that no input reaches stay out of reach. That limit is the reason the authors of the next method fine-tuned the weights instead.

Core

Textual inversion on a frozen linear denoiser

Train one new embedding against a frozen toy model, and see its ceiling. The setup builds a frozen linear denoiser that predicts the clean image as $\hat{\mathbf{x}}_0 = a_t\mathbf{z}_t + b_t\mathbf{M}\mathbf{c}$ , with fixed numbers $a_t$ and $b_t$ for each noise level and a fixed matrix $\mathbf{M}$ . The prompt embedding is $\mathbf{c} = \mathbf{q} + \mathbf{P}\mathbf{v}$ . Here $\mathbf{q}$ is what the rest of the prompt contributes, $\mathbf{P}$ is a frozen linear text encoder, and $\mathbf{v} \in \R^5$ is the new token's embedding. The noisy inputs are fixed, $\mathbf{z}_{t_j} = (1 - t_j)\mathbf{x}_i + t_j\boldsymbol\epsilon_{ij}$ , for $N = 4$ images $\mathbf{x}_i$ and $J = 6$ noise levels $t_j$ .

Implement ti_loss_grad(v, prob). It returns the loss $L(\mathbf{v}) = \frac{1}{NJ}\sum_{i,j}\lVert\hat{\mathbf{x}}_0(\mathbf{z}_{t_j}, \mathbf{c}) - \mathbf{x}_i\rVert^2$ and its gradient with respect to $\mathbf{v}$ .
Implement textual_inversion(prob, lr, steps), gradient descent from $\mathbf{v} = \mathbf{0}$ . It returns the final $\mathbf{v}$ and a list of losses, one per step, each computed before that step's update.

Nothing in prob may change. The tests also check that the loss stays large at the optimum, which is the ceiling in action.

import numpy as np

def ti_loss_grad(v, prob):
    raise NotImplementedError

def textual_inversion(prob, lr=0.05, steps=300):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

DreamBooth, fine-tuning with the model's own samples as a guard

DreamBooth fine-tunes the whole denoiser on the subject's photos (Ruiz et al. 2022). Each photo is captioned with a unique identifier next to the subject's class, as in "a [V] dog". The class word lets the model reuse what it knows about dogs. The identifier should be a token the model associates with almost nothing. A real word like "special" carries a meaning the model would have to unlearn. A random string of letters is split into single-letter tokens, which carry strong meanings of their own. So the authors sampled rare tokens from the vocabulary. For the Imagen model they used tokens of at most three characters with ids between 5,000 and 10,000 in its T5 vocabulary.

Fine-tuning every weight on four photos fails in two ways. The model forgets how to make other dogs, so "a dog" starts producing your dog. This is called language drift, a term the authors borrowed from work on fine-tuning language models. And its outputs collapse onto the few poses and views in the photos. The fix is to keep training on the model's own idea of the class.

📐 DreamBooth with prior preservation

Let $\mathbf{z}_0$ be latents of the 3 to 5 subject images with prompt $\mathbf{c}$ ("a [V] dog"). Let $\mathbf{z}_0^{\text{pr}}$ be latents of about 1,000 images that the frozen model generated for the class prompt $\mathbf{c}_{\text{pr}}$ ("a dog") before fine-tuning. DreamBooth minimizes

\mathcal{L}(\theta) = \mathbb{E}\Big[\big\lVert\boldsymbol\epsilon_\theta(\mathbf{z}_t, t, \mathbf{c}) - \boldsymbol\epsilon\big\rVert^2 + \lambda\,\big\lVert\boldsymbol\epsilon_\theta(\mathbf{z}^{\text{pr}}_{t'}, t', \mathbf{c}_{\text{pr}}) - \boldsymbol\epsilon'\big\rVert^2\Big],

with independent noise $\boldsymbol\epsilon, \boldsymbol\epsilon'$ and times $t, t'$ for the two terms. The minimization is over the denoiser's weights, all of them in the paper and often only LoRA adapters in practice. For a flow model each term becomes $\lVert\mathbf{v}_\theta - (\boldsymbol\epsilon - \mathbf{z}_0)\rVert^2$ .

In plain words, the first term teaches the model your dog. The second keeps asking it to make dogs the way it used to, with its own earlier samples as the data. The weight $\lambda$ sets how far the class may move. The authors used $\lambda = 1$ . For Stable Diffusion they trained about 1,000 steps at learning rate $5\times10^{-6}$ , about 5 minutes on one A100.

🔬 The loss as the paper writes it

The paper writes the loss with a prediction of the clean image and a time weight $w_t$ . That differs from the form above only by a positive weight per noise level. The reason is that the two predictions are tied by the same line as the data and the noise. On this chapter's path, $\mathbf{z}_t = (1 - t)\hat{\mathbf{x}}_\theta + t\hat{\boldsymbol\epsilon}_\theta = (1 - t)\mathbf{x}_0 + t\boldsymbol\epsilon$ , so $\hat{\mathbf{x}}_\theta - \mathbf{x}_0 = -\tfrac{t}{1 - t}(\hat{\boldsymbol\epsilon}_\theta - \boldsymbol\epsilon)$ . The two squared errors therefore differ by the factor $t^2/(1 - t)^2$ at each noise level, and other noise schedules give other positive factors.

A linear toy shows how the prior term works. A frozen "generator" maps a prompt embedding $\mathbf{c}$ to a mean image $\mathbf{M}\mathbf{c}$ . The prompt "a [V] dog" is the sum of a small rare-token vector and the "dog" vector. We fine-tune $\mathbf{M}$ on one subject, with and without the prior term.

Python · cell 4

Without the prior term, 91 percent of the subject's offset leaks into every "a dog". The reason is that the gradient of a squared error in $\mathbf{M}\mathbf{u}$ is an outer product with $\mathbf{u}$ . So each step changes $\mathbf{M}$ only along the training prompt's embedding $\mathbf{u} = \mathbf{e}_V + \mathbf{e}_{\text{dog}}$ . That direction is mostly "dog". With the prior term, the same fit puts the subject on the rare token and leaves the class where it was. A prompt orthogonal to the training prompt, here "a cat", does not move in either case. That is why drift shows up first in the class and its neighbors. The toy tracks only the mean image, so it shows drift but not the second failure, the loss of variety, which the prior term also counters.

Custom Diffusion and LoRA, fine-tuning less

Fine-tuning every weight gives a full copy of the model per subject. Two methods train far less.

The first, Custom Diffusion (Kumari et al. 2022), started from a measurement. When a text-to-image model is fine-tuned on a new concept, the weights that change most are in the cross-attention layers. There, only the key and value projections read the text (§34.3). So Custom Diffusion trains only those projections, plus a new modifier token $V^\ast$ . It guards against drift with about 200 real images from a web dataset whose captions closely match the target prompt, instead of generated ones.

The cell above counted those matrices at 19.2 million numbers, 77 MB in 32-bit floats. That is consistent with the paper's report of storing 75 MB instead of 3 GB. Training took about 6 minutes on two A100s. Two concepts trained separately can be merged afterwards, by solving a constrained least-squares problem on the key and value matrices in closed form.

The second is LoRA (§14.5). It restricts the update of a frozen weight matrix $\mathbf{W}_0$ to a product of two thin matrices, $\mathbf{h} = \mathbf{W}_0\mathbf{x} + \tfrac{\alpha}{r}\mathbf{B}\mathbf{A}\mathbf{x}$ . Here $\mathbf{A}$ has $r$ rows, $\mathbf{B}$ has $r$ columns, the rank $r$ is small, and $\mathbf{B} = \mathbf{0}$ at the start. That is the same zero start as ControlNet's output layer. We write the scale as $\alpha/r$ , not $\gamma$ , to keep $\gamma$ for guidance.

In open-source tooling the common practice is DreamBooth's loss with LoRA adapters instead of full fine-tuning. The cell above shows why the files are tiny. Rank 4 on the cross-attention keys and values is 198,144 numbers, 0.4 MB in 16-bit floats. Adapters on every attention projection of the SD 1.5 U-Net, self-attention and cross-attention alike, take 1.6 MB at rank 4 and 13 MB at rank 32.

Stacking several LoRAs is not free. Their updates add, $\Delta\mathbf{W} = \Delta\mathbf{W}_1 + \Delta\mathbf{W}_2$ . So each acts on the other's prompts wherever their input directions overlap, and the network's nonlinearities compound the interaction. That is why two concepts trained separately can blend into each other when loaded together.

Method	What is trained	Size at SD 1.5 scale	Time per subject	Strength	Typical failure
Textual inversion	one token embedding	768 numbers, 3 KB	about 5,000 steps	composable, tiny	cannot reach details the frozen model cannot make
DreamBooth	all denoiser weights	860M, 3.4 GB in fp32	about 1,000 steps, 5 minutes on an A100	highest fidelity	language drift and lost variety without the prior term, overfitting
Custom Diffusion	cross-attention $\mathbf{W}_K, \mathbf{W}_V$ and a token	19.2M, 77 MB	250 steps, 6 minutes on 2 A100s	several concepts at once, mergeable	less fidelity than full fine-tuning on some subjects
DreamBooth with LoRA	low-rank adapters	0.2M to a few M	minutes	small files, swappable per user	adapters interfere when stacked
Encoders (IP-Adapter, PhotoMaker)	nothing per subject	one shared adapter	one forward pass	instant	identity detail limited by one embedding
In-context references (FLUX.2, Nano Banana Pro)	nothing per subject	none	one generation	instant, many references	closed or very large models, attention cost grows with references

In an interview the question is usually which row to pick for a product. The answer turns on fidelity, storage per user, and latency, and the second interview question at the end walks through it.

Editing a real image by noising it

Suppose you have a crude painting, a few colored strokes for a sky, a field, and a tree, and you want a photograph with that layout. A diffusion model can start sampling from any noise level, not only from pure noise. Under enough noise, the painting and a real photo with the same layout look alike, since the noise hides the difference in texture. So noise the painting to that level and let the model denoise it into a photo. This method is called SDEdit (Meng et al. 2021).

This is the strength argument of every image-to-image pipeline. Libraries round differently. The diffusers library runs $\lfloor t_0S\rfloor$ steps. It starts from the point of its fixed sampling grid that leaves exactly that many steps, which can lie a little below $t_0$ . SDXL's refiner, a second model that adds a little noise to the base model's output and denoises it again (§34.1), is SDEdit at a low strength.

What does the strength trade? For Gaussian data we can answer exactly. The result needs one quantity, the fraction of the guide that survives.

∴ SDEdit on Gaussian data

Let the data be $x_0 \sim \mathcal{N}(\mu, s^2)$ in one dimension, let the model be exact, and run SDEdit on a guide $g$ with strength $t_0$ . Define

\rho(t_0) = \frac{s(1 - t_0)}{\sqrt{s^2(1 - t_0)^2 + t_0^2}} \in [0, 1].

With the deterministic sampler (the flow ODE, solved exactly), the output is $\mathcal{N}\big(\mu + \rho\,(g - \mu),\ s^2(1 - \rho^2)\big)$ .
With the exact stochastic sampler, which adds fresh noise at each step as the original paper does, the output is $\mathcal{N}\big(\mu + \rho^2(g - \mu),\ s^2(1 - \rho^4)\big)$ .

In both cases, a guide drawn from the data gives an output distributed as the data. For $d$ -dimensional data $\mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$ the same holds along each eigenvector of $\boldsymbol\Sigma$ , with $s$ replaced by the square root of its eigenvalue.

∂ Proof, by moving to the coordinates of the fast-sampling section

Divide by the signal coefficient, $y = z_t/(1 - t)$ and $\sigma = t/(1 - t)$ , so that $y = x_0 + \sigma\epsilon$ . In these coordinates §33.4 showed that every Gaussian diffusion has the same probability-flow ODE, $\mathrm{d}y/\mathrm{d}\sigma = (y - D(y;\sigma))/\sigma$ , with $D$ the posterior mean of $x_0$ . For Gaussian data, $D(y;\sigma) = \mu + \frac{s^2}{s^2 + \sigma^2}(y - \mu)$ , the Wiener filter of §33.3. Substituting,

\frac{\mathrm{d}(y - \mu)}{\mathrm{d}\sigma} = \frac{\sigma}{s^2 + \sigma^2}(y - \mu) \quad\Longrightarrow\quad y(\sigma) - \mu = \big(y(\sigma_0) - \mu\big)\sqrt{\frac{s^2 + \sigma^2}{s^2 + \sigma_0^2}},

since the right side's logarithmic derivative is $\tfrac12\,\mathrm{d}\log(s^2 + \sigma^2)/\mathrm{d}\sigma = \sigma/(s^2 + \sigma^2)$ . At $\sigma = 0$ the output is $\mu + (y_0 - \mu)\,s/\sqrt{s^2 + \sigma_0^2}$ , where $\sigma_0 = t_0/(1 - t_0)$ and $y_0 = g + \sigma_0\epsilon$ . Multiplying numerator and denominator by $1 - t_0$ shows that $s/\sqrt{s^2 + \sigma_0^2} = \rho(t_0)$ . So the output is $\mu + \rho(g - \mu) + \rho\sigma_0\epsilon$ , with variance $\rho^2\sigma_0^2 = s^2\sigma_0^2/(s^2 + \sigma_0^2) = s^2(1 - \rho^2)$ , which is item 1.

For item 2, the exact reverse process started at $y_0$ ends at a sample from the posterior $p(x_0\mid y_0)$ , because the reverse-time process has the same joint law as the forward one (§33.3). That posterior is Gaussian with mean $\mu + k(y_0 - \mu)$ and variance $s^2(1 - k)$ , where $k = s^2/(s^2 + \sigma_0^2) = \rho^2$ . Averaging over $\epsilon$ , the mean is $\mu + \rho^2(g - \mu)$ and the variance is $k^2\sigma_0^2 + s^2(1 - k)$ . Using $\sigma_0^2 = s^2(1 - k)/k$ , the first term is $s^2k(1 - k)$ , and the total is $s^2(1 - k)(1 + k) = s^2(1 - \rho^4)$ .

If $g \sim \mathcal{N}(\mu, s^2)$ , item 1's output has variance $\rho^2s^2 + s^2(1 - \rho^2) = s^2$ and item 2's has $\rho^4s^2 + s^2(1 - \rho^4) = s^2$ . In $d$ dimensions, rotate to the eigenvectors of $\boldsymbol\Sigma$ . The forward process adds the same noise in every direction, so the coordinates stay independent and each obeys the one-dimensional result. ∎

In plain words, SDEdit keeps a fraction $\rho$ of how far the guide is from typical data and replaces the rest with fresh, realistic variation. The fraction falls from 1 at $t_0 = 0$ to 0 at $t_0 = 1$ . The stochastic sampler keeps only $\rho^2$ , so at the same strength it is less faithful to the guide than the deterministic one.

✏️ An unrealistic guide, four standard deviations out

Let $\mu = 0$ , $s = 1$ , and $g = 4$ , a guide no real image resembles. At $t_0 = 0.5$ , $\rho = 0.5/\sqrt{0.25 + 0.25} = 0.707$ , so the deterministic output is $\mathcal{N}(2.83, 0.50)$ . Its expected squared distance from the guide, a measure of unfaithfulness, is $(1 - \rho)^2g^2 + s^2(1 - \rho^2) = 1.37 + 0.50 = 1.87$ . Its KL divergence from the data distribution, a measure of unrealism, is $\tfrac12\big(\rho^2g^2/s^2 - \rho^2 - \log(1 - \rho^2)\big) = \tfrac12(8 - 0.5 + 0.69) = 4.10$ .

Raising $t_0$ to 0.8 gives $\rho = 0.243$ , a squared distance of 10.1, and a KL of only 0.47. Faithfulness and realism move in opposite directions, and no strength gives both for a guide this far from the data.

The cell runs SDEdit with an exact denoiser and Euler steps, and compares the simulated output with the theorem over a range of strengths.

Python · cell 5

The simulated means and variances match the theorem to within the error of the Euler steps. At $t_0 = 1$ the variance is 0.975 instead of 1. That gap is the error of Euler's method, which is first order in the step size (§33.4). It is about 0.05 with 50 steps and about 0.01 with 200.

The last line holds the most useful intuition. In $d$ dimensions each direction keeps its own fraction. At $t_0 = 0.5$ , a direction with standard deviation 3 keeps 95 percent of the guide's deviation, while one with standard deviation 0.1 keeps 10 percent. Natural images put most of their variance in coarse, low-frequency structure and little in fine detail (§33.1). So a moderate strength keeps the guide's layout and colors and regenerates its texture, which is exactly what one wants from a stroke painting. Meng et al. found that strengths between 0.3 and 0.6, in the time variable of their model, worked for reasonable guides. A guide far from any real image forces a larger strength and a looser match.

Interview

SDEdit with an exact denoiser

Implement sdedit(z0, t0, denoise, S, rng) in three steps.

Noise the guide z0 to $\mathbf{z}_{t_0} = (1 - t_0)\mathbf{z}_0 + t_0\boldsymbol\epsilon$ , drawing $\boldsymbol\epsilon$ with rng.standard_normal(z0.shape).
Take $n = \lceil t_0S\rceil$ equal Euler steps of the flow ODE from $t_0$ down to 0. Each step calls denoise(z, t) once, which returns $\hat{\mathbf{x}}_0$ , and moves along the velocity $(\mathbf{z} - \hat{\mathbf{x}}_0)/t$ .
For $t_0 = 0$ , return the input unchanged.

The setup provides gauss_denoiser(s), the exact denoiser for $\mathcal{N}(0, s^2)$ data, and the tests compare your outputs with the theorem.

import numpy as np

def sdedit(z0, t0, denoise, S, rng):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Inverting an image, and why guidance breaks it

SDEdit throws away the guide's fine detail by design. To edit a real photograph while keeping everything except the thing we change, we want the opposite. We find the exact noise at $t = 1$ from which the model's deterministic sampler regenerates the photo. Then we run the sampler again from that noise with a modified prompt.

Running the deterministic sampler backward like this is DDIM inversion. §33.4 introduced it as a property of the sampling ODE, the differential equation that the deterministic sampler solves step by step. An ODE with a smooth velocity carries each noise to exactly one image, so it can be run backward from the image to its noise. In this chapter's coordinates DDIM inversion is one line, because for a rectified flow a DDIM step and an Euler step are the same step. An Euler step moves from the current point along the current velocity for one step size.

Proof. $\mathbf{z}_t + (t' - t)(\hat{\boldsymbol\epsilon} - \hat{\mathbf{z}}_0) = (1 - t)\hat{\mathbf{z}}_0 + t\hat{\boldsymbol\epsilon} + (t' - t)\hat{\boldsymbol\epsilon} - (t' - t)\hat{\mathbf{z}}_0 = (1 - t')\hat{\mathbf{z}}_0 + t'\hat{\boldsymbol\epsilon}$ . ∎

So DDIM inversion on a grid $0 = t^{(0)} < t^{(1)} < \dots < t^{(S)} = 1$ is $\mathbf{z}_{k+1} = \mathbf{z}_k + h_k\mathbf{v}(\mathbf{z}_k, t^{(k)})$ , with step $h_k = t^{(k+1)} - t^{(k)}$ . It reads the velocity at the point it stands on, the only point it has. Generation walks the same grid down, $\hat{\mathbf{z}}_k = \hat{\mathbf{z}}_{k+1} - h_k\mathbf{v}(\hat{\mathbf{z}}_{k+1}, t^{(k+1)})$ . So each step of the round trip reads the velocity at two different points, one at each end of the step. That mismatch is the whole source of error, as a toy we can do by hand shows.

✏️ A round trip by hand

Take a one-dimensional toy whose velocity is proportional to the point, $v(z) = \lambda z$ , the same at every time. Invert $z_0 = 1$ with $\lambda = 0.6$ in one step of size 1. The inversion step reads the velocity at $z_0$ , which is 0.6, and lands at $z_1 = 1.6$ . The generation step reads the velocity at $z_1$ , which is 0.96, and returns $\hat z_0 = 1.6 - 0.96 = 0.64$ . The error, 0.36, is exactly the difference between the two velocity readings.

With $S$ steps of size $h = 1/S$ , each pair of steps multiplies by $(1 + \lambda h)(1 - \lambda h) = 1 - \lambda^2h^2$ . So the round trip returns $(1 - \lambda^2/S^2)^S \approx 1 - \lambda^2/S$ . With 10 steps that is 0.965, and with 50 it is 0.993. Five times the steps give about one fifth of the error.

Now add guidance. Let the unconditional velocity be $0.3z$ and the conditional one $0.6z$ , so the guided velocity at scale $\gamma$ is $(0.3 + 0.3\gamma)z$ . At $\gamma = 7.5$ the coefficient is 2.55, and the same round trip returns 0.51 with 10 steps and 0.88 with 50. Last, invert at $\gamma = 1$ and regenerate at $\gamma = 7.5$ . The round trip returns $(1 + 0.6h)^S(1 - 2.55h)^S$ , which tends to $e^{0.6 - 2.55} = 0.14$ however many steps you take.

The toy already shows the three facts this part proves and then measures. The error is first order in the step size. Guidance makes it larger. And inverting with one guidance scale and regenerating with another never converges. To state the first two in general we need one term. A field is $L$ -Lipschitz if moving the input by a distance $\delta$ moves the output by at most $L\delta$ .

∴ The round-trip error of DDIM inversion

Let the velocity field $\mathbf{v}(\mathbf{z}, t)$ be $L$ -Lipschitz in $\mathbf{z}$ at each $t$ , and invert $\mathbf{z}_0$ along the grid with largest step $h$ . Suppose the velocity changes slowly along the inversion path, $\lVert\mathbf{v}(\mathbf{z}_{k+1}, t^{(k+1)}) - \mathbf{v}(\mathbf{z}_k, t^{(k)})\rVert \le Mh_k$ for every $k$ . Then regenerating from $\mathbf{z}_S$ along the same grid returns $\hat{\mathbf{z}}_0$ with

\lVert\hat{\mathbf{z}}_0 - \mathbf{z}_0\rVert \;\le\; M\sum_{k=0}^{S-1}h_k^2\prod_{j<k}(1 + Lh_j) \;\le\; M\,h\,e^{L}.

Now take a guided field $\mathbf{v}_\gamma = (1 - \gamma)\mathbf{v}_\varnothing + \gamma\mathbf{v}_c$ whose branches have constants $(L_\varnothing, M_\varnothing)$ and $(L_c, M_c)$ along the same path. The triangle inequality gives $L_\gamma \le |1 - \gamma|L_\varnothing + \gamma L_c$ and $M_\gamma \le |1 - \gamma|M_\varnothing + \gamma M_c$ . So for $\gamma \ge 1$ , when both branches share the constants $L$ and $M$ , the bound grows like $(2\gamma - 1)\,e^{(2\gamma - 1)L}$ .

∂ Proof of the round-trip bound

Let $\mathbf{e}_k = \hat{\mathbf{z}}_k - \mathbf{z}_k$ , so $\mathbf{e}_S = \mathbf{0}$ . Subtract the inversion step $\mathbf{z}_{k+1} = \mathbf{z}_k + h_k\mathbf{v}(\mathbf{z}_k, t^{(k)})$ from the generation step,

\mathbf{e}_k = \hat{\mathbf{z}}_{k+1} - h_k\mathbf{v}(\hat{\mathbf{z}}_{k+1}, t^{(k+1)}) - \mathbf{z}_{k+1} + h_k\mathbf{v}(\mathbf{z}_k, t^{(k)}) = \mathbf{e}_{k+1} + h_k\big[\mathbf{v}(\mathbf{z}_{k+1}, t^{(k+1)}) - \mathbf{v}(\hat{\mathbf{z}}_{k+1}, t^{(k+1)})\big] + h_k\big[\mathbf{v}(\mathbf{z}_k, t^{(k)}) - \mathbf{v}(\mathbf{z}_{k+1}, t^{(k+1)})\big],

after adding and subtracting $h_k\mathbf{v}(\mathbf{z}_{k+1}, t^{(k+1)})$ . The first bracket is at most $L\lVert\mathbf{e}_{k+1}\rVert$ by the Lipschitz condition and the second at most $Mh_k$ , so $\lVert\mathbf{e}_k\rVert \le (1 + Lh_k)\lVert\mathbf{e}_{k+1}\rVert + Mh_k^2$ . Unrolling from $k = S - 1$ down to 0 gives the sum, in which the error made at step $k$ is amplified by every later backward step $j < k$ . Each factor satisfies $1 + Lh_j \le e^{Lh_j}$ and the steps sum to 1, so the product is at most $e^{L}$ , and $\sum_k h_k^2 \le h\sum_k h_k = h$ . For the guided field, $\lVert\mathbf{v}_\gamma(\mathbf{a}) - \mathbf{v}_\gamma(\mathbf{b})\rVert \le |1 - \gamma|\,\lVert\mathbf{v}_\varnothing(\mathbf{a}) - \mathbf{v}_\varnothing(\mathbf{b})\rVert + \gamma\lVert\mathbf{v}_c(\mathbf{a}) - \mathbf{v}_c(\mathbf{b})\rVert$ , and the same holds for the change along the path. ∎

In plain words, each round-trip step errs by the step size times how much the velocity changed during the step. Later steps amplify earlier errors by at most the field's Lipschitz constant. The error is first order, so doubling the number of steps halves the bound. Guidance multiplies the difference between the conditional and unconditional predictions by $\gamma$ . So the guided velocity changes faster along the path and is more sensitive to where it is read, and the bound grows with $\gamma$ .

A bound says only what can happen. The two cells below measure what does happen, on a mixture of three 2D Gaussian classes. For this data both branches of guidance are known exactly, so every error we see comes from the sampler.

Python · cell 6

Python · cell 7

The second cell shows the three facts at scale. First, inverting and regenerating with the same guidance follows one ODE both ways, so the error is pure discretization. With 50 steps it grows with $\gamma$ , from 0.028 at $\gamma = 1$ to 0.102 at $\gamma = 7.5$ . It also falls in proportion to the step size, fourfold from 50 to 200 steps, first order as the theorem says.

Second, the inverted noise at $\gamma = 7.5$ is nothing like a Gaussian draw. Its mean is $(11.6, -4.5)$ and its standard deviations are near 7 and 8, instead of 0 and 1. Mokady et al. 2022 observed the same for Stable Diffusion. There, inverting with a large guidance scale gives a noise vector that may fall outside the Gaussian distribution, which hurts editing.

Third, and worst for editing, is the mixed case. Inverting without guidance gives good, typical noise, and regenerating with guidance is what makes edits take effect. But these follow two different ODEs, which never meet however small the steps. So the error stays at 0.24 for $\gamma = 2$ and 0.81 for $\gamma = 7.5$ , at 50 steps and at 200. Editing needs guidance, so plain inversion cannot be both accurate and editable. That is the interview answer to "why does DDIM inversion fail with classifier-free guidance?"

Null-text inversion, bending the guided path back

The fix keeps the accurate unguided inversion as a reference. Then it adjusts the one input of the guided step that the user does not care about. In classifier-free guidance the unconditional branch reads an embedding of the empty prompt, the null embedding $\varnothing$ . Changing it per step bends the guided path without changing the model or the prompt.

📐 Null-text inversion

Invert the image with $\gamma = 1$ to get a pivot trajectory $\mathbf{z}_0^\ast, \mathbf{z}_1^\ast, \dots, \mathbf{z}_S^\ast$ , indexed by grid point. Then generate with the editing guidance scale $\gamma$ , starting from $\bar{\mathbf{z}}_S = \mathbf{z}_S^\ast$ . At each step $k = S, \dots, 1$ , optimize a separate null embedding $\varnothing_k$ , the text embedding fed to the unconditional branch, so that the guided step lands on the pivot,

\varnothing_k = \arg\min_{\varnothing}\ \big\lVert\mathbf{z}^\ast_{k-1} - \operatorname{step}_\gamma(\bar{\mathbf{z}}_k, t^{(k)};\, \mathbf{c}, \varnothing)\big\rVert^2, \qquad \bar{\mathbf{z}}_{k-1} = \operatorname{step}_\gamma(\bar{\mathbf{z}}_k, t^{(k)};\, \mathbf{c}, \varnothing_k),

starting each optimization from $\varnothing_{k+1}$ . The model weights and the prompt embedding $\mathbf{c}$ stay fixed. Editing then reuses $\bar{\mathbf{z}}_S$ and the optimized $\varnothing_1, \dots, \varnothing_S$ with an edited prompt.

In plain words, the unguided inversion finds where the image's path goes, and the per-step null embeddings bend the guided path back onto it. Mokady et al. used 50 DDIM steps, guidance 7.5, and 10 optimization iterations per step with early stopping, about one minute per image on an A100. They found that one embedding per step reconstructs far better than a single shared one.

In our toy, the stand-in for the null embedding is the unconditional branch's three mixture weights, three numbers per step. We fit them by Gauss-Newton, which linearizes the step around the current weights and solves the resulting least-squares problem, a few times per grid point.

Python · cell 8

Optimizing the null branch brings the guided reconstruction error from between 0.24 and 0.81 down to 0.030, about the round-trip error of unguided inversion itself. It cannot go lower here, for a reason worth understanding. Once the noise is low enough that a point's class is unambiguous, the mixture weights no longer change the prediction. So the null branch has no grip on the last steps. A real null embedding has tens of thousands of numbers and a grip on every step, and Mokady et al. report reconstructions close to what the autoencoder alone allows.

The widget runs the three methods on one image of the same mixture, which you can drag. Dashed lines invert it to noise and solid lines regenerate it with guidance. The plot below draws the round-trip error of all three methods against γ, the current one in bold.

Research

Inversion error against guidance

Measure the three facts above yourself. The setup gives velocity(z, t, gamma), the exact guided velocity for class 0 of a three-class 2D mixture, and real_images(n). Implement three functions.

ddim_invert(x, S, gamma) takes Euler steps upward on the grid np.linspace(0, 1, S + 1).
ddim_generate(z, S, gamma) takes Euler steps downward on the same grid.
round_trip_errors(x, S, gammas, gamma_invert=None) returns one error per $\gamma$ in gammas. For each, it inverts with gamma_invert, or with $\gamma$ itself when gamma_invert is None, regenerates with $\gamma$ , and measures the mean Euclidean distance to x.

The tests check the theorem's predictions. Inversion without guidance is nearly exact, the error grows with $\gamma$ , it converges at first order, and mismatched guidance does not converge.

import numpy as np

def ddim_invert(x, S, gamma):
    raise NotImplementedError

def ddim_generate(z, S, gamma):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Editing by steering attention

Inversion gives us the noise. To change only the dog into a cat, the edited generation must also keep the original's layout, and the layout lives in the cross-attention maps. Recall from §34.3 that the cross-attention weights form a matrix $\mathbf{A} \in \R^{T\times T_c}$ , with one row per image token and one column per prompt token. The column for a word, its attention map, marks where on the image that word is read. These maps fix the composition early in sampling. So copying them from one generation to another copies the composition. That observation is the basis of Prompt-to-Prompt (Hertz et al. 2022).

📐 Prompt-to-Prompt

Generate with the source prompt $\mathcal{P}$ and the edited prompt $\mathcal{P}^\ast$ in parallel, from the same initial noise. At step $t$ let $\mathbf{M}_t$ and $\mathbf{M}_t^\ast$ be the cross-attention maps of the two generations. In every cross-attention layer of the edited generation, replace its maps by $\operatorname{Edit}(\mathbf{M}_t, \mathbf{M}_t^\ast, t)$ while keeping its own values. Three edit functions cover most uses.

Word swap ("a big red bicycle" to "a big red car"). Use $\mathbf{M}_t$ for the early steps, where $t$ is above a threshold $\tau$ , and $\mathbf{M}^\ast_t$ after that.
Adding a phrase ("a castle next to a river" to "children drawing of a castle next to a river"). For each token $j$ of $\mathcal{P}^\ast$ with a match $A(j)$ in $\mathcal{P}$ , use column $A(j)$ of $\mathbf{M}_t$ , and keep $\mathbf{M}_t^\ast$ 's column for new tokens.
Reweighting. Multiply the column of one token by a factor $c \in [-2, 2]$ to strengthen or weaken its effect, as for "snowy" mountain.

The threshold trades fidelity to the edit against fidelity to the layout. In the paper's example of turning a bicycle into a car, injecting the source maps at every step forced the car into a bicycle's shape. Injecting them for only the early steps kept the scene and let the shape change.

The method needs no training. It only runs the two generations as one batch. But it has two requirements. The prompts must share structure, since the maps are matched token by token. And for a real image the source generation must reproduce the photo, which is what null-text inversion provides. That is why the two methods are usually used together.

Editing by instruction and in context

Inversion and attention control work, but they ask the user for a full description of the image and need a minute of optimization per photo. The current approach trains the model to take the image itself as an input, together with a sentence that says what to change.

InstructPix2Pix (Brooks, Holynski, and Efros 2022) had no editing data, so it made some. The authors wrote 700 triplets by hand, each an image caption, an edit instruction, and the edited caption. They fine-tuned the language model GPT-3 on these, and it generated a corpus of 454,445 triplets from captions in LAION, a web-scraped image-caption dataset. For each caption pair, Stable Diffusion with Prompt-to-Prompt generated 100 candidate before-and-after image pairs, and a CLIP-based filter kept the consistent ones.

The model starts from Stable Diffusion 1.5 with extra input channels in its first convolution. It reads the noisy latent stacked with the latent of the input image. The weights on the new channels start at zero, the same zero start once more. With two conditions, the input image $c_I$ and the instruction $c_T$ , guidance gets two scales.

📐 Guidance with two conditions (InstructPix2Pix)

Train with $c_I$ alone dropped for 5 percent of examples, $c_T$ alone for 5 percent, and both for 5 percent. At sampling time, with guidance scales $s_I$ for the image and $s_T$ for the text, use

\tilde{\boldsymbol\epsilon} = \boldsymbol\epsilon_\theta(\mathbf{z}_t, \varnothing, \varnothing) + s_I\big(\boldsymbol\epsilon_\theta(\mathbf{z}_t, c_I, \varnothing) - \boldsymbol\epsilon_\theta(\mathbf{z}_t, \varnothing, \varnothing)\big) + s_T\big(\boldsymbol\epsilon_\theta(\mathbf{z}_t, c_I, c_T) - \boldsymbol\epsilon_\theta(\mathbf{z}_t, c_I, \varnothing)\big),

which costs three network evaluations per step.

In plain words, the first difference pushes toward the input image, and the second pushes toward the instruction given that image. Raising $s_T$ makes the edit stronger, and raising $s_I$ keeps more of the input's structure. The authors found $s_T$ between 5 and 10 and $s_I$ between 1 and 1.5 usually best. An edit took about 9 seconds on an A100.

∂ Where the two-scale formula comes from

The formula is classifier-free guidance applied to a chain rule of probabilities. By Bayes, $\log p(\mathbf{z}\mid c_I, c_T) = \log p(\mathbf{z}) + \log p(c_I\mid\mathbf{z}) + \log p(c_T\mid c_I, \mathbf{z}) + \text{const}$ . So the conditional score is the unconditional one plus two terms. One is $\nabla\log p(\mathbf{z}\mid c_I) - \nabla\log p(\mathbf{z})$ , for the image. The other is $\nabla\log p(\mathbf{z}\mid c_I, c_T) - \nabla\log p(\mathbf{z}\mid c_I)$ , for the instruction given the image. Scaling each term separately and converting scores to noise predictions (§33.5) gives the definition. ∎

In-context editing removes the special input channels. FLUX.1 Kontext (Black Forest Labs 2025) encodes the input image with the same frozen autoencoder as the output. It appends the input's latent tokens $\mathbf{y}$ to the target's tokens $\mathbf{x}$ in the image stream of its flow transformer. It starts from the FLUX.1 text-to-image model and trains with the plain rectified-flow loss $\lVert\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{y}, \mathbf{c}) - (\boldsymbol\epsilon - \mathbf{x})\rVert^2$ , on millions of curated (input, instruction, output) triples.

The context tokens are clean, never noised. They are told apart from the target by their position code. Each token carries a 3D rotary position $(i, h, w)$ . The first coordinate is 0 for every target token and $i$ for every token of the $i$ -th context image, while the spatial positions $(h, w)$ of the images still line up. When there is no input image its tokens are left out, so one model does both generation and editing. The authors also tried stacking the input along channels, as InstructPix2Pix does, and found it worse. FLUX.1 Kontext edits a $1024\times1024$ image in 3 to 5 seconds. Its open [dev] version is a transformer with 12 billion parameters. It is guidance-distilled, meaning it was trained to output the guided prediction in one network call instead of two.

Inpainting, filling a masked region

Inpainting regenerates the pixels inside a region so that they fit the rest of the image. The region is given by a mask $\mathbf{m}$ , which is 1 where the image is known and 0 in the hole. The training-free method works with any model. It overwrites the known part at every step with a noised copy of the original.

📐 Inpainting by replacement, with resampling

Start from $\mathbf{z}_1 \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and walk a decreasing grid. For each step from $t$ to $t' < t$ , take the model's step to get $\mathbf{z}^{\text{gen}}_{t'}$ . Noise the known image afresh to the new level, $\mathbf{z}^{\text{known}}_{t'} = (1 - t')\mathbf{z}^{\text{known}}_0 + t'\boldsymbol\epsilon$ , and combine the two,

\mathbf{z}_{t'} = \mathbf{m}\odot\mathbf{z}^{\text{known}}_{t'} + (\mathbf{1} - \mathbf{m})\odot\mathbf{z}^{\text{gen}}_{t'} .

Resampling (Lugmayr et al. 2022) then noises $\mathbf{z}_{t'}$ back up to $t$ with the forward process, $\mathbf{z}_t = \kappa\,\mathbf{z}_{t'} + \sqrt{t^2 - \kappa^2t'^2}\,\boldsymbol\xi$ , with $\kappa = (1 - t)/(1 - t')$ and fresh $\boldsymbol\xi \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ . It repeats the step, so each level is visited $r$ times in all before moving on. At $t' = 0$ the known region equals the input exactly.

The re-noising formula is the forward process run from $t'$ to $t$ . Given $\mathbf{z}_0$ , it maps $\mathcal{N}((1 - t')\mathbf{z}_0, t'^2\mathbf{I})$ to a Gaussian with mean $\kappa(1 - t')\mathbf{z}_0 = (1 - t)\mathbf{z}_0$ and variance $\kappa^2t'^2 + t^2 - \kappa^2t'^2 = t^2$ , which is the forward distribution at $t$ . The square root is real because $t(1 - t') \ge t'(1 - t)$ when $t \ge t'$ .

Why is replacement alone not enough? §34.5 worked this out for extending a video. The model's update of the hole moves toward $\mathbb{E}[\mathbf{z}^{\text{hole}}_0\mid\mathbf{z}_t]$ , the best guess given the whole noisy image. Correct inpainting needs $\mathbb{E}[\mathbf{z}^{\text{hole}}_0\mid\mathbf{z}_t, \mathbf{z}^{\text{known}}_0]$ , the best guess given the clean known region too. The freshly noised known region carries only part of that information. Resampling gives the model several chances at each level to reconcile the two parts, at $r$ times the calls. RePaint, the method that introduced it, used 250 steps, $r = 10$ , and jumps of 10 steps back and forth.

A smooth 1D Gaussian signal with a hole in the middle shows how far each method gets. For Gaussian data the exact conditional distribution of the hole is known, so we can score each method against the right answer.

Python · cell 9

Python · cell 10

Replacement alone misses the right answer by 0.61 on average. It also leaves the hole far too variable, with standard deviation 0.74 where the truth is 0.26, the signature of a fill that ignores its surroundings. Four passes per level cut the error to 0.38, and ten passes cut it to 0.29, at 4 and 10 times the calls. A model that sees the mask and the known region as inputs learns the conditional directly. With an exact model its error is essentially zero, at one times the calls.

That is why production systems train dedicated inpainting models. Stable Diffusion's inpainting model reads 9 input channels. Four hold the noisy latent, four the encoded masked image, and one the mask, shrunk to the latent grid. The weights of the 5 new channels were initialized to zero on top of the ordinary checkpoint. It was trained for 440,000 steps on synthetic masks, a quarter of which masked everything, so it remains a text-to-image model.

Interview

Inpainting by replacement, with resampling

Implement inpaint_replace(x_known, mask, denoise, S, rng, resample=0, trace=None) for rows of 1D signals. Start from rng.standard_normal(x_known.shape) at $t = 1$ and walk np.linspace(1, 0, S + 1). For each step from a to b, make resample + 1 passes, so resample is $r - 1$ in the definition's terms. Each pass does four things.

Compute $\hat{\mathbf{x}}_0$ = denoise(z, a).
Take the DDIM step $\mathbf{z}^{\text{gen}} = (1 - b)\hat{\mathbf{x}}_0 + b\hat{\boldsymbol\epsilon}$ , with $\hat{\boldsymbol\epsilon} = (\mathbf{z} - (1 - a)\hat{\mathbf{x}}_0)/a$ .
Noise the known signal to level b and combine it with $\mathbf{z}^{\text{gen}}$ through the mask.
Except after the last pass, re-noise back up to level a.

If trace is a list, append a copy of the state at the end of each step. Draw every random number from rng.

import numpy as np

def inpaint_replace(x_known, mask, denoise, S, rng, resample=0, trace=None):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Two practical details decide how an inpainting result looks. First, masks are usually drawn by a segmentation model, today a promptable one such as SAM (§27.6). They are shrunk by the autoencoder's factor $f = 8$ to the latent grid, so the latent mask is blocky and cannot follow the drawn outline exactly. Second, the decoder re-renders the whole image, so even the known region changes slightly after the round trip through the latent space. That is why tools commonly paste the original pixels back outside the mask and blend across the boundary. Outpainting, extending an image beyond its border, is the same problem with the mask on the outside.

Measuring an edit

An edit has two jobs. It must do what was asked, and it must leave everything else alone. Every metric measures one of the two, and the tools of §34.6 split along that line. Two embedding models appear throughout. CLIP maps images and captions into one space where matching pairs point the same way (§30.2). DINO is an image encoder trained without labels (§28.4).

Personalization. DreamBooth measured subject fidelity as the average cosine similarity between embeddings of generated and real images of the subject, with DINO and CLIP image embeddings. It measured prompt fidelity with the CLIP score between each generated image and its prompt. The authors argued for DINO because, trained without labels, it keeps the differences between two subjects of the same class, which CLIP's embedding tends to merge.
Instruction editing. InstructPix2Pix plotted consistency with the input image (CLIP image similarity) against agreement with the instruction. The second is CLIP directional similarity, the cosine between the change in image embeddings and the change in caption embeddings, $\cos\big(E_I(\mathbf{x}_{\text{edit}}) - E_I(\mathbf{x}), E_T(\mathcal{P}_{\text{out}}) - E_T(\mathcal{P}_{\text{in}})\big)$ , where $E_I$ and $E_T$ are CLIP's image and text encoders. Sweeping $s_I$ traces a curve, and a method is better only if it is better at matched consistency.
Preservation. For a local edit, compare the unedited region of input and output directly. PSNR measures the pixel error on a log scale, SSIM compares local structure, and LPIPS is a distance between deep features, all computed outside the mask. For faces, add the similarity of an identity embedding. FLUX.1 Kontext added multi-turn consistency, whether a character survives several edits in a row without drifting.
Judgment. As for generation, the final word is human preference on a fixed benchmark, with the statistical care of §34.6. KontextBench is one, with 1,026 image-prompt pairs over local edits, global edits, character reference, style reference, and text editing.

Provenance, watermarks, and their limits

Everything above makes it easier to produce a convincing image of something that never happened. Two questions follow. Was this image generated, and has it been altered?

The two directions are not symmetric. If a provider marks everything its model makes, the provider can later show, with a secret key, that an image came from its model. No one can show from the pixels that an image is real, because a missing mark proves nothing. The image may come from another model, from an open model that marks nothing, or have had its mark removed. Proving where something came from takes signed records attached to it, which is what content credentials do.

Watermarks inside the generator

A watermark is a signal hidden in an image that a detector with the right key can read and a viewer cannot see. The first design builds it into the model. Stable Signature (Fernandez et al. 2023) works in two stages. First it trains a watermark extractor network to read a $k$ -bit message from images, robustly to crops and compression. Then it fine-tunes the latent decoder with two losses, a perceptual loss that keeps images unchanged and a message loss through the frozen extractor. Afterwards every image the decoder produces carries one fixed signature.

Fine-tuning took about a minute on one GPU and saw under 500 images. The mark lives in the decoder's weights, so every image a released model makes carries it with no extra step. Removing it takes replacing or retraining the decoder. Detection is a hypothesis test, and the threshold is a calculation you can do.

✏️ How many bits must match

Stable Signature uses $k = 48$ bits. For an image the model did not make, assume each extracted bit matches the signature independently with probability $\tfrac12$ . Then the number of matches $M$ is $\operatorname{Binomial}(48, \tfrac12)$ , with mean 24. Flag an image when at least $\tau$ bits match. The false positive rate is the binomial tail $P(M \ge \tau) = 2^{-48}\sum_{j\ge\tau}\binom{48}{j}$ . With $N$ signatures to check, one per user, it is $1 - (1 - P(M \ge \tau))^N \approx N\,P(M \ge \tau)$ . The cell finds the threshold for a false positive rate of $10^{-6}$ .

Python · cell 11

The thresholds, 41 matching bits for one signature and 44 for a thousand, are the ones the paper reports. At that false positive rate it detected about 90 percent of generated images even after they were cropped to 10 percent of their area.

The second design needs no training. Tree-Ring (Wen et al. 2023) writes a key into the Fourier transform of the initial noise. The key is constant on rings around the zero frequency, so the pattern survives rotations. The unmodified sampler then runs as usual.

To detect the key, the provider inverts a suspect image to its noise by DDIM inversion and looks for the rings there. It inverts with the empty prompt, since the original prompt is unknown. So detection inherits every inversion error this section measured. Its statistics must be calibrated on the noise that inversion actually returns. The authors note that the inverted noise of natural images is close to Gaussian and derive a $p$ -value from that. They used 50 steps, a ring radius of 10, and guidance 7.5. The ring version then reached an AUC of 1.000 on clean Stable Diffusion images and 0.975 averaged over a set of attacks. The AUC is the area under the ROC curve, 1 for a perfect detector and 0.5 for a coin flip (§0.11).

Research

Detecting a Tree-Ring key

Build the detector side of a Tree-Ring watermark. Work on a $32\times32$ noise array through its centered spectrum, F = np.fft.fftshift(np.fft.fft2(x, norm="ortho")), in which white noise has complex coefficients of unit average power. The setup gives ring_mask(), the frequencies at radius 1 to 8 from the center, and ring_key(), a real key that is constant on each ring. Implement three functions.

embed_ring(noise, key, mask) overwrites the masked coefficients with the key and returns the real array.
ring_score(noise_hat, key, mask) returns the matched-filter statistic $\zeta$ below, which is standard normal for clean noise.
detect(noise_hat, key, mask, alpha) is a one-sided test at false positive rate $\alpha$ .

\zeta = \frac{\sum_{i\in\mathcal{M}}\operatorname{Re}\big(\bar k_i\,y_i\big)}{\sqrt{\sum_{i\in\mathcal{M}}|k_i|^2}}, \qquad y = \text{the centered spectrum of the recovered noise}.

Here $\mathcal{M}$ is the set of masked frequencies and $k_i$ the key's coefficient at frequency $i$ . The tests feed you noise recovered through a lossy inversion that keeps only 30 percent of the signal.

import numpy as np

def embed_ring(noise, key, mask):
    raise NotImplementedError

def ring_score(noise_hat, key, mask):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Watermarks applied afterwards, and what they cannot do

SynthID-Image (Gowal et al. 2025) takes the other route. It is a post-hoc watermark, applied to an image after generation and independent of the model. The paper documents what deploying one at internet scale requires. Quality is judged largely by human studies. The mark must survive everyday transformations and resist a range of attackers. And quality, robustness, and the number of hidden bits trade off against each other. SynthID has marked over ten billion images and video frames. Google states that all media generated by its tools, including Nano Banana Pro, carry it, and it offers a verification tool in the Gemini app.

📌 Watermarks are not detectors

A watermark lets its owner show that a marked image came from its system. It cannot show that an unmarked image is real, and it can be attacked. Saberi et al. 2023 studied watermarks that change an image only slightly. For such watermarks they proved a trade-off under diffusion purification, an attack that noises the image and denoises it. Any detector then suffers a high rate of evasion, marked images that pass as unmarked, or a high rate of spoofing, unmarked images flagged as marked. They also removed such watermarks in practice with small changes to the image. Watermarks with larger changes resisted purification but fell to an adversarial attack that uses a substitute model. And black-box access to a watermarking method was enough to spoof it, making real images register as watermarked.

Removal by regeneration is SDEdit at a small strength. So the noising argument of this section is also the main attack on invisible watermarks. The practical consequence is defense in depth (§19.7), several imperfect layers instead of one. Watermarks raise the cost of passing off a provider's images as real. Signed provenance records the history of content that carries it. Neither replaces judgment about content that carries neither.

Signed provenance with C2PA

C2PA, the Coalition for Content Provenance and Authenticity, publishes the standard for content credentials. Its central object is the manifest.

📐 C2PA manifest

In the C2PA specification (version 2.1), a manifest attached to an asset holds assertions, statements about the asset. They say which tool created it, with the digital source type "trainedAlgorithmicMedia" for AI-generated media. They say which actions edited it, and which earlier assets it was made from, called its ingredients. The assertions are bound into a claim, which the signer signs with an X.509 certificate, the standard format for public-key certificates, producing the claim signature. A hard binding, a cryptographic hash (SHA-256 or longer) of the asset's bytes, ties the claim to the exact file. A soft binding, a fingerprint or an invisible watermark, lets a manifest stored elsewhere be found again for a copy that lost it. Validation checks the signature against trusted certificates and recomputes the hash.

In plain words, a manifest is a signed receipt that travels with the file. It proves who signed the stated history and that the file has not changed since. It says nothing about a file that does not carry one. Stripping the metadata removes it. That is why the specification allows manifests to be stored externally and recovered through a soft binding, and why watermarks and credentials complement each other. A manifest also moves trust to the signer. A valid signature from a camera maker or a model provider is only as meaningful as the signer's honesty and key security.

Erasing a concept from the weights

Watermarks label outputs. A provider may also want a model that cannot make certain things at all. Filtering the training data is the upstream fix, but it is expensive and imperfect. Stable Diffusion 2.0 was trained on data filtered to remove explicit images. Yet it still produced 417 images that a nudity detector flagged, on the 4,703 prompts of the Inappropriate Image Prompts benchmark. SD 1.4 produced 796, as measured by Gandikota et al. 2023.

Their method, Erased Stable Diffusion (ESD), fine-tunes the model against its own guidance. For a concept prompt $c$ , it trains $\boldsymbol\epsilon_\theta(\mathbf{z}_t, c, t)$ toward

\boldsymbol\epsilon_{\theta^\ast}(\mathbf{z}_t, t) - \eta\big[\boldsymbol\epsilon_{\theta^\ast}(\mathbf{z}_t, c, t) - \boldsymbol\epsilon_{\theta^\ast}(\mathbf{z}_t, t)\big],

where $\theta^\ast$ is the frozen original and $\eta > 0$ sets the strength. That target is classifier-free guidance with scale $\gamma = -\eta$ , pointing away from the concept, distilled into the weights. So prompting for $c$ now steers away from it. Fine-tuning only the cross-attention weights (ESD-x) erases what a specific prompt calls up, such as an artist's style. Fine-tuning the other weights (ESD-u) erases a concept however it is worded, such as nudity. On the same benchmark, ESD cut the 796 flagged images of SD 1.4 to 134.

Like every safety fine-tune, erasure is measured, not guaranteed. Against reproducing training images, the memorization measurements of §34.6 and deduplication of the training data remain the first line of defense.

Choosing a method

Goal	First choice	What it costs	Watch for
Follow a sketch, pose, or depth map	ControlNet	42 percent more weights, every step	text and control disagreeing at high conditioning scale
Same, cheaply, or several conditions at once	T2I-Adapter	77M weights, run once per image	weaker control, since it ignores the noise level
Use an image as a prompt (style, object)	IP-Adapter	22M weights	identity details lost in one embedding
Put a specific subject in new scenes	DreamBooth with LoRA, prior preservation	minutes per subject, small file	drift and lost variety without the prior term, stacking interference
Keep the layout of a painting, regenerate the rest	SDEdit	nothing, fewer steps than sampling	the realism and faithfulness trade-off set by $t_0$
Change one thing in a real photo, open model	null-text inversion and Prompt-to-Prompt	about a minute per image	prompts that do not share structure
Change one thing by instruction	an instruction or in-context editing model	one generation, longer sequence	drift over many turns
Replace a region	a mask-conditioned inpainting model	one generation	seams and decoder changes outside the mask
Show that images came from your model	watermark at generation, plus C2PA credentials	small quality cost, key management	removal by regeneration, spoofing, stripped metadata

In PyTorch

The block checks the two structural facts of this section with autograd. The first is ControlNet's zero start around a small convolutional block. The second is that IP-Adapter recovers the base layer exactly at $\lambda_{\text{ip}} = 0$ , while concatenation changes the text path.

🔥 In PyTorchfor a machine with PyTorch

With the diffusers library, the production versions of these pieces take a few lines. The block is shown, not run, since it downloads models.

⚠️ Where control and editing go wrong

Initializing a ControlNet's copy randomly instead of from the base throws away the reason it works. So does sharing the tensor instead of copying it, since training the copy then trains the locked model too.
Adding the control to both branches of guidance cancels it when there is no prompt.
Running DreamBooth without the prior term, or for too many steps, makes "a dog" mean your dog and collapses the poses.
Comparing SDEdit strengths across libraries compares different edits, unless you check whether each rounds $t_0S$ up or down and whether it noises to $t_0$ exactly or to the nearest grid level.
Inverting with one guidance scale and generating with another, then blaming the model for a poor reconstruction, misreads the round-trip theorem. Two different ODEs never meet.
In inpainting, pasting clean known pixels at intermediate steps, instead of noising them to the current level, shows the model an image at the wrong noise level and leaves visible seams.
Calibrating a watermark detector on clean noise, rather than on what the inversion actually returns, gives false positive rates that do not hold on real detections.

🧭 What to remember

Control methods add an input path, change a few parameters, or change the sampling around a fixed model. The best input paths start at exactly zero, so the model begins as the base model. ControlNet's zero convolutions also open from the output side, since only the output layer learns on the first step, and LoRA's $\mathbf{B} = \mathbf{0}$ does the same. The same zero start appears in adaLN-Zero and in the zero-initialized input channels of editing and inpainting models.
ControlNet copies the encoder and middle block (361M parameters for SD 1.5) and runs at every step. T2I-Adapter (77M) runs once per image. IP-Adapter (22M) adds a separate image cross-attention whose weight $\lambda_{\text{ip}} = 0$ recovers the base exactly, unlike concatenation, which blends text and image by the query's scores.
Personalization trades fidelity for size. Textual inversion learns 768 numbers, DreamBooth all the weights with a prior-preservation term against language drift, Custom Diffusion the cross-attention keys and values, and LoRA a low-rank update that is the usual compromise.
SDEdit keeps a fraction $\rho(t_0) = s(1 - t_0)/\sqrt{s^2(1 - t_0)^2 + t_0^2}$ of the guide's deviation in each direction. So the strength trades faithfulness for realism, keeping coarse structure and regenerating detail.
DDIM inversion's round-trip error is first order in the step size and grows with guidance, and inverting at $\gamma = 1$ then generating at $\gamma > 1$ never converges. Null-text inversion keeps the unguided path as a pivot and optimizes per-step null embeddings, and Prompt-to-Prompt edits by reusing cross-attention maps.
Instruction and in-context editing replaced inversion tricks, from InstructPix2Pix with two guidance scales to FLUX.1 Kontext with context tokens and FLUX.2 and Nano Banana Pro with many references. Mask-conditioned models replaced inpainting by replacement, which misses the right conditional even with an exact model.
Watermarks let a provider prove its images are generated, never that an image is real, and they can be removed by regeneration or spoofed. C2PA credentials prove a file's signed history and nothing about files without one. The same pattern, a compact representation and a generator conditioned on it, returns with the same provenance problem for sound, in the codec models of §37.3 and §37.4, and for voices in §37.6. Conditioning video generation on actions instead of text turns these generators into the action-conditioned world models that let an agent imagine the consequences of what it does.

Zhang, Rao & Agrawala (2023). Adding Conditional Control to Text-to-Image Diffusion Models. ControlNet, with the zero-convolution argument, the ablations that justify the copy, and the sudden convergence phenomenon.
Ye et al. (2023). IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models. Decoupled cross-attention, its 22M-parameter budget, and the comparison with concatenation.
Ruiz et al. (2022). DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. Rare-token identifiers, prior preservation against language drift, and the DINO subject-fidelity metric.
Meng et al. (2021). SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. Editing by noising and denoising, with the realism and faithfulness trade-off measured and bounded.
Mokady et al. (2022). Null-text Inversion for Editing Real Images using Guided Diffusion Models. Why guidance breaks DDIM inversion, pivotal inversion, and per-step null embeddings, used with Prompt-to-Prompt.
Brooks, Holynski & Efros (2022). InstructPix2Pix: Learning to Follow Image Editing Instructions. A synthetic editing dataset from GPT-3 and Prompt-to-Prompt, zero-initialized input channels, and guidance with two conditions.
Black Forest Labs (2025). FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space. Editing by appending clean context tokens with a positional offset, and the KontextBench benchmark.
Saberi et al. (2023). Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks. The evasion and spoofing trade-off for watermarks under diffusion purification, and attacks on stronger watermarks.

Made for larger screens

Control, personalization, and editing

Three ways to steer a frozen generator

ControlNet, a trainable copy behind two zero layers

Why the zero layers do no harm and still learn

A ControlNet block and its gradients

Training and using a ControlNet

Lighter adapters for structure and for image prompts

Teaching the model one new subject

Textual inversion, one new word

Textual inversion on a frozen linear denoiser

DreamBooth, fine-tuning with the model's own samples as a guard

Custom Diffusion and LoRA, fine-tuning less

Editing a real image by noising it

SDEdit with an exact denoiser

Inverting an image, and why guidance breaks it

Null-text inversion, bending the guided path back

Inversion error against guidance

Editing by steering attention

Editing by instruction and in context

Inpainting, filling a masked region

Inpainting by replacement, with resampling

Measuring an edit

Provenance, watermarks, and their limits

Watermarks inside the generator

Detecting a Tree-Ring key

Watermarks applied afterwards, and what they cannot do

Signed provenance with C2PA

Erasing a concept from the weights

Choosing a method

In PyTorch

Read more

Watch next