Text conditioning is how a sentence steers an image generator. A diffusion model makes an image by starting from noise and removing a little of it at each of many steps, and at every step it is told what the prompt says. Three things decide how well it follows the prompt. A text encoder turns the prompt into vectors. A route into the network carries those vectors to every part of the image. And the captions of the training images taught the network what those vectors mean. Most of the familiar failures of image generators come from one of these three stages, not from the network that draws.

Here is a concrete case. Give Stable Diffusion 1.5 the prompt "a red cube on top of a blue sphere, next to a sign that says OPEN LATE" and look at a few samples. The colors often swap or spread, so that both objects come out reddish. The cube sits beside the sphere instead of on it. Sometimes one object is missing altogether. And the sign carries letter-like marks that spell nothing. Yet the same model draws a red cube, or a blue sphere, without trouble when asked for one thing at a time. So the failure lies on the path from the sentence to the network, and in the sentences the network was trained on. Chefer et al. (2023) named two of these failures. A missing object is catastrophic neglect, and a color that lands on the wrong object is a failure of attribute binding.

The network that draws is the denoiser, a U-Net or a transformer built in §34.2. It does not work on pixels. It works on a latent, a small grid of numbers that an autoencoder made from the image (§34.1), for example $64\times64\times4$ numbers for a $512\times512$ photograph. The denoiser cuts the noisy latent into image tokens, reads them together with the noise level $t$, and predicts how to remove the noise. A condition can enter it in two places. One is a modulation, a scale and a shift computed from a single vector and applied to the features of every block. The other is an attention layer that reads a sequence of vectors. This section fills both places with a sentence.

We start by following one prompt into Stable Diffusion 1.5 with every shape counted. A four-token example worked by hand then shows why a color word can reach the wrong object. Then we take the three stages in turn, the encoder, the route, and the captions, with the engineering each one needs. That engineering covers padding, guidance with several encoders, extra inputs that describe the training data, spelling, and the attention maps that diagnose binding.

One prompt, followed into the network

A text encoder does not read letters. It reads token IDs, integers that index a fixed vocabulary of word pieces (§2.2). Stable Diffusion 1.5 uses the text half of CLIP, a model trained to match images with their captions (§30.2). Its tokenizer lower-cases the prompt and splits it into pieces from a vocabulary of 49,408. Every word of "a red cube on top of a blue sphere" is common enough to be one piece, so the prompt is 9 tokens. A start token and an end token are added, which makes 11 positions.

A batch of prompts must form one array, so every prompt is padded to the same length. CLIP's length is 77 positions, and Stable Diffusion 1.5 fills the 66 spare positions with copies of the end token. OpenAI's original CLIP code pads with ID 0 instead (§30.2), but the tokenizer that Stable Diffusion uses pads with the end token. These pads matter more than they seem, and a later part of this section is about them.

The words reach the image through cross-attention (§4.6), attention in which the questions come from one sequence and the answers from another. Here every image token asks a question, its query. Every text token offers a key, which is compared with the query, and a value, which is averaged into the answer. So each image token receives a weighted average of the words, with more weight on the words whose keys match its query better.

§34.1 runs this whole pipeline on a toy problem small enough to read every number. Here we look closely at the text part of it.

The text path is cheap in compute. Projecting the text to keys and values costs 0.44 percent of a U-Net call, so caching them saves about that much. All of cross-attention, projections, scores, and weighted sums together, is 16.0 billion multiply-adds, 4.7 percent of the call. What the text path decides is not cost but content, which word reaches which image token. That is where the failures of the opening come from.

Why a color word can reach both objects

A cross-attention layer gives each image token its own weights over the words. If the key of "red" matches the queries of the cube's tokens and the sphere's tokens equally well, both objects receive the same amount of red. Here is the smallest case that shows it, small enough to check by hand.

The example separates the three stages, and each one limits what the image can follow. The encoder decides whether its vector for "cube" carries "red". The route decides how image tokens find that vector, here one cross-attention layer with nothing to break the symmetry of the red key. And the data decides whether training ever rewarded getting it right, which requires captions that state colors at all.

An encoder can carry "red" into "cube" only if each token's vector depends on the words around it, which is what contextual means. Even then, the direction matters. CLIP's text tower is causal, so each position reads only the words before it (§30.2). The vector at "red" has read only "a red" and cannot know which noun follows. The vector at "cube" has read "a red cube" and can carry the pair. A bidirectional encoder such as T5 lets every position read the whole sentence. Whether a trained encoder actually binds the pair is an empirical question, and for CLIP the answer of §30.6 is often no.

In practice, this is the first thing to ask when a model mixes up attributes. Does the encoder's vector for the noun know its adjective, and did the training data ever reward using it? An interviewer asking "why are the colors swapped?" wants these three stages named, not a guess about the denoiser.

With a full unit of red in "cube", the cube's tokens receive 0.860 and 0.955, but the sphere's still receive 0.424 and 0.232, more than with none. The weak sphere query, token 3, leaks more, because it puts 0.140 of its weight on "cube" against token 4's 0.045. A contextual encoder makes the right object redder. It does not stop the leak, which the route and the data must also fix.

What a text encoder knows

Almost every text-to-image diffusion model since Imagen (Saharia et al. 2022) conditions on a frozen pretrained text encoder, one whose weights stay fixed while the generator trains. GLIDE (Nichol et al. 2022), OpenAI's text-guided diffusion model from just before, was the exception. It trained its own text transformer together with the denoiser.

Freezing has two benefits. The text vectors of the whole dataset can be computed once and stored, so the encoder adds almost nothing to the cost of training. Imagen gives this as a reason to freeze, and eDiff-I, NVIDIA's text-to-image model, trains this way (Balaji et al. 2022). And the encoder's knowledge of language stays intact while the denoiser learns. Encoders differ in what their own training taught them, and the difference shows up in the images.

CLIP's text tower. CLIP was trained to match images with their alt-text, the short descriptions written for web images. Training pulls each image toward its own caption and away from the other captions in a batch (§30.2). Its text tower is a GPT-2-style transformer with a causal mask. It reads lower-cased tokens in a window of 77 positions. The start and end tokens take two of them, so a prompt keeps at most 75 tokens. Its pooled output, one vector for the whole prompt, is the final layer's vector at the end token, projected into the space it shares with images.

Because of that training, CLIP knows what is visually distinguishable, which is what a generator needs for objects and styles. It also inherits every limit of §30.6. On word order and binding it behaves much like a bag of words, an encoder that ignores order. It counts poorly. And in practice it uses fewer than about 20 of its 77 positions (Zhang et al. 2024).

Stable Diffusion 1 feeds the 77 final-layer vectors of CLIP ViT-L/14 (width 768, 123 million parameters) to cross-attention. Stable Diffusion 2 switched to the larger OpenCLIP ViT-H and reads its second-to-last layer, the penultimate layer. SDXL and SD3 read the penultimate layers of their CLIP encoders too. Practitioners believe that the last layer is the one most specialized to the contrastive objective at the end token, so the layer before it keeps more detail per token. That belief has good results behind it, but it is not a measured law.

T5's encoder. T5 (Raffel et al. 2020) is an encoder-decoder language model (§8.1) trained on web text to fill in masked spans. A generator keeps only the encoder. It is bidirectional, so every position reads the whole sentence. It uses a vocabulary of 32,000 pieces. It encodes positions as learned biases on relative distances, so no fixed table of absolute positions caps the prompt length. T5-XXL's encoder has about 4.7 billion parameters (Imagen counts 4.6 billion). In the v1.1 release that SD3 and FLUX.1 use, its width is 4,096. It has never seen an image, but it has read far more text than any caption corpus holds.

Language and vision-language models. The next step replaces T5 with a decoder-only language model, one trained to predict the next token, that has read even more text and often images too. These encoders add world knowledge, long and instruction-like prompts, many languages, and the ability to read reference images for editing (§34.7). They cost memory, and they have one structural problem. A decoder-only model is causal, so the vectors of early tokens never see later words, the same asymmetry as in CLIP's text tower.

Several encoders at once. Nothing forces one encoder. eDiff-I (Balaji et al. 2022) conditions on T5-XXL, CLIP's L/14 text encoder, and a CLIP image embedding together. It reports that CLIP alone gives the right objects but weak composition, counting, and text. T5 alone composes better but renders some objects, such as dog breeds, worse. Both together do best. SD3 uses two CLIP encoders and T5-XXL.

Movie Gen, Meta's video generator (Polyak et al. 2024), uses three encoders with three jobs. UL2, a large T5-style language model, reads the language. A MetaCLIP model, a CLIP trained on curated data and fine-tuned to read 256 tokens instead of 77, adds visual concepts. And ByT5, a T5 that reads raw bytes, reads any text the prompt asks to appear in the video. Each output is projected to the same width with its own linear layer and LayerNorm, and the three are concatenated into one sequence.

The choice of encoder is a design-review question you should be able to defend. CLIP gives visual concepts and a ready pooled vector. T5 gives composition and long prompts. A language model gives knowledge and instructions. Each comes at a memory cost that a later part of this section counts.

Three ways text enters the denoiser

A prompt arrives as two arrays. One is a pooled vector $\mathbf{c}_{\text{vec}} \in \R^{d_{\text{vec}}}$, a single summary of the whole prompt. The other is a sequence of token vectors $\mathbf{C}_{\text{ctx}} \in \R^{T_c\times d_c}$, one for each of the $T_c$ text tokens, each of width $d_c$. Each array has its own way into the network.

In plain words, the pooled vector tells every image token the same thing, roughly what kind of picture this is. Cross-attention lets every image token look up the words it needs. Joint attention also lets the words look back at the image.

Imagen measured the first choice directly. In its base model, conditioning on a pooled vector alone, by mean or by attention pooling, was noticeably worse than attending over the sequence of token vectors (Saharia et al. 2022, Appendix D). A pooled vector cannot say which word goes where. So every modern system keeps the sequence and uses the pooled vector only as a global extra.

Three consequences follow from the shapes.

Caching. In cross-attention the keys and values depend only on the text, which is the same at every one of the $S$ sampling steps. So $\mathbf{C}_{\text{ctx}}\mathbf{W}_K$ and $\mathbf{C}_{\text{ctx}}\mathbf{W}_V$ can be computed once per prompt and reused. In joint attention the text tokens change from block to block and from step to step, so nothing can be cached.

Competition. Cross-attention spends all of its attention on the text by construction. In joint attention the text competes with the image inside each query's softmax. §34.2 proved that an image query's joint-attention output is a blend $\lambda\,(\text{self-attention}) + (1 - \lambda)\,(\text{cross-attention})$. The image share $\lambda$ is the fraction of the query's softmax mass that falls on image keys. At initialization SD3's text gets under 4 percent of each image token's attention, and it must earn more through learned scores.

A hybrid. GLIDE and Imagen's base model append the projected text vectors to the keys and values of the image self-attention. Image queries then attend over image and text keys in one softmax, while the text is never updated. By the same identity, that is the blend of self-attention and cross-attention, without a text stream.

An interviewer will ask you to compare these routes on what they can express and what they cost. The short answer is global and cheap, per word and cacheable, or per word in both directions with nothing cached.

Padding and the 77-token window

Prompts have different lengths, and a batch needs one array, so every prompt is tokenized and padded to a fixed length $T_c$ (77 for CLIP). For prompt $b$, let $\mathbf{m}_b \in \{0, 1\}^{T_c}$ mark its real tokens with 1, start and end tokens included. Whether the pad positions take part in attention is a design choice, and the two options behave very differently. A mask removes the pads by setting their scores to $-\infty$ before the softmax. Without a mask, the pads are attended to like any other token.

Proof. Masked, every pad contributes $e^{-\infty} = 0$ to both the numerator and the denominator of the softmax, so only real tokens remain. Unmasked, split the sum,

With a shared key, $Z_P = n_Pe^{s_P}$, and $\pi$ increases with $n_P$ because $x/(Z_R + x)$ increases with $x$. ∎

In plain words, without a mask a short prompt is a few words diluted in a sea of pads, and how diluted depends on the prompt's length. With a mask, the prompt's influence does not depend on how it was padded.

With 8 real tokens the pads take 0.816 of the attention, close to their 0.896 share of positions. Even with 40 real tokens they take 0.381. Masked, they take exactly nothing.

So should every model mask? Not after the fact. Stable Diffusion 1's tokenizer pads with the end token, and its causal text tower gives every pad a vector that has read the entire prompt, because the pads come after it. The U-Net was trained attending to all 77 vectors with no mask, and the reference pipeline passes none. Its pads are a second, smeared copy of the prompt, and the model learned to use them. Adding a mask at inference would hand the network inputs it never saw in training.

Other models made other choices. SDXL's second tokenizer pads with "!" instead of the end token. FLUX.1 [dev]'s pipeline pads T5 to 512 tokens and passes no mask to the transformer. T5-based models such as PixArt-α pad to 120 tokens and pass the encoder's mask to cross-attention. Each is a valid convention, and the rule is the same for all of them. Sample with the same padding token, the same length, and the same mask the model was trained with.

The window is a second trap. A CLIP prompt longer than 75 tokens is cut off, and the reference pipelines only print a warning. There are three remedies, from cheapest to most thorough.

• Encode a long prompt in chunks of 75 tokens and concatenate their vectors. This practitioner trick works because cross-attention accepts any $T_c$, though the pooled vector still sees only one chunk.
• Stretch CLIP's position embeddings and fine-tune it on long captions. Long-CLIP does this to reach 248 tokens (Zhang et al. 2024), and Movie Gen's MetaCLIP to reach 256.
• Use an encoder without a short window, such as T5 or a language model.

Interviewers like this topic because the bug is silent. A model sampled with the wrong mask or the wrong pad token still makes images, just worse ones, and only a comparison with the training code catches it.

import numpy as np

def masked_cross_attention(X, C, keep, Wq, Wk, Wv):
    """X (B, T, d), C (B, T_c, d_c), keep (B, T_c) bool. Returns O (B, T, d_v) and A (B, T, T_c)."""
    raise NotImplementedError

Training one model for guidance, with several encoders

Classifier-free guidance (§33.5) makes a model follow its prompt more strongly. At every step it computes two predictions, one with the prompt and one without, and moves further in the direction the prompt adds. With velocity prediction, the network $\mathbf{v}_\theta$ predicts the direction in which the noisy latent $\mathbf{z}_t$ moves at time $t$, and guidance uses

In plain words, start from the prediction without the prompt, then add $\gamma$ times what the prompt changes. Here $c$ is the condition, $\varnothing$ is the null condition meaning "no prompt", and $\gamma$ is the guidance scale, with $\gamma = 1$ meaning no guidance.

One network learns both predictions, because training replaces $c$ by $\varnothing$ with some probability $p_{\text{uncond}}$, typically 0.1 to 0.2. Exercise 33.5b trains this for a class label. A prompt raises two questions that a label hides. What exactly is $\varnothing$ for a text encoder? And what should happen when there are several encoders?

The null condition is part of the model. Implementations differ, and each model must be sampled with the null it was trained with. Stable Diffusion 1 dropped its text conditioning 10 percent of the time, according to its model card. Its pipeline encodes the empty prompt "" for the unconditional branch. That array is far from zero, because it still holds the start and end tokens and the pads that follow them. SDXL's reference pipeline instead uses all-zero vectors when no negative prompt is given (its configuration flag force_zeros_for_empty_prompt).

Others differ again. Imagen zeroes the text embeddings. GLIDE, fine-tuned for guidance after its main training run, replaced 20 percent of its token sequences with the empty sequence. eDiff-I and SD3 zero the whole output of a dropped encoder. A negative prompt (§33.5) replaces $\varnothing$ by the encoding of another sentence, such as "blurry", which guidance then steers away from, at no extra cost.

Several encoders can be dropped separately. If each encoder is dropped on its own coin flip, the model learns to work with every subset of them. That is what lets SD3 run without its largest encoder at inference.

Proof. The events "encoder $i$ is kept" are independent. So the probability of a pattern of kept and dropped encoders is the product of the individual probabilities, $1 - q_i$ for each kept one and $q_i$ for each dropped one. The unconditional and full cases are $\mathcal{K} = \varnothing$ and $\mathcal{K} = \{1, \dots, m\}$. With equal rates, every subset of size $k$ has probability $(1 - q)^kq^{m-k}$, there are $\binom{m}{k}$ of them, and $q^m = u$ gives $q = u^{1/m}$. ∎

For a fixed unconditional rate, no choice of independent rates gives the full configuration a larger share than equal rates do, as the derivation below shows. So a model that must run on subsets pays with fewer training steps on the full set. At the same $u$, the only way to buy them back is to drop all the encoders together part of the time, instead of always independently.

The cell checks the theorem against 200,000 simulated training examples with SD3's rates.

The widget draws the share of training examples that see each subset of the three encoders, starting at SD3's rate with independent coins. Before you switch to one shared coin for all three, predict what happens to the unconditional rate, and how often the model then trains with T5 missing but both CLIP encoders present. Then set the rate to 10 percent with independent coins, and ask whether the model is still trained for guidance.

import numpy as np

def drop_conditions(embs, nulls, probs, rng):
    """embs: list of m arrays, embs[i] of shape (B, ...). nulls[i] has shape embs[i].shape[1:].
    Returns (out, kept) with out a list of m new arrays and kept a (B, m) bool array."""
    raise NotImplementedError

def uncond_fraction(probs):

import numpy as np

def drop_prob_for_target(u, m):
    """The equal per-encoder drop probability that makes the unconditional rate u with m encoders."""
    return u / m

def presence_distribution(q):
    """q: drop probabilities of m encoders. Returns P(exactly k encoders present), k = 0..m."""

Guidance at sampling time, batched

Training gives the network both predictions. Sampling must compute both at every step. The bookkeeping takes a few lines, and those lines are easy to get wrong.

Guidance-distilled models such as FLUX.1 [dev] remove the doubling. They take $\gamma$ as an input and produce the guided prediction in a single call (§33.5). And how much does the text encoder itself cost? Almost nothing in compute and a great deal in memory, as the next cell counts for SD3.

import numpy as np

def cfg_sample(prompts, z1, S, gamma, enc, net, negative=""):
    """prompts: list of B strings. z1: (B, D) noise at t = 1. Returns z at t = 0 after S Euler steps
    on t = linspace(1, 0, S + 1), guided by gamma against the negative prompt."""
    raise NotImplementedError

Captions decide what can be followed

Better encoders and richer routes help only if the training data taught the denoiser what the encoder's vectors mean. The cleanest way to see this limit is to imagine a perfect model, one that has learned the training data exactly.

Proof. The model produces $a^\star$ with probability $\int \mathbb{1}[a(\mathbf{x}) = a^\star]\,p_{\text{data}}(\mathbf{x}\mid c)\,d\mathbf{x} = p_{\text{data}}(a = a^\star\mid c)$, by the definition of the conditional distribution of a function of $\mathbf{x}$. Independence of $a$ and $c$ means $p_{\text{data}}(a\mid c) = p_{\text{data}}(a)$ for every $c$. ∎

In plain words, a model learns what "red" means only from images whose captions say "red" and are right about it. Real models are not exact, so the theorem describes a ceiling. For web alt-text the ceiling is low. Alt-text is written for search engines and screen readers. It names the main subject and usually leaves out the background, the layout, the colors of secondary objects, and any text in the image (Betker et al. 2023, Esser et al. 2024). PixArt-α (Chen et al. 2023) counted 6.4 nouns per image in raw LAION captions, and 20.9 after the vision-language model LLaVA rewrote the captions of the same images.

Recaptioning replaces poor captions with better ones written by a model. DALL-E 3 (Betker et al. 2023) trained an image captioner and fine-tuned it twice. One version writes short captions of the main subject, and the other long descriptive captions of the whole scene.

Training only on synthetic captions has a hazard. A generative model fits every regularity of its text. If the captioner always ends with a period, always starts with "a", or always writes 60 words, the image model will work well only on prompts that look like that. So DALL-E 3 picks either the synthetic caption or the original for each image at data-loading time, with a fixed probability.

The DALL-E 3 team compared identical T5-conditioned models trained with 65, 80, 90, and 95 percent descriptive captions. They scored each with CLIP score, the cosine similarity between CLIP's embeddings of a generated image and of a caption. They dropped the 65 percent run midway because it was far behind, and found that a higher synthetic share always gave a higher CLIP score against the original captions. DALL-E 3 itself trained on 95 percent synthetic captions.

Prompt upsampling fixes the mismatch this creates. A model trained on long descriptive captions is outside its training distribution on a user's short prompt. The DALL-E 3 report has GPT-4 rewrite each short prompt into the captioner's descriptive style before sampling. It is the padding rule again, applied to whole sentences. Sample with inputs drawn from the distribution you trained on. Qwen-Image's system prompt, which asks the encoder to describe colors, quantities, text, and spatial relations, pushes the encoder's representation the same way from inside.

The captioner's blind spots become the generator's. DALL-E 3's report lists three. Its captioner was unreliable about spatial words such as "to the left of" and "underneath", and so was the image model. The captioner made up plant and bird species, and the image model became unreliable on species names. And the captioner transcribed words in images well enough for DALL-E 3 to render text, but the rendered words had missing or extra letters. The authors suspected T5's tokens, a failure taken up below in the part on spelling.

In practice, recaptioning is usually the largest single gain in prompt following a team can buy. The question an interviewer asks next is the cost. Users must be served through a prompt rewriter, and the model inherits every error of the captioner.

Micro-conditioning on size, crop, and frame rate

Data pipelines make choices the captions never mention. Images smaller than the training resolution are thrown away or upscaled. Every image is randomly cropped to a square. Videos are resampled to some frame rate. If the model cannot see these choices, their effects leak into its samples as blur, cut-off heads, and wrong motion speeds. SDXL's fix is to tell the model, through extra inputs called micro-conditioning.

In plain words, a side effect of the data pipeline becomes a setting. The model learns "this image was originally 300 pixels tall and upscaled" as a fact about its input. So it can produce a sharp image when told the original was large.

SDXL (Podell et al. 2023) used three such conditions.

Size. Throwing away every image with a side shorter than the 256-pixel pretraining resolution would have discarded 39 percent of SDXL's data, while upscaling them teaches blur. Conditioning on the original size $c_{\text{size}} = (h_{\text{orig}}, w_{\text{orig}})$ keeps all of the data. A controlled check on class-conditional ImageNet at 512 pixels measured FID-5k, where lower is better. Discarding the images below 512 pixels left 70,000 images and gave 43.84. Keeping all images without conditioning gave 39.76, and size conditioning gave 36.53.

Crop. Stable Diffusion 1.5 and 2.1 often drew objects cut off at the frame, such as heads missing their tops, which SDXL traced to random cropping in training. So SDXL records the offsets $c_{\text{crop}} = (c_{\text{top}}, c_{\text{left}})$ of each random crop in training and conditions on them. Inference then asks for $(0, 0)$, an uncropped, centered composition.

Aspect ratio. A final fine-tuning stage groups images into buckets of different aspect ratios with about $1024^2$ pixels each, and conditions on the bucket's size $c_{\text{ar}} = (h_{\text{tgt}}, w_{\text{tgt}})$.

Video adds a frame rate. Movie Gen (Polyak et al. 2024) chose the other route into the network and wrote the rate into the prompt. It prepends a token such as "FPS-16" to every caption. In pretraining it samples clips at their original rate, at least 16, and in fine-tuning at 16 or 24. The two routes trade off predictably. A sinusoidal embedding is exact and cheap, and a text token can be added to any text-conditioned model without changing the architecture. §34.5 meets both.

In an interview, micro-conditioning is the answer to "how do you use low-resolution or cropped data without the model learning blur or cut-off heads?" Turn the defect into an input, then choose its value at inference.

Why generators misspell

A subword tokenizer (§2.2) turns a common word into one ID. The denoiser can draw the letters of that word only if the encoder's vector for the ID already knows them. Return to the opening prompt. CLIP lower-cases "OPEN LATE" to "open late", two tokens with IDs 1488 and 2325. Nothing in those two integers says O, P, E, N. And nothing in contrastive matching or span filling forces the encoder to learn the letters.

Liu et al. (2022) measured this directly with a benchmark they call WikiSpell. They froze each text encoder and fine-tuned the decoder that comes with it to spell each word from the encoding. With its encoder frozen, T5-XXL spelled only 66 percent of the most common 1 percent of English words. In that bucket 87 percent of words are a single T5 token, whose letters are hidden. ByT5, which reads UTF-8 bytes (§2.3), spelled 94 to 99 percent at every size and every frequency. Among models that read word pieces, near-perfect spelling appeared only at the scale of PaLM. Its 540-billion-parameter version, prompted with 20 examples instead of fine-tuned, reached 99 to 100 percent in every frequency bucket.

The gap carried over to images. Liu et al. trained image models that differed only in their text encoder, and checked the rendered words with text recognition, counting a word as right only if every letter was. Encoders that read characters rendered words correctly more than 25 points more often than T5 on frequent words, and more than 30 points more often on rare ones. The gain held even for a ByT5-XL encoder 43 percent smaller than T5-XXL. Adding a small ByT5 encoder of 220 million parameters beside T5-XXL, a 4.8 percent increase in encoder size, was enough to make the model read characters.

The production systems agree. DALL-E 3's authors suspected their T5 encoder for their misspellings. SD3 lost most on written text when T5 was removed (the 38 percent win rate above). And Movie Gen routes only the quoted visual text of a prompt through ByT5.

Spelling is also a data problem, by the captions theorem. A model renders "OPEN LATE" only if training captions transcribed the words in their images. In a random sample of 100 LAION images, Liu et al. found text in about 71 percent. In about 60 percent the caption's text matched the visible text. Recaptioning can strengthen that signal or destroy it, depending on whether the captioner reads. Qwen-Image builds text rendering by a curriculum. It starts from images without text and grows to paragraph-level text, with synthetic rendered data in English and Chinese.

So if a product needs legible text in images, the fix is an encoder that sees characters, plus captions that transcribe text. Bigger denoisers alone do not solve it, and an interviewer will expect you to say why.

Binding, neglect, and reading attention maps

The failures of the opening example have names. Neglect is an object of the prompt that never appears, as when "a cat and a frog" gives only a cat (Chefer et al. 2023). Attribute mis-binding is a modifier on the wrong object, as when "a pink sunflower and a yellow flamingo" gives a yellow sunflower and a pink flamingo (Rassin et al. 2023). Entanglement is two similar objects merging into one.

Tang et al. (2023) found entanglement for co-hyponyms, two nouns of the same kind such as two animals. They also found that descriptive adjectives attend too broadly over the image. Counting and spatial relations fail too, and their encoder side is in §30.6. Each failure traces to the three stages, an encoder that does not bind, a route that lets a word reach the wrong place, and captions that never rewarded getting it right.

Cross-attention gives a direct view of the route, because its weights say where each word is read. This is step 6 of the walkthrough at the start of the section, made precise.

The widget below draws these maps for three prompts, one for each failure. Its "steered" switch shows the maps after the latent is nudged during sampling, the fix described just after it. It opens on the binding prompt, "a red cube and a blue sphere", at step 10. Before reading the numbers, predict how much of the attention of "red" falls on the cube, and how much of "blue" on the sphere. Then pick the neglect prompt, "a cat and a frog", and predict the peak of "frog" as sampled and when steered, and what the image will contain in each case.

The maps in the widget are synthetic, but the patterns are the published ones. They also suggest fixes that need no retraining. During sampling, one can steer the latent so that the maps behave. Attend-and-Excite does this for neglect.

In plain words, find the subject that no patch of the image is attending to, and nudge the noisy latent until some patch attends to it strongly. The smoothing makes the network grow a region rather than one isolated hot cell.

Chefer et al. applied the update only during the first 25 of 50 steps, because later updates no longer moved the layout and caused artifacts. That agrees with eDiff-I's finding that sampling relies on the text early and almost ignores it late. With Stable Diffusion 1.4 at guidance scale 7.5 on an A100 GPU, one image took about 5.6 seconds plain and 9.7 seconds with Attend-and-Excite. It took about 15.4 seconds when extra refinement steps were needed, since every update needs a backward pass through the U-Net.

The same idea fixes binding. Parse the prompt to pair each modifier with its noun, then push each modifier's map to overlap its own noun's map and not the others. That is the method of SynGen (Rassin et al. 2023). It measures overlap with a smooth distance between maps, a symmetric KL divergence (§0.5), rather than the thresholded IoU above, so the loss has a gradient.

These are patches at inference time with real costs, and they need a model whose cross-attention maps can be read and steered. The ablations earlier in this section suggest that most of the lasting gains in prompt following came from the three stages themselves, better encoders and better captions above all. Measuring adherence properly, with detector-based and question-answering checks such as GenEval, TIFA, and DSG and with human studies, is the subject of §34.6. Steering cross-attention maps to edit images returns in §34.7.

import numpy as np

def binding_report(A, pairs, nouns, tau=0.5):
    """A: (S, T, T_c) cross-attention weights, column 0 is the start token.
    pairs: list of (attribute index, noun index) into the T_c text tokens. nouns: list of noun indices.
    Returns (iou, neglect) with iou of shape (len(pairs),) and neglect a float."""
    raise NotImplementedError

Choosing how to condition on text

The two tables collect the choices of this section, first the route into the denoiser and then the encoder.

In PyTorch

The block below is a cross-attention sublayer as a U-Net or a cross-attention transformer uses it. It has a key padding mask and text keys and values computed once per prompt. Its output projection is initialized to zero, so a new sublayer starts as the identity, as in PixArt-α. The block also shows per-encoder dropout for training and one batched guidance step.

Read more

• Saharia et al. (2022). Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. Imagen, the study that showed a larger frozen text encoder helps more than a larger image model, with the ablations of pooling against attending to the sequence.
• Balaji et al. (2022). eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers. CLIP and T5 together with independent dropout, what each encoder contributes, and why text matters early in sampling.
• Podell et al. (2023). SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. Two CLIP encoders, the pooled vector, and the size, crop, and aspect-ratio micro-conditioning, with the measurements that motivated them.
• Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. SD3, with three encoders dropped independently, the exact conditioning arrays (Appendix B.2), the caption ablation, and the T5 removal study.
• Betker et al. (2023). Improving Image Generation with Better Captions. The DALL-E 3 report on descriptive recaptioning, blending ratios, prompt upsampling, and the limits a captioner passes on.
• Liu et al. (2022). Character-Aware Models Improve Visual Text Rendering. WikiSpell and DrawText, the controlled evidence that subword encoders cannot spell and byte-level ones can.
• Chefer et al. (2023). Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models. Neglect and binding failures in Stable Diffusion, read from cross-attention maps and fixed at inference.
• Wu et al. (2025). Qwen-Image Technical Report. A vision-language model as the condition encoder of a large joint-attention transformer, and a data curriculum for rendering text.

Watch next

▶

▶

▶