The Text That Pulls the Image
Until now our models drew freely from everything they had learned, with nobody able to ask them for a particular subject. But how do you persuade a storm of fleas to turn on command into a sunset or a precise silhouette? Our words must become active forces that guide the brush of diffusion at every step. And how hard we let the text pull at the image is a subtle double-edged lever, one we can measure and master.
Key terms: text-to-image conditioning cross-attention random seed classifier-free guidance (CFG)
The words that guide the brush: conditioning
Up to this point, the network cleaning the noise received only two clues: the coordinates of the point and the level of noise it stands at. Now we set before it a third essential witness: a description in words. The text is first turned into a numerical representation (a vector of meaning) and offered to the network as an extra reference during training. This partnership is called conditioning.
From that moment on, the noise guessed by the network no longer depends only on the position of the point in space but also on the instruction received. The same flea-eaten point, at exactly the same level of degradation, will be pushed in one direction if we whisper the word “spiral” to it, and in a completely different direction if we ask it for a “ring” or a “cross”.
In complex systems of visual generation the text does not enter as a single rigid block. Every region of the canvas analyses the words of the description through the mechanism of cross-attention (cross-attention): one word pulls the brush in the top corner of the image, while another guides the details at the base. Our model on the page uses the most direct form, with a single global label, because the image is simplified to a single point.
The three descriptions available here are simple words whose geometric forms partly overlap in the plane. Whoever wants to see the effect of the text must follow the harmony of the final shape, not merely where the points land.
The seed and the word: who decides the composition?
The random noise from which reverse diffusion starts is determined mathematically by a number called the seed (or seed). By keeping this seed fixed and changing only the guiding word, we can separate with perfect clarity what chance brings and what our wish dictates.
All three panels above started from exactly the same initial points of noise. The only thing that separated them was the word offered to the network at every step. If you change the seed, the exact place where each speck lands changes; but the overall contour stays faithful to the chosen word. The golden rule of generation is simple: the word decides what is drawn; the seed of chance decides which one of the possible variants is born.
The guidance lever: how we amplify the voice of the text
If we train the network in the ordinary way, we notice a troublesome timidity: the description in words is only one of several inputs, and the model often tends to take a neutral average over everything it has seen, ignoring the specific request.
The modern solution rests on a clever strategy: during training we hide the word from time to time and let the network work with no description at all. At generation we ask it for two parallel predictions: one in which it listens to the text and one in which it has no text at all. The difference between the two reveals exactly the force of the word: where our request pushes, compared with blind drawing.
This difference can be amplified by a multiplying factor called guidance (or Classifier-Free Guidance). At a value of one we use the network’s natural prediction; at zero the word is ignored entirely; and above one we deliberately exaggerate the impulse of the text, forcing the points to obey the chosen subject.
The two edges: clarity versus rigidity
The guidance lever is, however, a double-edged sword. The harder we force the word, the more the points are crushed towards the most typical and stereotyped centre of the requested shape. The result becomes clearer and more docile, but loses its natural variety, resembling itself excessively.
The table below measures both aspects at once: the percentage in which the result really belongs to the requested shape, and the degree of spread (diversity) of the points.
Think of a talented draughtsman receiving directions while he sketches on the canvas. A few clear suggestions help him hit the desired subject. But if you shout instructions at every stroke and hold his hand rigid, the drawing becomes stiff, repetitive and artificial. And if you overdo it beyond measure, the work is ruined for good.
From this comes the behaviour anyone using text-to-image tools observes: at high guidance the results are striking and very obedient, but repetitive and saturated; at low guidance they are rich, poetic and varied, though they risk straying from the text. It is not a battle between good and bad but a living compromise between fidelity and creative freedom.
So far all our computations have taken place directly on the screen, pixel by pixel. For a high-resolution photograph this volume would be enormous. In the next lesson we shall see how the whole of diffusion can be moved into a far more elegant compressed space: the latent space.