Working in the Compressed Space

The nightmare of pixels: where the computing effort goes

If we applied diffusion directly on the screen, at every step of cleaning the noise the neural network would receive a huge image and return a prediction of the same size. For an ordinary photograph a thousand pixels on a side, coloured in the three fundamental channels (red, green, blue), that means over three million numbers processed at every step, dozens of times over, for a single frame generated.

And yet most of that numerical flood carries mere surface detail: little threads of noise, coarse grain, minute variations of light on a leaf. All of it counts for the final feeling of sharpness, but it has nothing to do with the great decisions taken along the way: whether we draw a house or a tree, whether the sky is clear, or where exactly the figures stand in the scene.

Thus a brilliant idea was born: why move mountains of pixels every fraction of a second, when we can take all the hard decisions in a narrow space and let the texture and the make-up be applied once, at the end?

The autoencoder: the architect and the builder

The piece that makes this move possible works like a team of two partner networks. The first half (the encoder) gathers the large image into a concentrated handful of essential numbers. The second half (the decoder) takes that handful of numbers and rebuilds a complete image from them. Trained together, they learn a sacred rule: what comes out at the end must faithfully resemble what went in. This pair is called an autoencoder, and the handful of numbers in the middle is called the latent representation (or latent space).

Nobody mechanically dictates which features to keep. Because the space in the middle is a very narrow pass, the system comes of itself to retain only the vital information from which the picture can be rebuilt — that is, exactly the large contours that tell one image from another. The narrows are the best teacher of synthesis.

The figure below uses a simplified form of linear compression, computed rigorously from the geometric examples. Although a modern neural network is far richer, the fundamental properties are identical: reduce the number of latent values and watch how the canvas keeps its meaning even when compressed drastically.

The architectural sketch: what is really kept

The question that settles whether the move is worth making is not whether the rebuilt image is identical pixel for pixel, but whether the numbers kept still let one read unambiguously what is in it.

The table below measures both dimensions at once: the visual error of reconstruction (how much is lost on the screen) and the accuracy of recognition (how easily a simple rule can find the correct shape by looking only at the few compressed numbers).

Look at the contrast between the two columns: the fine rebuilding of the pixels degrades gradually as we cut away values, but semantic recognition holds up admirably! Even with twenty-five times fewer numerical values, the model knows precisely which drawing is there.

This is exactly the role of the architect’s working sketch: it contains neither the texture of the plaster nor the exact shade of the floorboards, but it contains everything indispensable for deciding where the walls rise and where the doors open.

Latent diffusion: the dance on the sketch

Here comes the great turning point that opened the road to the modern image generators (such as Stable Diffusion). Instead of applying noise and making heavy computations on millions of pixels, we move the whole laboratory of diffusion into the latent space!

The training photographs are passed once through the encoder, turning into compressed sketches. Then the whole dance of diffusion takes place exclusively there: the noise is added to the latent numbers, the network learns to guess it there, and the 50-step road back cleans only this compact sketch. Only at the end of the process does the result pass once through the decoder, which turns it into a high-resolution photograph. This is latent diffusion.

The saving in resources is enormous and can be computed precisely, because the effort of a network grows in direct proportion to the number of values it handles.

The compression acts on both sides of the image: a reduction of eight times per side means an area 64 times smaller! Even though each latent point carries more channels of information than the three colours of an ordinary pixel, the net gain stays of the order of tens of times, making it possible for generation to run even on an ordinary laptop.

What is lost: the small inventions of the decoder

This formidable speed has a hidden price, however. The decoder that paints the final pixels cannot read the past with an infallible magnifying glass: it invents plausible details and textures to match the numbers it receives, drawing on the images it saw during training.

From this come the famous flaws anyone who has looked closely at such results knows: the minute text on a label in the background looks like credible text but contains no real letters; fine fabrics take on strange synthetic patterns; and small faces in the distance are given slightly deformed or hallucinated features. These are not defects of the diffusion process but improvisations of the decoder, visible exactly at the microscopic scale at which it was left to invent.

Diffusion is today the queen of image synthesis, but it is not the only generative family and it was not the first. In the next lesson we shall set diffusion beside its other great relatives — the adversarial networks and the generative flows — comparing step by step the reliability and the speed of each.