The Road Back
Looking at a screen full of random fleas, it is impossible to guess which image exactly was destroyed, because an infinity of different photographs could have arrived at exactly the same chaos. But if the blunt question has no answer, we can put a far wiser one: what is the average of all the plausible images? Cleaning the noise step by step, like a black-and-white photograph developing slowly in a laboratory tray, a single clear silhouette emerges from the fog.
Key terms: the reverse process (denoising) the weighted average a sampling step gradual developing generalisation vs copying
A question with no answer, and the rescue by probabilities
The forward process of the last lesson is a one-way street, easy to travel: you take a photograph, add noise to it by a recipe and obtain a frame eaten by fleas. But the road back hides an enormous barrier. One and the same dirtied image could have come from a cat, from a castle or from a clock, each combined with a different, fitting noise. Consequently, the blunt question “which image exactly was it at the start?” has no unique mathematical answer.
The rescue comes through a subtle change of perspective: instead of looking for an absolute truth hidden in the fog, we ask ourselves: “out of all the possible images in the world, what is the average of those compatible with what can be glimpsed now?”. Every plausible variant receives a weight in proportion to how well it fits the rest of the signal. This weighted average can be computed — and it becomes the compass guiding us at every step of the reconstruction.
A step of return works like this: we compute this estimated average image, subtract from the dirtied frame part of the noise separating us from it, and then add back a minute drop of fresh noise, but somewhat less than we took out. This repeated gesture is called a sampling step, and the whole backward march is reverse diffusion.
Why would we add noise back into an image we are trying to clean? It seems a waste, but it is the vital secret of diversity! Without this minute infusion of chance at every step, the road would slide mechanically towards the same faded mathematical average — a uniform grey. The fresh noise injected is what shakes the system gently, allowing it to choose a new and distinct composition at every run.
A laboratory in miniature: when the formula can be written exactly
In everyday life the set of all possible photographs is infinite and impossible to close up in a simple recipe — which is why we shall let a neural network approximate this average. But to see the mechanism in action, we can build a universe in miniature, in which we know all the possible shapes in advance: just six geometric pictograms.
Looking at an image invaded by fleas, the computer instantly reckons how well it fits each of the six shapes. The average we are after is nothing but a blend of the six drawings, weighted by these scores of fit. Here there is no blind guessing and no machine learning: the mathematical formula is perfectly exact.
Look carefully at the panel on the right (“what it thinks it was”). At the beginning of the journey, when the noise is dense, it shows no distinct silhouette but a ghostly superposition of all six: their collective average, since no shape is favoured. As the noise falls, the balance tips steeply: the weights separate, and the average clears spectacularly into a single well-defined shape.
The phenomenon recalls exactly the developing of a photograph in a darkroom: when you dip the white paper into the solution, it is not the hairs or the texture of the skin that appear first but the great contrasts of mass. The large contours show up first, and the fine details settle meekly only at the end.
The great decision: when the shape is chosen
A natural question is: how early in this process is the fate of the image sealed? Is the shape decided about halfway along, or in the very first steps? The measurement below runs the process twenty times starting from different seeds of noise and notes the exact step beyond which the favoured shape never changes again.
The decision is taken astonishingly early, while to our eye the image still looks like meaningless fleas! Although the human gaze makes out only a grey ant-heap, the mathematics has already chosen the winner for good. The rest of the road deliberates nothing further: it patiently draws the shape that the initial chance settled from the very start.
Copying versus creation: why imperfection is a virtue
Here the deep reason comes to light why in a real system we prefer an approximate neural network to an exact mathematical formula.
The exact formula we have used is a captive of its little universe: it can produce nothing but what is already among the six examples. At the end, the image obtained does not merely resemble one of the shapes — it is the identical copy of one of them.
A neural network trained on millions of images will never learn an exact average. It retains an intuitive approximation, sprinkled with small deviations and syntheses. Thanks to these imperfections, the road back does not reproduce a photograph seen in training but creates a fresh, plausible composition. What would look like a technical lapse is precisely the spark of creation: genuine novelty is born of the living approximation of the past.
If we used a huge network for a minute set of data, it would tend towards the exact formula and begin to churn out faithful copies of the examples — the phenomenon of overfitting (overfitting) that we met in the earlier parts.
So far we have leaned on a rigid mathematical formula. In the next lesson we shall replace that formula with a genuine neural network, which we shall train in real time, on this very page, on a handful of 2D points.