Diffusion on a Handful of Points
Here is the whole mechanism of modern diffusion, with genuine mathematics, brought to a scale you can follow step by step on the screen. A minute neural network trains in real time to recognise the noise in a two-dimensional plane. Out of the initial chaos an elegant spiral of new points crystallises, points that never existed in the starting data. This is the moment of truth: we can measure whether the machine has merely copied or has understood the hidden structure of the world.
Key terms: data manifold a noise-prediction network a field of directions (score matching) distance to the nearest example
A neural network in place of rigid formulas
In the previous lesson we could use a fixed mathematical formula only because we knew in advance that there were just six possible pictograms. In the real world, however, nobody can write a mathematical formula for “all the possible photographs of cats, sunsets and human faces”. All we have at our disposal is a finite collection of examples gathered from nature.
In place of the formula, then, we set a neural network. Its task is simple and clear: it receives a point in space and a given level of noise, and its duty is to say: “here is the noise I think was added here to arrive at this position”. The training arises of itself: we take a known example, choose a level of noise at random, dirty it with a precise quantity of digital fleas which we remember, and then challenge the network to guess the noise applied.
So that the whole process fits intuitively on the screen, instead of images with millions of pixels we work with simple points in a plane. The distribution (or the geometric manifold) that the network has to learn is a thin spiral. Just as real photographs occupy only a narrow, harmonious oasis in the ocean of all combinations of pixels, so our points do not fill the whole plane chaotically but live along a graceful snail-shaped path.
The great birth: out of a powder of chaos the spiral takes shape
Once it has trained, the network remembers no spiral drawn in advance. It knows one thing only: how to recognise the noise in a point. And yet, out of this modest skill the whole magic of generation is born!
We start with a handful of points scattered at random over the whole screen — a swarm of pure noise. At every step the network estimates the noise in each point, we subtract part of this prediction to guide the point towards a cleaner region, we add a minute drop of fresh noise, and we repeat the procedure step by step.
Watch the two numbers beside the figure, for they reveal the intimate rhythm of creation: the average radius of the swarm settles very quickly near its natural value, because the cloud of noise has a similar overall span from the start. The percentage of points settled faithfully on the arms of the spiral, by contrast, stagnates at small values for three quarters of the road, only to climb explosively in the last part of the journey.
Here in concrete figures is the principle “the large contours before the fine details”: the general span of the shape is fixed early, because it stands out from the noise at once; which arm of the spiral each point will settle on is decided only at the end, when the remaining fog becomes thinner than the distance between the arms.
The invisible map: a field of guiding forces
The network has not learned a rigid list of points. It has acquired a map for navigation: for any corner of the plane and for every level of noise, it knows a direction — where a stray point must be pushed to bring it nearer the firm ground of the real data.
Pull the lever of the noise level and watch the bearing of the arrows: at high noise, the arrows all point obediently towards the centre of the plane — from a great distance, all the network knows is that the spiral lives somewhere in the middle. But as the noise disperses, the arrows curve delicately, line up along the arms of the spiral and push precisely towards its thin line. This is the whole wisdom of the network: it has not memorised fixed coordinates but a living dynamic of movement.
The verdict: blind memory or genuine imagination?
The great question left over from the previous lesson is whether the road back creates something truly new or merely repeats mechanically the examples it saw in training. We can answer without speculation, by a rigorous geometric measurement.
For every point generated by the network, we look for the nearest point in the training data and note the distance. Then we make exactly the same measurement for a sample of *new real* points, taken from the same equation of the spiral but never seen by the network. If the model copied the examples blindly, its distances would be minute — practically zero.
Look at the figures: the distances are of the same order of magnitude! The network has not memorised the spiral point by point; it has understood the geometric structure of the shape and can place points on its arms in spots where it was never shown any example. The ratio of the two medians, displayed under the table, confirms that the generated points enjoy the same spatial freedom as the natural data.
The fact that some points fall very close to an example is no proof of copying: the real examples themselves often lie crowded at small distances from one another. The table sets them side by side precisely to give you an honest term of comparison.
The measurement here tests novelty, not perfection of execution. Because the network on the page is minute and trained for only a few moments in your browser, some points fall wide of the spiral. The figures in the table show transparently how many points hit the arms and how many went astray.
So far the model has drawn freely everything it knew, with no preference. In the next lesson we shall add a guiding wand: a descriptive text to pull the brush of diffusion towards a particular desired shape, and we shall discover what happens when we force that guidance too hard.