In-Context Learning

What changes and what does not: the miracle before our eyes

The text we supply to an assistant before it begins to write is called the prompt. In it we can put anything: a direct request, a role for it to adopt, an earlier conversation or — most spectacular of all — two or three worked examples of how we want the answer to look.

When we show it these examples in the prompt (a technique called few-shot prompting), the model’s behaviour is transformed as if by magic. It takes on the style, the format and the logic of the task at once, solving the next exercise exactly on the pattern offered. And yet, throughout this process, not one of the billions of parameters of the network has been changed by so much as a decimal! This astonishing behaviour is called in-context learning, although the name is somewhat paradoxical: we are not talking about classical learning by changing the circuits, but about an adaptation in flight.

The model is like a brilliant jazz musician who listens to the first two bars of a new tune and joins the band at once, playing impeccably in the key and rhythm of the piece. He did not go home to practise that piece for months; what he possesses is a superior ability to recognise and take up patterns in real time.

A new rule at every string: the mathematical experiment

To see this mechanism in its pure state, we have prepared an interactive mathematical experiment: we train a small transformer on a task where the rule changes at every attempt.

Every problem has a secret slope (a mathematical straight line), chosen at random. The model receives a few points from that line, affected by a small measurement noise, followed by a new coordinate for which it must guess the corresponding value. Because the slope is always a different one, the network cannot store it in its permanent memory; it is obliged to deduce it each time solely from the points we offer it in the context.

The blue line is the model’s prediction. With no example at all, the line is almost horizontal: having no clue, the safest decision is to stay around zero. As you press the button and add examples, the blue line turns and moulds itself ever more faithfully onto the hidden orange line.

Watch the sum of the weights shown in the panel: it stays strictly unchanged! Not a number in the mathematical brain of the network has undergone any adjustment; all that has changed is the information set before its gaze.

How well it does

The graph below measures the average error over hundreds of new problems, as a function of the number of examples put in the prompt. For comparison, we have set alongside it two classic statistical benchmarks:

The first benchmark is the traditional method of least squares (classical linear regression). The second is the optimal estimator, which knows in advance the possible limits of the slopes.

The model’s error falls spectacularly with every new example shown, although the weights are frozen. Here is the undeniable proof: the model improves its performance in real time solely on the basis of the text in its working window.

What training actually learned: the algorithm in the weights

Look carefully at the first row of the table: when it has a single point of reference, the classical method of least squares suffers an enormous error, because a single point close to zero can throw the computed slope towards infinity. Our neural model, by contrast, does not lose its head!

It behaves like an experienced statistician: when the data are few, it stays cautious and anchors itself to the mean; as we add solid data, it begins to trust them and follows their direction. Nobody programmed these formulas into it explicitly. The behaviour sprang spontaneously from the hundreds of thousands of exercises it went through during training.

Here lies one of the deepest revelations of modern artificial intelligence: the weights of the network do not memorise a simple encyclopaedia of fixed answers but an internal algorithm for processing and learning. The weights embody the method and the general reasoning, while the context brings the specific data of the problem at hand. This is the great division of labour: what is universal and permanent lives in the weights, and what is particular and provisional lives in the prompt.

Why this property appears: the echo of human structure

No engineer designed a particular piece of the transformer with the aim of doing in-context learning. This ability appeared of itself as the most efficient mathematical strategy for predicting texts written by people.

Human language abounds in structured patterns: tables, translations set side by side, source code with examples, explanations followed by summaries. If a network had tried to memorise one rigid rule for all situations, it would have failed wretchedly on a great part of the texts. The optimisation algorithm discovered a far more supple road: it shaped the attention circuits so that they read the local pattern from the first lines and extend it naturally onwards.

Here our great journey through the realm of language ends: we set out from a single binary bit, saw how transistors and logic gates are born, built processors and neural networks, understood attention and training at scale, until we arrived at assistants able to converse and to learn from context.

But the universe of intelligent machines does not stop at words. In the next part of the course we leave the universe of text and enter the world of visual forms: we shall discover how, starting from a chaos of static noise and blind dots, a network can gradually sculpt an image of astonishing beauty.