The Goal and Its Measure

A mark in place of a real wish

Throughout this course we have trained networks always using one number: a cost function we wanted to bring down, a reward we wanted to raise, or a score on a set of tests. But that number was never our true wish. Our real wish was that the machine should “recognise handwritten digits”, “answer politely and usefully” or “draw a convincing picture”. The number was only an approximation — what we managed to put into mathematics out of a rich and nuanced human intention.

In the lessons on assistants we already noticed this distance: the reward model does not necessarily learn a person’s subtle thoughts, only the patterns in the answers they preferred. If human evaluators tend to prefer slightly more detailed answers, the machine quickly registers that length brings extra points. As long as the pressure of search is moderate, the difference between the mark obtained and the real quality of the answer stays small and harmless.

The essential question of this lesson is: what happens when we build an extremely powerful engine of search that forces the winning of that mark to the utmost? This optimisation pressure can be studied precisely by a simple and intuitive method: selecting the best answer out of a number k possible variants.

Choosing the best of k variants

Let us imagine putting a question to a model and asking it to generate a group of k different answers. Each answer has three independent qualities, chosen at random: how correct it is, how clear it is phrased, and how long the text came out.

Our real goal — what a rational user wants — is correctness, clarity and a balanced length: a slightly fuller answer is helpful, but an answer far too long becomes tiring, redundant and loses its usefulness.

The automatic measure (the mark it receives), however, is computed by a simplified formula: correctness and clarity bring points, and length is counted always as a plus. At first sight it looks a fairly fair mark: it agrees with our wish on two criteria out of three, and on length it errs only when the text passes a certain limit.

Let us see what happens when the machine chooses the answer with the highest mark out of a sample of k variants. When k = 1, the machine takes the first answer with no selection at all. When we raise k to 4096, the machine searches tirelessly for the best-marked answer out of thousands of possibilities.

The graph above shows a fascinating phenomenon. The blue curve (the official mark) rises ceaselessly: the more variants the machine searches among, the higher the mark of the answer it finds. The orange curve (the real quality wanted) also rises at first: at small values of k, mark and usefulness go hand in hand, because the machine chooses more correct and clearer answers.

But look at what happens from a certain point on: the two curves part dramatically! The real quality reaches a peak and then begins to collapse. Why? Because the answers found are already fairly correct and clear, and the only way the machine can still squeeze out extra points is to find the longest and most padded answer possible. Without realising it, the system has begun to sacrifice real quality in order to “cheat” the marking scheme.

The slider shows that the size of the error in the marking formula dictates only when the collapse appears, not whether it will happen. Even with a formula that errs very little, a sufficiently powerful search will discover that small imperfection and exploit it to the extreme.

Why the damage grows with the power of the machine

The optimising system has no access to our real wish. It sees only the number we asked it to maximise. For it, the answer selected at k = 4096 is an absolute success — the highest score ever obtained! That the answer has become an interminable and unreadable composition is not visible in its mathematical formula. This discrepancy can be seen only from outside, by a person who knows what they really wanted.

This is the living demonstration of a famous principle from the social sciences, known as Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure.” If you judge a medical clinic by short waiting times, the doctors may hurry patients through; if you judge a school by the marks on a standardised test, the teachers will teach nothing but that test; if you ask an AI model to maximise a superficial mark, it will find the most unexpected ways of dodging the spirit of the request.

In this perspective, the notion of “superintelligence” denotes an optimiser with an enormous k : a system able to explore millions of options in order to maximise the given formula, far more efficiently than any human could. The risk does not come from the machine becoming conscious or ill-intentioned, but from its being too efficient in pursuing an imperfect target. Nobody knows how to describe a complex human value by a formula without loopholes, and a sufficiently powerful system will always exploit the loopholes.

How we protect ourselves: the instruments of alignment

The problem has no simple magical solution, but it does have clear engineering safeguards. Three of them are tested in the table below:

1. Tempering the pressure of search: we do not let the machine search among thousands of extreme variants but constrain it to stay close to the behaviour people demonstrated in the initial examples.
2. Several judges: instead of one mark, we use several independent formulas of evaluation; two rulers with different weaknesses compensate for and correct each other.
3. Barriers and common-sense thresholds: before marking the answer, we reject outright any variant that passes natural limits (a text excessively long or aggressive, for instance), as a safety check applied before the marking.

None of these measures offers a perfect formula, because a perfect formula for human values does not exist. But all of them keep the real quality high where the simple mark would have let it collapse. The price paid is the same: we voluntarily give up part of the brute force of optimisation in favour of safety.

This is the practical shape taken by the discipline called the alignment of artificial intelligence (AI alignment): not the illusory search for an absolute mathematical definition of the good, but the careful building of multiple rulers, protective filters and common-sense limits around formulas we know from the start to be imperfect.

What is certain and what remains an open debate

The simulation we have explored describes a mathematical certainty: for any imperfect marking scheme there is an intensity of optimisation beyond which real usefulness suffers, and that critical point is reached the sooner the more full of gaps the scheme is. What the simulation cannot tell us is how quickly real-world systems will reach that level of force, and how well our protective measures will hold.

Here the opinions of researchers part: some hold that future systems will remain docile tools under careful human control, where common-sense filters will be enough. Others warn that the economic pressure to build ever more autonomous and faster agents will force optimisation beyond our capacity to supervise it. Both perspectives acknowledge the same curve; the difference lies in the judgement of where on it we stand today.

What we can take away as a clear lesson is that the distance between what we want and what we measure is no temporary defect of computers but a property of any process of optimisation. In the last lesson of the course we shall take this whole fascinating road again from the beginning, running through it from the bottom up in a single overview.