How It Is Measured

What a score really is

At root, a standardised test (a benchmark) is simply a collection of questions whose correct answers we already know. We send each question to the model, count the right answers and compute the percentage of success. This process is called evaluation (or benchmark evaluation).

From this apparently banal definition follow two fundamental truths, often overlooked. First, the questions were chosen by particular people with their own preferences and prejudices — so the score reflects only skill at that particular set of tasks, not some supposed general cleverness. Second, any test is only a narrow sample from the infinite universe of possibilities: the score obtained is no divine certainty but a statistical estimate, inevitably accompanied by a margin of error.

To illustrate this, we have prepared three different models, trained on the same basic material (calculations with two numbers) but with distinct experiences of learning: the first practised mostly additions, the second was fed chiefly on subtractions, and the third received a balanced mixture of both operations.

Four tests, four rankings

Press “Train and measure” and look at the results: each of the four tests proclaims a different champion! None of the models can be called “the best” in any absolute way without immediately saying what we are measuring against. The final ranking is not an intrinsic property of the neural network but a reflection of how the examination was composed.

Even a supposedly “neutral” test, combining several subjects, subtly brings in distortions of its own: if one task is far harder than the rest, all the models will fail at it equally, so that the podium will be decided by the easy or incidental questions.

How much of the difference is noise

Because the test holds a finite number of questions, the score computed naturally fluctuates: it is the uncertainty of a score (the statistical confidence interval). If a model scores 82% on a set of 100 questions, its true skill lies somewhere between 75% and 89%. To halve this margin of uncertainty, we would have to quadruple the number of questions in the test.

This mathematical reality explains why the public leaderboards of artificial intelligence swing ceaselessly from month to month, without the models having undergone any real change. When the difference between first and second place is only 1–2 per cent, the ranking is dictated by the statistical luck of which questions happened to be in the test, not by any palpable intellectual superiority.

Comparison on exactly the same questions (identical papers) allows a somewhat finer statistical distinction, because the models often stumble at the same traps. Yet this apparent precision evaporates instantly as soon as we take the models out into the real world and confront them with new, unforeseen questions.

When the test has passed through the training

There is a still more insidious danger: the “leaking of the examination papers”. Standardised tests are public on the internet, and the robots that gather texts for training models absorb billions of web pages indiscriminately. When the questions and the official solutions of a test end up in the network’s study material, the phenomenon is called contamination (data contamination).

The experiment below trains the balanced model again from the same starting values, but slips a few of the exercises from the official test into its learning material.

Look at what happens: the score on the “contaminated” official test rises spectacularly, giving the illusion of extraordinary progress, while on entirely new questions real performance stagnates or even degrades. The model has not become more capable; it has simply learned the test by heart, like a pupil who saw the answer sheet before the lesson.

The tragedy is that this involuntary fraud cannot be detected by looking only at the published test. It comes to light solely when the evaluator holds a secret set of fresh questions, kept strictly under lock and key.

Why it is not easily cleaned up

At first sight the solution seems obvious: why not delete the test questions from the training data? In practice it is an all but impossible mission.

The archives of the internet hold thousands of billions of words. A question from a test may appear paraphrased, translated into another language, quoted in a blog article or discussed at length on a technical forum. What is more, even if the data were entirely clean, Goodhart’s law steps in: as soon as a measurement becomes a commercial target, all the developers optimise their networks expressly to tick that test, and the indicator loses all objective relevance.

This is why the most honest modern evaluations are the dynamic ones: entirely new questions, framed after the model was finished, and blind voting platforms (such as the chat arenas) where real users judge the answers without knowing which model is behind them. Any ruler left too long in plain sight ends up, inevitably, deforming the very thing it meant to measure.

Up to here we have treated the model as a “black box”, looking only at what goes in and what comes out of the door. In the next lesson we shall open the box itself: we shall step inside the network to see how the neural circuits light up and what real concepts come into being in its deep layers.