Training at Scale

The great reading: what a machine learns from the libraries of the world

The fundamental task underlying the great language models is surprisingly simple: running through a fragment of text, the model must guess which token comes next. The correct solution is already written in black and white in that very sentence, at the step immediately following. There is no need for paid humans to put labels on every word by hand: everything ever printed is, by right, an exercise book ready to be worked through. This first massive phase is called pre-training.

From this comes the gigantic scale of contemporary artificial intelligence. Mankind has produced trillions of words in books, articles, source code and conversations. The crucial question is no longer where to gather examples but how the network’s behaviour changes as it is fed oceans of text.

The anatomy of a mistake: the three components of the error

When we test the model on fresh texts it has never seen before, its total mistake (measured by loss or perplexity) is not a chaotic number but the sum of three distinct sources:

The first is the entropy of the source (or irreducible error): the natural degree of mystery in living communication. Not even an omniscient reader could know with absolute certainty how a poetic sentence will end or what decision the author will take in the next line. Language has a dose of freedom and play that imposes a minimum barrier of uncertainty, impossible to bring down to zero.

The second is the capacity gap: the powerlessness caused by the size of the network. A small model, with little memory, simply does not have enough synapses to hold all the grammatical subtleties or the historical facts of the world. Even if we gave it an infinite library, it would soon reach a plateau because of its own structural narrowness.

The third is the estimation gap: the ignorance generated by a shortage of reading. The model would have parameters enough to grasp an idea, but it has not met enough concrete examples to calibrate its weights. This error dissolves gradually as the volume of data grows.

Data remove the estimation gap, parameters remove the capacity gap, and entropy stays untouched, an impassable floor. If you force one aspect and neglect the others, you soon hit a wall: you have an oversized engine running empty, or a flimsy chassis overwhelmed with fuel.

The laboratory experiment: the learning curves

We can watch this phenomenon with mathematical precision on a synthetic generator whose secrets we know entirely. The source below emits symbols according to a strict set of rules depending on the last three characters. Knowing the exact formula of the source, we can compute in advance the precise level of the entropy and the theoretical floor of each model.

We train four models of different sizes: from the smallest (which looks only one character back) to the largest (which takes four characters into account).

Every curve falls steeply at first, then levels off and refuses to fall further. The plateau is no accident: it is each model’s capacity gap. The smallest reaches its floor at once and learns nothing new, however many millions of characters we supply.

The model of the right size reaches exactly the entropy of the source. An oversized model, by contrast, has the same final floor, but because it has far more parameters to adjust, it needs far more text to reach its best performance. Being “bigger” does not automatically mean being “better”, unless the model is fed with data to match its size.

The harmony of the budget: which model do we choose at each stage?

Look at the ranking of the models at different data budgets in the table below:

The practical lesson is unambiguous: if you have little text, a huge model is sheer waste and gives poorer results than a compact one! Not having enough examples to anchor its thousands of parameters, the giant model begins to hallucinate and to memorise noise. As the mountain of data grows, the hierarchy inverts spectacularly, and the more capacious architectures take the lead.

The best size for the artificial brain grows hand in hand with the volume of reading offered. This common-sense rule governs the great economic decisions behind the scenes in artificial-intelligence laboratories.

Scaling laws: a predictable engineering recipe

When we plot the fall in error on logarithmic axes, the trajectory becomes a remarkably smooth straight line. A straight line on a logarithmic plot reflects a power law: every time we multiply the resources by a given factor, the distance from perfection falls by a fixed and predictable proportion.

These rules, called scaling laws, link three fundamental quantities: the number of the network’s parameters (the engine capacity), the volume of data read (the fuel) and the total power of compute (the electricity consumed by the processors). The famous Chinchilla study (2022) showed that, to get the most out of a compute budget, parameters and the volume of tokens must be doubled in equal proportion.

There remains, however, a great enigma that intrigues researchers: the scaling laws can estimate with laboratory precision how far the mathematical loss will fall, but they cannot anticipate what new abilities the model will acquire! The gradual fall of a fraction of error suddenly unlocks logical deductions, surprisingly fluent translations and the ability to write complex computer programs.

Where is the limit of human language?

In our simplified experiment the source had a memory of only three letters, so we could quickly reach the absolute floor beyond which there was nothing more to learn. Living human language, though, is infinitely deeper: an allusion in a novel may point back to a sentence spoken three chapters earlier, to an ancient Greek tragedy or to a nuanced psychological experience. Nobody has yet reached the absolute floor of language, and models keep improving as we give them more computing power and richer texts.

At the end of this massive stage of pre-training, the model becomes a master of probabilities: at every step it computes a distribution of chances for the next word. But how do these abstract percentages turn into coherent text spoken on the screen? In the next part we shall witness the birth of dialogue: we shall discover how the computer chooses the right word and how the personality of an artificial intelligence is shaped.