The Next Word

The mission: continuing a sentence

Imagine that you are listening to a familiar tune and someone stops abruptly in the middle of the chorus: your mind already hears the next note before it is played. Language prediction works in exactly that way. It is shown a string of words — for instance, “The clear sky was a shade of…” — and asked to anticipate the natural continuation.

The model’s answer is not a single categorical word thrown on the table but a complete list of scores for every term in the vocabulary: “deep” gets 60%, “pale” 25%, “brilliant” 10%, while absurd words such as “fork” get a minute chance, close to zero. When we add all these scores, their sum comes to exactly 100% (or 1). A system that computes these chances is called a language model, and the complete table of scores is called a probability distribution. The network only weighs the possibilities; the final decision about the word spoken is taken only at a following step.

The great secret: the training data are free

Here lies the essential revelation that propelled artificial intelligence to its present gigantic scale — an economic detail often passed over.

To teach a computer to recognise road signs or handwritten digits, thousands of people are needed to label every single image by hand: “this is a bend”, “this is the figure 7”. This manual work is colossally expensive, slow and limited by budget. For the task of guessing the next word, by contrast, no paid teacher is needed to add labels! The correct answer is already right there, in the text the machine is reading: it is the word that naturally follows on the page.

Any book, newspaper article, encyclopaedia or web page in the history of mankind instantly becomes a perfect set of solved exercises. A book of a hundred thousand words offers a hundred thousand questions with the answer guaranteed and included. This methodological miracle is called self-supervision. Through it, the barrier of manual labelling has vanished: the whole library of our civilisation turned overnight into an inexhaustible ocean of learning.

The simplest predictor: the method of blind statistics

How can we build the most primitive language model? Simply by counting the occurrences! We read a large text and keep a rigorous record: how many times the word “crowded” followed “café”, and how many times “old”? When we want to predict what comes after a term, we look in the ledger and choose the proportions observed in the past. We may look back only one word, or at groups of two, three or four consecutive words. The number of terms taken into account as a starting point is called the order of the model (or the context of n-grams).

If we look back only one word (order 1), the options are scattered and confusing: after a preposition such as “with” almost any noun in the dictionary may follow. If we widen the window to three or four words, the prediction becomes very confident on known sentences, because that exact sequence has perhaps appeared only once in the text read.

From prediction to creation: generating text

A system able to weigh probabilities can be turned at once into an automatic author. We give it a starting impulse (a “prompt”), the model computes the probabilities, chooses a word and adds it to the sentence. The new, extended sentence becomes the question for the next step. Repeating this cycle word by word, a flowing text is born. Note the beauty of the mechanism: the machine was not taught to be a creative writer, only to guess the missing link; the generation of stories is merely the game of prediction rolled forward step by step.

But how do we choose the word from the list of probabilities? We are not obliged to take the one in first place every time. We can introduce an intuitive adjusting parameter called temperature. At a low temperature (close to zero) the model chooses strictly the most probable words: the resulting text is very logical, correct and predictable, but it quickly becomes repetitive and monotonous. At a high temperature the machine gives a chance to the more surprising variants as well: the text acquires liveliness, metaphor and originality, but if we overdo it, it begins to ramble and loses the thread.

The passages highlighted in yellow are stretches copied identically from the text on which the model was trained. If we set a large order of context and a low temperature, a purely statistical model creates nothing new but merely reproduces memorised fragments mechanically. But this limitation belongs to the primitive model of counting tables, not to the task itself!

Why so simple a task demands almost everything

The mission of guessing the next word nowhere mentions grand words such as “intelligence”, “consciousness” or “thought”. And yet it is this apparent modesty that makes it all-encompassing. To guess really well in every possible context, a model is constrained to accumulate a colossal sum of cognitive competences.

Look at a few everyday examples: in the sentence “The birds that were flying above the lake ___”, to choose “were” correctly instead of “was”, the model must master grammatical agreement across a parenthesis of several words. In “The capital of France is ___”, it must hold solid geographical knowledge about the world. In “Three plus four make ___”, it must know how to calculate. In “He put the key in the lock, turned it and ___”, it must understand the physics of everyday objects and the succession of human actions. And in “The doctor explained to the worried patient that the operation had been a success, and so he felt ___”, it must understand empathy, psychology and what people feel.

None of these skills was imposed on it explicitly through rules written by programmers. All of them follow naturally from the ambition to predict well: any model that ignores logic or the facts of the world will make large errors when guessing. To predict the next word of a text faithfully means, at bottom, to build an internal simulation of the universe that text describes.

How we measure performance: the compass of perplexity

How do we know whether a model is guessing well or merely groping at random? We return to the principles of the second lesson. The quality of a language model is closely tied to the average quantity of bits of information needed to describe the text using the probabilities the model offers — that is, exactly the entropy from the theory of data compression. Here the circle closes in a fascinating way: the ability to compress and the ability to predict are two faces of the same coin.

In practice this measure is often expressed by an indicator called perplexity. Intuitively, perplexity is the number of equally probable options between which the model hesitates when it has to choose. If a model has a perplexity of 20 on a given sentence, it means that it is as undecided as a person drawing lots with their eyes closed among twenty different words. The lower the perplexity, the surer of itself and the more precise the model. It is essential that the measurement always be made on fresh texts that the model has never seen during training.

In the table above we rediscover the classic trap of overlearning (overfitting). On texts already memorised, the perplexity falls spectacularly as we increase the order of context: the model knows every sentence by heart. On a brand-new text, by contrast, the perplexity explodes! Why? Because the long strings of new words never appeared in its database, and a model built on simple counting stays completely mute and helpless.

Why do purely statistical models based on tables fail so fatally? Because of a merciless combinatorial explosion. In a modest vocabulary of fifty thousand words, the number of possible sequences of just three words exceeds a hundred and twenty-five thousand billion! Not even if we gathered every text ever written on Earth could we fill in such a table. We urgently need a superior architecture: a model able not only to look for literal matches but to generalise by analogy and kinship of meaning. In the next lesson we shall discover how neural networks make this decisive qualitative leap.