How the Word Is Chosen

What the model offers and what is missing

When all the computations in the attention layers and the neural networks are done, the model does not produce a ready-made sentence. For the position immediately following in the text, it produces only a list of scores for every fragment of a word (token) in the vocabulary. Once these numbers are turned into percentages, we obtain a probability distribution: a ranking of all the possible continuations, from the natural to the absurd. That and nothing more. Not a single word has yet been spoken.

The passage from the list of probabilities to a concrete word on the screen is a completely separate step, called sampling. This step is not learned during training on text; it is a rule we set from outside. For exactly the same model and the same question, we can obtain radically different answers merely by changing the way we draw the winning ticket from the urn. When you read a text generated by a digital assistant, you are in fact watching a dance between two forces: what the network computed, and the algorithm that chose from its computations.

The model in the figures below is a small character counter trained on the text of the first chapters of this course. It looks at most six characters back and falls back on a shorter context when the sequence has rarely been met. Although its vocabulary and its memory are small compared with today’s giants, the mechanism of selection is identical: at every step we have a list of possibilities before us and must decide how to take the next step.

The most probable piece and its trap

At first sight, the wisest thing would be to choose every time the option in first place: the fragment with the greatest probability. This direct strategy is called greedy choice (or decoding of the greedykind). At every step the model seizes without hesitation on the surest variant.

The greedy choice, however, has a flaw that few suspect before seeing it at work: it is completely rigid. Starting from the same beginning, it will always produce, letter by letter, exactly the same text. Worse, natural language does not work by placing the most banal word after another for ever. Following blindly only the surest road of the moment, the model very soon arrives in a state it has passed through before, and from there it begins to repeat itself endlessly. The text sticks in a monotonous loop it can no longer escape. The most predictable local step leads into a vicious circle.

Temperature: between freezing and boiling

If the strict choice of first place leads to mechanical repetition, we can try the opposite: to throw a die at random in proportion to the model’s probabilities. If one letter has a 60% chance, another 30% and another 10%, the die will choose them in exactly those proportions. The text becomes more varied, but another problem appears: the long tails of the distribution, that is, the thousands of improbable variants, come to be drawn too often, spoiling the coherence of the sentence.

To master this balance, we place between the model and the text an essential lever called temperature (usually written T), by analogy with the physics of heat. When it is cold, atoms freeze and stay in their state of least energy; when it is hot, they stir strongly and explore unexpected states.

In generating text, the temperature lever controls how much we flatten or sharpen the probabilities. When the temperature is brought down towards zero, the differences deepen enormously: the leading option becomes almost certain, and the rest of the variants disappear. The text becomes precise, concentrated and predictable. When the temperature rises above one, the differences are wiped out, and the die gives almost equal chances to all the variants. At temperature one we leave the probabilities exactly as the model saw them, and at temperature zero only the absolute peak remains, that is, the greedy choice.

It is worth noticing that this lever cannot create alternatives out of nothing where the model is absolutely sure. If after a given context there is only one possible continuation (gathering 100% of the probability), it will remain the only one chosen at any temperature. Temperature works solely on the uncertainty and the shades that already exist in the network’s computations.

How the generated text sounds

The real effect of sampling is seen on a whole text, where the decisions link up step by step. Every letter or word chosen is added to the end of the text and becomes the context for the next step. A single slightly bolder decision sends the whole sentence on a new trajectory.

The little counter in this demonstration has learned only the sound-shape of English words and which letters usually stand together, not the deep sense of thoughts. The point of the exercise is to see how the choosing behaves: at a very low temperature the text sticks in a mechanical refrain, and at too high a temperature it loses even the structure of its syllables.

Measuring the trade-off

We can measure this trade-off precisely by following two quantities: how often the sequences of characters repeat (the sign of sterile loops) and how content the model is with the choices made (the average probability assigned to the characters chosen along the way).

A slight rise in temperature from zero to 0.2 makes the repetition collapse spectacularly, the text becoming suddenly freer and more fluent. The average probability, on the other hand, begins to fall gradually as we raise the temperature.

The first row of the table reveals a great surprise, however: the greedy choice takes at every step the variant of greatest probability, yet the text generated as a whole is not the most probable one! Often the average confidence is higher at a low but non-zero temperature. Getting stuck in monotonous loops, the greedy choice comes to take clumsy steps that it is then obliged to repeat for ever. To choose the best variant in the short term is not to find the best road in the long term — a lesson that holds far beyond neural networks.

The right temperature does not follow from a fixed formula. It is settled according to what we expect of the assistant: for exact tasks, such as programming or solving problems, we prefer low temperatures; for natural dialogue, summaries or creative writing, we choose a moderate one.

Cutting the tail: nucleus sampling (top-p)

Temperature changes the slope of the whole distribution, but it does not make the improbable options vanish altogether. In a modern vocabulary of tens of thousands of fragments, the thousands of odd words in the tail of the ranking gather together a mass of probability large enough for one of them to be drawn at random, spoiling the logic of the sentence.

The second control lever cuts this tail instead of deforming it. We order all the variants by decreasing probability and keep only the first, until the sum of their probabilities reaches a set threshold, 90% for instance (p = 0.90). All the other variants in the tail are removed completely, and the die is thrown only among the leading options. This method is called nucleus sampling (or top-p).

The great advantage of the nucleus over a rigid cut is that it adapts itself to the context. Where the continuation is obvious (“The capital of France is…”), a single option may gather over 90% at once, so the rest are cut instantly. Where the context is open and allows many good variants (“On the table there lay a…”), the nucleus widens automatically and keeps dozens of plausible options.

Temperature and nucleus sampling work hand in hand: the first regulates the tone of the choice (how sharp the difference between options is), and the second acts as a safety net preventing the model from wandering off into absurdity.

Even with impeccably tuned sampling, a pre-trained model knows one thing only: how to continue a text that has been begun. If you put a question to it, its natural reflex is often to compose another question or to continue with a page of a manual, because that is what it saw on the internet. In the next lesson we shall discover how this immense statistical memory is turned into a cooperative assistant, able to hold a dialogue and to answer commands.