From Continuer to Assistant

What is missing after pre-training

Pre-training teaches a single task: to guess the next fragment of text out of a huge library of books and web pages. The model that comes out of this stage is a master of the patterns of language, but it has not the faintest idea that it is a computer program or that somebody at the keyboard is waiting for its help.

If you write it a question such as “How is rain formed?”, it treats it as any old piece of text and looks for the statistically most frequent continuation. On the internet, a question is often followed by another question (“And why does it snow in winter?”), by a multiple-choice list (“A: by condensation, B: by evaporation”), by a scene from a novel or even by a forum quarrel. A clear, direct answer is only one of the thousands of possible continuations, and not necessarily the commonest. The model does not refuse to help you out of stubbornness; it simply does not know that it has been asked anything.

Fine-tuning on dialogues

The solution is to teach it the good manners of conversation. We show it a new set of data, arranged strictly in the form of dialogue: every request or question from a user is always followed by an impeccably written answer from an assistant. The training continues with the same mechanism of predicting the next word, but on these carefully chosen examples. The stage is called supervised fine-tuning (often abbreviated SFT, in English).

The quantity of text needed here is surprisingly modest. While pre-training swallows thousands of billions of words gathered from the whole internet, fine-tuning needs only a few thousand or tens of thousands of quality examples written by people. In this phase the model learns no new notions about the world — it already knows the facts from pre-training. What it learns is a posture: how to behave as a useful, kindly and concise guide.

The interactive table above follows two things on new questions: how often the answer has the form of a finished informative sentence (the attitude of an assistant) and how often it contains at least one term from the original question.

The first column rises quickly right after the first training pairs: the model catches the tone and style of an answer at once. The second column changes more slowly, because the substance comes from what it accumulated before. Fine-tuning provides the assistant’s coat; it cannot create out of nothing the knowledge that was not taken in during pre-training.

In a large model this transformation looks spectacular, because the network already has a vast quantity of encyclopaedic information in its memory and fine-tuning merely unlocks it in the form of dialogue. In the small counter on this page, whose memory is minute, you can see the two forces separated in their pure form.

What cannot be written in a workbook

Why do we not stop at fine-tuning on dialogues? Because for most complex requests there is no single perfect answer that somebody could write as a supreme template. If you ask “explain the theory of relativity for a ten-year-old” or “write a poem about autumn”, there are thousands of possible variants, and their quality depends on clarity, rhythm, warmth and tact.

It is extremely hard and costly for human experts to write hundreds of thousands of perfect answers to every imaginable question. But people have a remarkable advantage: a natural asymmetry of judgement. We find it very hard to write a masterpiece from scratch, but very easy to look at two ready-written texts (A and B) and say on the spot: “A is clearer and more to the point than B”.

On this asymmetry the next revolutionary step is based: instead of writing demonstrative examples, we gather preference comparisons. We move from the rigid question “what is the correct text, letter by letter?” to the natural question “which of the two variants does a person prefer?”.

The reward model

Human comparisons (“A is better than B”) cannot be applied directly to the language network, because a general preference does not tell the model which words exactly had to be corrected at each step. That is why we build an automatic referee: a reward model.

This referee is a separate neural network that receives a question together with an answer and awards a numerical mark (a score of satisfaction). It is trained on thousands of comparisons made by people, following an intuitive rule: every time the human evaluator preferred answer A to answer B, the reward model must give A a higher mark than B. The absolute value of the mark does not matter, only the correct ranking.

The weights visible in the graph reflect the criteria the referee has inferred from your choices. Nobody programmed rigid rules into it: it observed which features tip the balance and learned to value concision, the absence of repetition or the clear structure of sentences.

The accuracy indicator measures how often the reward model correctly anticipates the preference on a new pair of texts it has not seen before. This is the proof that the referee is not mechanically memorising the examples but catching the evaluator’s general taste.

What is really reinforced: the traps of optimisation

A reward model cannot read people’s inward intentions; it detects only statistical correlations between features of the text and the votes received. If two qualities often appear together in the answers that were liked, it will reward them both without discrimination.

If you experiment with the automatic annotator above, you will notice that, whatever main criterion you choose, a second feature that happened to accompany the rule in the generated texts always rises alongside it.

In real assistant models this trap is called Goodhart’s law: when a measurement becomes a direct target of optimisation, it loses its value as a measure. If human evaluators liked longer and more detailed answers, the reward model learns that “length means quality”. Before long the assistant begins to give interminable answers, excessively ceremonious or full of an artificial confidence. Many of the oddities of commercial assistants come precisely from this discrepancy between what we actually wanted and what the reward function managed to measure.

Best of several, and learning from preferences

Once we have a trustworthy automatic referee, we can set it to work improving the assistant by two complementary methods.

The most direct method is the selection of the best of several (known as Best-of-N): when the user puts a question, the model generates several independent variants of an answer behind the scenes (eight, say). All are marked by the reward model, and only the winner with the highest mark is delivered to the user. It is a simple and robust approach, although it consumes more computing power for every answer.

The second method, far deeper, is learning from preferences (or alignment by reinforcement learning — RLHF). Here we change the internal weights of the generating model directly: it is rewarded whenever it produces well-rated texts and discouraged when it goes off the rails, while keeping a safety anchor to the initial model so as not to destroy its natural fluency of language.

By joining these three essential stages — the vast pre-training on books, the fine-tuning on dialogues and the refinement through human preferences — a simple engine for predicting words is turned into a cooperative and intelligent assistant.

But what in fact happens in the innards of the computer in every second while the assistant is composing its answer? How much text can it keep in memory, how does it avoid recomputing everything from scratch for every letter, and why does it need a special buffer memory? In the next lesson we shall lift the bonnet on the engine in full operation.