Why It Invents Things

What happens when it does not know: the actor who cannot keep silent

When you ask an assistant about an obscure fact it has never met, no error alarm goes off in its network. It does not enter a state of blockage and does not show a blank screen. Instead it does exactly what it was built to do at every step: it computes a distribution of probabilities and draws from it the most plausible next word.

The answer that spreads across the screen has an impeccable bearing: rounded sentences, a calm and convincing tone, proper names and exact years. In the cognitive sciences this phenomenon is called confabulation (known in everyday speech as hallucination): the production of a flowing, elegant narrative in place of a documented truth.

The model is like an actor in improvised theatre who has one absolute prohibition: he may never fall silent on stage. If a bizarre line about an invented event is shouted from the hall, the actor will not stop to say that he does not know the play; he will improvise a resounding monologue on the spot, choosing the words that sound most natural in context. It is important to remember: the model is not lying. To lie presupposes knowing the truth and deciding knowingly to say something else. The model cannot make this distinction, because nowhere in the mathematics of its matrices is there a register of objective truth.

What enters the reckoning: plausibility vs truth

The small demonstration model below is a character counter that looks only at the last few signs of the question. In the figure, the characters that really count for the decision are highlighted, while the rest of the sentence is faded out:

If two completely different questions end with the same sequence of letters, our model will generate exactly the same continuation. It is not a chance error: the deep meaning of the first part of the question never even reached the decision circuit.

Of course, a modern assistant has attention windows covering thousands of words at once, so it will not be fooled by the last few letters. But the nature of the computation stays identical: the network measures the harmonious compatibility of words with each other, not their correspondence with the physical reality outside the text.

What the score of a text actually measures

To every sentence it analyses, a language model can assign a numerical probability score. Users often read this score as an index of certainty or of factual truth. Let us see what it actually measures.

In the experiment below we set the model to judge and rank eight possible answers for each question. If the score measured factual correctness, the true answer ought always to come first:

In our little toy model the ranking looks chaotic because its short memory misses the beginning of the question. Large models do incomparably better, but the principle is the same: the score reflects how natural and predictable the succession of words is in that language, not whether the statement agrees with historical or scientific reality.

Calibration: confidence displayed vs certainty in fact

Another fundamental question is this: when the model states that an option has an 80% probability, is it right in 8 cases out of 10? This correspondence between the mathematical confidence displayed and the real rate of success is called calibration.

We measure this on a new text, never seen during training. We divide the predictions into groups by level of confidence and check how often the option ranked first really was the correct one:

Look at the graph: nearly all the points fall below the ideal dotted diagonal. This points to a universal problem of language models: they are over-confident. They display categorical certainty even when they are merely guessing. The largest gap appears not at the top but in the middle range, where the model seems fairly sure of itself without having a solid basis.

Even perfect mathematical calibration would not solve the problem of truth. It would tell us only how sure the model is that it is guessing the next symbol of the text, not whether that sentence corresponds to the truth of the physical world.

Why it cannot simply say “I don’t know”

It would be so simple and convenient if, when it has no answer, the model would say openly: “I don’t know”. But in the mechanism of a transformer the phrase “I don’t know” is not an internal state of alert or an admission of ignorance; it is merely one string of letters among others, with a certain statistical probability.

In the great library on which the model was trained — encyclopaedias, books, scientific articles and web pages — the texts almost never contain the line “I don’t know”. Books explain, articles assert, and authors speak with authority. The model has taken a clear lesson: after a question, the natural text continues with a detailed exposition of facts, not with a pause for modesty.

Through fine-tuning and alignment with human feedback, modern assistants are trained to utter refusal formulas such as “I do not have enough information” more often. But even this reaction is only a linguistic pattern learned on certain phrasings, not a conclusion springing from honest introspection. This is why the problem cannot be remedied by adding to the request exhortations such as “please be very careful and do not lie”: your words only shift the distribution of probabilities a little, and cannot create an organ of factual verification that simply does not exist in the architecture.

If we cannot easily change the convictions engraved in the billions of parameters, can we perhaps change its behaviour on the fly, without touching its network at all? What happens if we put a few examples before it in the text of the question itself? In the next lesson we shall discover one of the most spectacular properties of modern assistants: learning directly from context.