What Happens When It Answers
An answer does not appear all at once; it is built word by word, and every new fragment calls for a complete pass through all the circuits of the network. What the computer manages to keep in memory between the steps decides whether the sentence appears in a few seconds or in tens of minutes — and the limit of that working space explains why the assistant sometimes seems to “forget” the beginning of the conversation.
Terms: inference prefill autoregressive generation key-value memory (KV cache) context window
The two phases of an answer
Running an already-trained model in order to produce an answer is called inference. Although on the screen everything looks like one continuous process, inference unfolds in two distinct stages, with completely different rhythms of computation. Anyone who has sent a message to an assistant has felt them both:
The first stage is the prefill (or the processing of the prompt). The model receives everything you sent: the question, the system instructions and the earlier exchanges. Because all these words are already known, the graphics card can process them all simultaneously, in parallel, in a single huge pass of computation. From this comes the small initial pause of a fraction of a second after you press Enter.
The second stage is autoregressive generation (the writing proper). The model computes the first word of the answer. Once chosen, that word is stuck onto the end of the text, and the network is run again to guess the second word. Here parallel computation is impossible: you cannot know the fiftieth word before choosing the forty-ninth! That is why the words begin to flow across the screen in a steady, staccato rhythm, one after another.
What is recomputed for nothing, and the KV memory
If we programmed this process in the most naive way, we would meet with a disaster of performance. To write the hundredth word, the model would have to reread all ninety-nine earlier words from scratch, recomputing the same attention layers it has already passed through ninety-nine times!
This repetitive work is an enormous waste of energy. If we look at the attention mechanism, we notice a saving detail: every word looks only backwards, at the terms preceding it. Word number five does not change its meaning, its key or its value merely because we have got as far as writing word fifty.
Consequently, the keys and values of every word can be computed once and kept in an ultra-fast working notebook. This buffer memory is called the key-value memory (or KV cache). At every new word, the model computes only the query of the current position, compares it with the keys already stored and adds the new pair to the notebook. This is no approximation that lowers quality, but the intelligent refusal to repeat the same multiplication of numbers for ever.
The stopwatch: measuring it live
In the interactive window below, a small real transformer model runs directly in your browser. It writes the same text in both ways, and the stopwatch measures the milliseconds spent as the answer grows longer.
The difference between the curves is astonishing. Without the KV memory, the effort needed for every new word grows with the length of the text, so the total time explodes, curving steeply upwards. With the KV memory, the effort for each word stays constant, and the total time grows smoothly, straight and predictably.
On short texts inside a web page, the theoretical slope is slightly softened by the fixed operations of the operating system. But on real servers generating thousands of words, this difference is what separates a fluent real-time conversation from an agonising wait of several minutes.
The most important point: the final result is rigorously identical in both scenarios, letter for letter. The KV memory cuts no corners and compromises none of the network’s thinking; it is a pure optimisation of computation.
What the memory costs: the fight for room
In computing there are no free miracles: we have saved computing time, but we pay in memory occupied. For every position in the conversation we must keep vectors of keys and values at every level and at every attention head of the network.
As the context stretches to tens of thousands of words, this buffer comes to occupy tens of gigabytes of ultra-fast video memory (VRAM). For short questions the weight is carried by the network’s fixed parameters; for long conversations or whole documents, the KV memory often becomes costlier and heavier than the model itself!
To tame this thirst for memory, modern models use ingenious architectures such as grouped-query attention (GQA), setting several computing heads to share the same keys and values. Thanks to these innovations it has become possible for today’s assistants to read hundreds of pages in a single breath.
The context window and the illusion of “forgetting”
The working memory of an assistant is not infinite; it has a maximum size fixed by construction, called the context window. This window holds everything the network can have before its eyes at a given moment: the initial instructions, your question, the earlier exchanges and the text it is generating now.
When a discussion grows too bulky and exceeds this capacity, the first words have to leave the window to make room for new ones, as on a conveyor belt. Nothing breaks down and no error message appears: quite simply, the beginning of the conversation ceases to exist in the equations of attention.
From this is born the phenomenon users often describe by saying that “the artificial intelligence has forgotten what I told it ten messages ago”. But the model has not “forgotten” in the human sense. People forget memories they once had in mind; for a language model, what has left the context window never existed. It lives exclusively in the present of the tokens before its eyes at that very moment.
Now we understand how the assistant thinks and computes: a structure trained on language, shaped by dialogue and accelerated by the key-value memory. And yet a troubling enigma remains: why does this sophisticated system sometimes invent facts, historical dates or non-existent books, and assert them with unshakeable confidence? In the next lesson we shall discover why models hallucinate without a trace of bad faith.