What an Answer Costs

The test task

To compare the three techniques of efficiency fairly, we set them to work on one and the same geometric problem: dividing the plane into four distinct regions, marked off both by distance from the centre (radii) and by orientation (angles). It is a frontier curved and winding enough that a minute network cannot untangle it at one go, but compact enough that all the training can run directly in your browser.

Not all the weights for every answer

In a classical dense language model, every word passes through absolutely all the layers and all the billions of weights of the network. But is it really necessary for every circuit to light up for every syllable?

A far cleverer alternative is the architecture with a a mixture of experts (Mixture of Experts, or MoE). The network is made up of dozens of specialised sub-models (called “experts”), and at the input there is a small module of decision (a router). For every token of text, the router chooses just two suitable experts and directs the computation exclusively to them. The total capacity of the model stays gigantic in memory, but the active computing effort for each word falls several times over.

Think of a modern polyclinic with dozens of specialist doctors: when you come in with a knee problem, at reception you are sent only to the orthopaedist and the radiologist. There is no need for all fifty doctors in the hospital to examine you before you get the right diagnosis!

Look at the interactive table: the highlighted row matches exactly the configuration the router was trained for (two active experts). If we try to use fewer experts than the optimum, the accuracy falls visibly; but if we force all the experts to be called at once, performance does not rise — it degrades. The experts learned to collaborate in small, complementary teams, and if they all speak at once their signals jam one another.

Fewer digits for every weight

Every one of the thousands of billions of parameters of a model is, in origin, a decimal number stored with high mathematical precision (on 16 or 32 bits, that is, 2 to 4 bytes per number). During training this millimetric fineness is indispensable for adjusting the minute slopes of learning. But once the model is finished and only answers questions, do we really need so many decimals?

The technique of rounding the numbers intelligently is called quantisation. The architecture stays untouched — the same connections, the same multiplications — but every number is compressed to 8 bits, 4 bits or even fewer. It is exactly the same principle we met in the first lessons with the sampling of sound: turning a fine, costly scale into a scale with wider steps.

The analogy with a digital image is perfect: if you reduce a photographic palette from 16 million colours to 256 carefully chosen shades, the human eye at a distance will barely notice the difference, while the file becomes four times lighter. Only if we cut too deep — below the critical threshold of 3 or 2 bits — does the image suddenly fall apart into ugly patches of colour. So too with the model: compressed moderately, it keeps its fluency almost intact, saving enormously on the memory of the graphics card.

A small model that learns from a large one

The third strategy does not change the existing giant model but sets it in the role of teacher to instruct a compact apprentice model. This method is called the distillation of knowledge.

What is interesting is that the apprentice model is not set to memorise only the bare answers (true or false) but learns from the fine nuances and the probabilities the teacher model emits — including from its hesitations and its subtle associations. Yet the great practical secret of distillation does not lie in some magical formula of teaching: if we had perfect labels in abundance, the small model would learn just as well directly from the data. Distillation shines when examples labelled by people are rare and extremely costly, while raw unlabelled texts are unlimited. We can question the digital teacher billions of times on new data, and its explanations become the apprentice’s study manual.

What does not shrink

All three of these great optimisations aim exclusively at the internal parameters of the network (the weights): how many of them are active at once, with how many decimals we represent them, or how many parameters we keep in all. But there is one component none of them can shrink by itself: the context of the discussion.

The temporary memory of the conversation (known as the KV cache) swells in direct proportion to the number of exchanges and pages traded with the user. Even if you choose a model four times smaller in number of parameters, if the dialogue stretches over hundreds of thousands of words, the consumption of video memory will be overwhelmingly dominated by this archive of the current conversation.

Quantisation applies to the keys and values kept as well, with the same kind of result: much room gained up to a threshold, then a sudden fall. It is the only one of the three that helps on both sides.

In this experiment we could measure the errors with a precise mathematical ruler, because the geometric boundaries were known exactly in advance. But when it comes to a language assistant that composes essays, summarises reports or makes jokes, the quality of an answer becomes a far more slippery matter. In the next lesson we shall discover how the performance of modern language models is measured, and why the industry’s standardised tests give rise to so much controversy.