Attention

The direct look: why we no longer make summaries

The previous lesson left us facing two major obstacles. First, we needed a piece of information at the beginning of the text to reach the end intact, without being crushed into an intermediate summary that fills up quickly. Second, we wanted to escape step-by-step computation, so that the processing of words could run simultaneously on the thousands of cores of graphics chips.

Both dilemmas are solved spectacularly by a single bold idea: if a position in the sentence needs clarification, we let it look directly at the words that hold that clarification! We no longer carry notebooks and no longer wait in a queue. But how can a network know which words to turn its attention to, if nobody has left it hand-written instructions?

The dialogue in the hall: Question, Badge and Content

To settle who talks to whom, every word receives three distinct roles (three vectors derived by learning):

The Query (or Query) is the question the current word puts: a vector that specifies exactly which piece of the puzzle it is missing. The Key (or Key) is the badge every word wears in plain sight: a description of the kind of knowledge it can offer. The Value (or Value) is the informational content proper that the word delivers if it has been chosen.

The match between a question and a badge is computed by a simple operation: the dot product. The more the two vectors point in the same direction on the semantic map, the stronger the match. The resulting scores are turned into percentage weights whose sum comes to 100%. The final result is a harmonious blend of the values offered by the other words, each weighed in proportion to the interest it aroused.

Imagine a large hall full of people where someone shouts: “Who has a medical kit?”. Everyone in the room checks their badge; those who know nothing about it keep quiet, and the qualified ones answer and offer their help. Nobody dictated in advance who would step in: the dialogue forms dynamically and instantly on the basis of the match between demand and supply.

Attention at work: a guiding beam

In the panel below you can follow an attention mechanism trained live on the memory task: the sequence begins with a symbol (“A” or “B”), continues with a neutral filler zone, and at the end is asked to identify the first character. You can read concretely the matches between the query of the last step and the keys of all the earlier positions.

Notice how the searchlight of attention shines with greatest intensity on position zero, where the symbol sought lies, ignoring the sterile characters in the middle completely. Nobody programmed this search: the network discovered by itself through training that this is the only strategy that solves the problem successfully.

Notice also a curious asymmetry: for sequences beginning with “A”, the initial position captures over ninety per cent of the gaze. For those with “B”, the attention is more dispersed, because the absence of clues for “A” is clue enough to infer “B”. In neural networks there is no requirement of abstract symmetry; models adopt any efficient mathematical shortcut that secures them the victory.

The comparative test: the memory that no longer dies away

Let us put the attention mechanism through the same hard test at which the recurrent network gave way after only fifteen steps. We run the training on increasing lengths and measure the accuracy on completely new strings.

The result is categorical: attention stays unbeatable and shows no sign of tiring as the distance grows! For it, the notion of physical distance does not exist: position 1 and position 100 lie at exactly the same computational effort — a single direct comparison between question and key.

The direct bridge: why the signal no longer dies

In recurrent networks, the error-correcting signal had to cross an endless series of successive multiplications until it was worn away entirely.

In the attention mechanism the road has a single step. The final output is linked by a direct bridge to the initial position, the factor of the link being the attention coefficient itself. The gradient signal flows unhindered along this footbridge without being crushed, arriving clean and strong at its destination. The fragile chain of Chinese whispers has been replaced by a direct telephone line.

The great unblocking: everything computed at once

The second triumph of attention has nothing to do with theory and everything to do with the pure engineering of chips — and it changed the fate of a whole industry.

The computation of the link between word 3 and word 20 no longer depends on the intermediate stages: it can be carried out in exactly the same millisecond as all the other possible pairs. All the questions and all the keys of the whole sentence are gathered into two huge matrices, and the matches are found by a single massive matrix multiplication — exactly the operation modern graphics cards were designed for.

Here the great explosion of artificial intelligence took place. Attention won not necessarily because it is a more philosophical model than recurrence, but because it could be trained on supercomputers thousands of times faster. In technology it is often not the most complicated idea that triumphs but the one best suited to parallelisation on a huge scale.

Many looks: attention with several heads

A single beam of attention can follow a single link at a time. In a human sentence, however, countless things happen at once: we must follow the agreement of subject and verb, work out what a pronoun refers to, and notice the direct object.

The architectural solution is simple: we train several attention mechanisms in parallel, called attention heads (attention heads). Each head holds its own matrices of questions, keys and values. With no directive imposed from outside, the heads specialise spontaneously through a division of labour: one follows syntactic relations, another temporal links, and another gender agreement.

A single layer of attention nevertheless has a logical limit: it can locate a word by its badge, but it cannot directly extract what lies immediately after that word. To carry out compound reasoning we must stack several such layers one on top of another — the decisive step towards the Transformer architecture of the next lesson.

The price of freedom: the quadratic cost

Nothing in engineering comes free. For every word to be able to look at absolutely every other word of a sentence of length N, the network must evaluate a grid of N times N connections. This effort grows with the square of the length of the text: if we double the number of words, the number of computations multiplies fourfold!

There is one more essential detail: attention is an extraordinary system of steering and routing, but it only moves information from one place to another without transforming it deeply. A brain needs not only to gather clues but also to digest them and draw conclusions. In the next lesson we shall assemble attention alongside a dense block of neural processing, giving birth to the famous Transformer brick.