Word Order
In the pure attention mechanism there is no notion of time or space: without an extra guide, the model considers the sentences “the dog bit the man” and “the man bit the dog” identical. To tell apart the natural order of thoughts, every word receives a seat number or a temporal fingerprint — from the hands of a sinusoidal clock to the modern geometric rotation of vectors.
Terms: permutation invariance positional encoding sinusoidal waves (a clock with many frequencies) relative distance rotary encoding (RoPE)
Attention’s great weakness: the bag of disordered words
Attention computes the match between a question and all the available keys, converts the scores into weights and adds up the resulting values. But notice a surprising detail: nowhere in this whole chain of operations does the physical position of the word in the sentence appear! Since addition is commutative (2 + 3 equals 3 + 2), the sum of the values gives exactly the same result whatever the order in which the terms arrived.
The consequence is radical: if we shake up the words of a sentence like dice in a cup, the output of attention stays unchanged. Not almost the same, but identical to the last decimal!
It is like a heap of books tipped out anyhow on a table: you recognise the title of each, but without the page numbers printed in the corner you have no way of knowing which chapter comes after which.
In the interactive experiment above, the weights are chosen at random in order to show that this blindness has nothing to do with the quality of the training but is an intrinsic mathematical property of the attention formula.
The chronological test: when order is everything
For human language, order is vital: it is one thing to say “the cat hunts the mouse” and quite another “the mouse hunts the cat”.
We can illustrate this impasse by a simple experiment: we generate a string beginning with the letters A and B in an order chosen at random (AB or BA), followed by neutral characters, and at the end we ask the network which letter appeared first. The vocabulary is identical in both cases; only the chronology differs.
Without an explicit marker of order, the model’s accuracy is exactly 50%, the same as tossing a coin. The model produces exactly the same mathematical vector for both variants, so it is forced to guess blindly. This is not a weakness of optimisation: the information about order simply does not exist in the data received!
The ticket with the seat number: the position marker
The solution is simple and ingenious: before entering the first Transformer storey, every word receives a small numerical supplement — a special vector reflecting only its position in the sentence (place 1, place 2, place 3 and so on). This bundle is called a position marker, and the procedure of assigning it is called positional encoding.
Once this stamp has been added, two appearances of the same noun in different places will have slightly different starting vectors, generating distinct keys and questions. Attention thus regains the ability to tell order apart without its internal formula suffering any change at all.
Why is the marker added by summing instead of being tacked on to the tail of the vector? Because the conveyor belt has a fixed width calibrated in advance; adding an extension would have needlessly enlarged all the matrices in all the layers. By addition, the word’s shade of meaning and the coordinate of its position interweave harmoniously in the same vector.
The table solution: labels learned from practice
The most direct method of assigning these markers is to create a table of learned vectors: we reserve an adjustable vector for position 0, another for position 1 and so on, letting the network discover their best values during training, just as it does for the words of the vocabulary.
Although it works well, this approach runs into a rigid barrier: the size of the table is fixed from the very start. If the model was trained with a catalogue of 512 positions, it will have absolutely no idea what to do if we give it a text of 513 words! For position 513 there is no row in the table, and the network is stuck.
The harmonic clock: sinusoidal encoding
To get past this limitation, the authors of the original Transformer architecture (2017) turned to a universal mathematical formula: sinusoidal encoding. Instead of storing static values in a catalogue, every position computes its own marker using the trigonometric functions sine and cosine at different frequencies.
Imagine the mechanism of a clock with many hands: the first components of the vector oscillate rapidly, like a nimble second hand, clearly distinguishing immediately neighbouring words; the next components move like the minute hand, and the last shift extremely slowly, like the hour hand, helping the network to tell the beginning of a chapter from its ending.
The relative magic: why distance matters, not address
The genius of the sinusoids lies not only in their elegance but in an essential geometric property: the dot product between the markers of two positions depends solely on the relative distance between them, not on their absolute placing in the document.
Every line of the table is uniform: the closeness between position 0 and position 5 is mathematically identical to that between positions 16 and 21! In human language this is what matters: the expression “the preceding word” has the same grammatical meaning whether we are in the first sentence of the novel or on the last page.
A shift by a fixed number of positions translates geometrically into a rotation by a fixed angle on the circle. A neural network can thus learn stable relative relations (such as the agreement of an adjective with the noun beside it) without being affected by where in the text the construction appears.
The modern standard: rotary encoding (RoPE)
The latest generation of models (such as LLaMA or Mistral) uses a still more refined technique: rotary encoding (or Rotary Position Embedding — RoPE). Instead of adding an extra vector onto the conveyor belt right at the start, they rotate the query and key vectors by an angle proportional to the index of their position, exactly at the moment when attention computes its dot products.
This approach is far cleaner: the residual belt stays pure and is not polluted with positional noise, and the effect of relative distance is injected precisely where it is consumed — in judging the match between two words.
With this last piece, the Transformer architecture is fully drawn: the text is cut into tokens, turned into semantic vectors, enriched with a sense of order and processed by a stack of deep blocks in order to predict the next symbol. But what exactly turns this mathematical skeleton into a giant capable of captivating conversation? The answer lies in training at enormous scale. In the last lesson of this section we shall explore the laws of scaling: how the volume of data, the power of computation and the number of parameters work together to generate modern intelligence.