Tokenisation

The dictionary that can never be finished

In the previous lesson we saw that every notion needs a vector of numbers of its own, which presupposes drawing up a complete catalogue of all possible words. But in a living language such a definitive list is an impossibility. Unexpected proper names, technological neologisms, typing errors, numbers or words invented in play will keep appearing. Any term missing from the catalogue runs into a wall: having no vector attached to it, the network can do nothing with it and is forced to replace it with a dry symbol for “unknown word” (the famous [UNK]), losing all the information.

In languages rich in grammatical forms the nightmare multiplies. Even in English, “run”, “runs”, “running”, “ran” and “runner” would have to be treated as completely separate entries in the catalogue, each with its own vector learned from scratch; in a language such as Romanian, where a single noun has a dozen forms, the catalogue explodes outright. Although any speaker recognises the common root at once, the model would have to rediscover this obvious link from hundreds of parallel contexts.

The other extreme: letters instead of words

A natural reaction would be to go down to the minimum level: why not work directly with letters? An alphabet has only a few dozen characters, the list is closed for ever, and any imaginable word can be transcribed without exception.

The price paid for this simplicity is enormous, however. An ordinary sentence of twenty words turns into a string of more than a hundred and thirty individual letters. The network would be obliged to process sequences six times longer for the same quantity of information. Worse still, an isolated letter carries almost no meaning: what does the letter “p” mean on its own? The model would have to spend an enormous amount of computational energy merely to reconstitute the sense of every word, letter by letter, every single time.

The best solution is a middle road: cutting the text into fragments of intermediate size, larger than a letter but smaller than or equal to a whole word. These bricks are called tokens. The principle matches the logic of Morse code from the first lesson perfectly: words met very often receive a compact piece of their own, while rare terms are elegantly assembled from a few smaller pieces.

The Lego bricks are born of counting

Who decides where we cut the words? Not a linguist and not a committee of experts. The division is made purely statistically, by an automatic algorithm. We start from the text broken down completely into letters. We count which two symbols appear most often side by side and unite them into a new brick. Then we count again on the new text and unite the next leading pair. We repeat this simple procedure a few tens of thousands of times.

This recipe is called merging frequent pairs (known internationally as Byte-Pair Encoding or BPE). It was invented in 1994 as a technique for compressing files and brought back to the fore in 2015 for language processing. The result is remarkable: the usual prefixes, the grammatical suffixes and the short words become unique pieces in their own right, without the algorithm knowing a single rule of morphology. For the model, the notion of a “word” does not come from a school textbook but from the pure statistics of reading.

In the interactive example above we use a small text of a few hundred words, so the merges faithfully reflect those sentences. In industrial systems in real life, tokenisers read whole libraries of tens of billions of words and stop the consolidation at a balanced vocabulary of fifty or a hundred thousand standard pieces.

How a new text is sliced

Once the list of merges has been fixed in the vocabulary, any new text can be sliced instantly by applying the rules in exactly the order learned. Very popular words stay intact as a single piece. Words never seen before, rare terms or unusual family names, by contrast, are automatically broken down into smaller pieces — but they are never lost!

The pieces highlighted in bold represent whole words captured in a single movement. The others are syllables or splinters of words. You can see the contrast clearly: ordinary sentences compact into a few large blocks, whereas texts loaded with figures, codes or rare names are ground into numerous small tokens.

The hidden inequality between languages

The Lego pieces obtained depend entirely on the language of the texts on which the initial counting was done. A tokeniser trained predominantly on English pages will not hold in its vocabulary the endings and particles peculiar to other languages, so it will break each of their words into a multitude of minute splinters of two or three letters. The figure below shows the effect in the mirror: two vocabularies of the same size, one trained on English and the other on Romanian, slicing the same English sentence.

This difference has major practical consequences. The memory and the speed of a language model are limited by the maximum number of tokens it can hold at once (the so-called context window), and the financial cost of API calls is charged on the number of tokens processed as well. A language insufficiently represented at the tokenisation stage is fragmented excessively, consumes memory two or three times faster and becomes far more expensive to use for exactly the same volume of ideas. It is a discreet structural inequality, fixed before the network proper has learned its first concept.

Blind spots: what the model cannot glimpse

From the moment the text has been sliced into tokens, the individual letters vanish from the network’s field of view. The model receives only a string of numerical codes corresponding to the pieces in the vocabulary. If the term “strawberry” makes up a single compact piece in the vocabulary, then to the network it is an atomic symbol, as opaque and indivisible as an Egyptian hieroglyph.

This technical reality explains blunders that often amuse and disconcert users. Why do the models stumble when asked “how many letters ‘r’ are there in the word strawberry”? Simply because the model does not see the component letters! The same reason makes reversing a word, composing fine acrostics or playing with phonetic rhymes surprisingly difficult: they work at a level of detail that tokenisation has hidden behind a single code.

The hesitation in elementary arithmetic has its root here too. Large numbers are not cut according to their positional value (units, tens, hundreds) but according to how often they appear in scanned books. The number “1024” might be a single token (being often met in computing), while “1025” might be sliced bizarrely into two or three pieces: “10” and “25”. To add these numbers, the network has to compensate for an arbitrary fragmentation that respects no mathematical logic.

The latest generation of models softens these clumsinesses by forcing the tokeniser to keep every digit as a separate piece and by drilling extensively on detailed spelling exercises. Even so, the structural barrier remains: the basic level at which a model thinks is not the letter but the token.

Thanks to tokenisation, any human text becomes a clean string of numerical identifiers, and each identifier rests on a vector learned on the semantic map. The pieces of the puzzle are ready. But the great question remains: what task do we set the network to solve with them? In the next lesson we shall see that an apparently banal mission — guessing the next word of a sentence — hides beneath it the key to the whole understanding of the world.