Vector Representation

A word is not a simple number

The first idea that comes to mind seems easy enough: why not simply number the words of the dictionary? “Dog” gets the number 17, “cat” gets 18, and “apple” gets 19. This naive approach, however, slips a false order into the data, one that leads the network astray. To a mathematical formula, 17 and 18 are immediate neighbours, and 18 lies exactly halfway between 17 and 19. But a “cat” is not an average of a dog and an apple! We saw in the first lessons that a binary label is only a catalogue address, not a description of the contents. If we take the address for the meaning, the computations lose all sense.

The second idea tries to do away with this artificial closeness. Instead of a single number, we give each word a long list of zeros, with a single 1 switched on at its own slot. This approach is called one-hot representation (or, in English, one-hot encoding). Its advantage is that it is perfectly impartial: it no longer assumes any forced kinship between words.

But this very neutrality makes it powerless. In the one-hot representation, any two different words stand at exactly the same mathematical distance from each other. To the network, “cat” is as foreign to “dog” as it is to “submarine” or “philosophy” — there is not a trace of meaning. What is more, the list becomes enormous: for an ordinary vocabulary of a hundred thousand words, each word would be a gigantic string of a hundred thousand zeros and a single 1.

The map of meaning: sense born of company

The real revolution came from a third approach: not to decide the numerical values by hand, but to leave them to machine learning. Each word receives a compact list of real numbers — a few dozen, three hundred or a thousand — which work exactly like GPS coordinates in a space of many dimensions. This ordered list of numbers is called a vector, and the placing of concepts on this many-dimensional map is called vector representation (or embedding).

But how can a machine find the right coordinates for each word? The secret was formulated with inspiration by the linguist John Firth as early as 1957: “a word is known by the company it keeps”. Words that express related notions tend to appear in similar contexts. We often say “I drank a hot tea” or “I drank a hot coffee”, but rarely “I drank a hot tractor”. Meaning is not an abstract definition in an immovable dictionary but a living pattern of use in everyday language — and occurrences in text can be counted and measured mathematically.

This intuition becomes an ingenious training game. From a vast library of texts we choose a word and ask the network to guess which terms lie in its immediate neighbourhood. The guessing itself is only a means to an end: what really interests us are the coordinates that are gradually adjusted in the process. To predict the contexts correctly, the model is obliged to push into the same region of the map those words that share the same companions in a phrase.

In the interactive example above we have prepared forty-eight short sentences made up of thirty-eight words. Each word receives only two coordinates, so that it can be drawn as a point on a flat screen. Without anyone having programmed categories such as “fruits”, “animals” or “cities”, the points migrate by themselves and form distinct islands, simply because they appear in similar sentences.

Beware the traps of contexts that are too narrow: if a word appears in only one kind of phrase, it is dragged by force into that narrow circle. In our little teaching text, “from” ends up glued to the fruit because it appears exclusively in “buy apple from market”. In a broad corpus of natural language it is used in thousands of different contexts and migrates naturally towards the neutral centre of the map.

Distance and bearing on the compass of meaning

On such a map of meanings, what matters is not the length of the vector (the distance from the centre of the diagram to the word) but the angle formed by the arrows of the two vectors. If we see each word as a compass needle pointing out from the origin, two words with close meanings will point in almost the same direction, however often or rarely they were met in the text.

The standard mathematical instrument for this comparison is called cosine similarity. It measures how well aligned the two directions are: it takes the value 1 if the arrows point in exactly the same direction, falls to 0 if they are perpendicular (with no relation of sense at all), and reaches −1 if they point in completely opposite directions. This is the proximity score displayed beside each neighbour in the diagram above.

The geometry of sense: directions with meaning

When such a representation is trained on billions of words from the internet, a spectacular phenomenon occurs that nobody programmed on purpose. The space acquires a geometry full of logic: the move from the vector “king” to the vector “queen” is almost parallel to the move from “man” to “woman”, from “actor” to “actress” or from “grandfather” to “grandmother”. One axis of the space comes to encode grammatical gender, another encodes the passage from singular to plural, and another links countries to their capitals.

The consequence is fascinating: we can solve linguistic analogies by simple additions and subtractions of vectors. If we take the coordinates of the word “king”, subtract the vector “man” and add the vector “woman”, we arrive mathematically at a point extremely close to “queen”. This property, brought to light in 2013 with the Word2Vec model, caused a real sensation in the scientific world, showing that relations of human thought can be mapped geometrically.

If we try this operation on our small teaching example, we notice that the model identifies the right region — all the feminine words appear together in the first places — but cannot finely tell them apart. “Queen”, “actress” and “grandmother” receive almost identical scores, and their order is purely a matter of chance. Why does this happen? Because in the few sentences of the exercise, the three words appear in exactly the same sentence patterns, without enough nuance to distinguish them.

Here we grasp the decisive role of sheer volume of data. The learning algorithm is identical; what a gigantic library brings is the infinite variety of particular contexts, the only thing that can separate one concept from another. A word has a well-defined identity only in so far as the situations in which it is used differ from those of its fellows. With little data, distinct concepts remain fused at the same point.

The great limitation: words with several faces

However ingenious this method is, it hides a fundamental weakness: it attaches to each word a single fixed vector, like a label stuck on once and for all. But what do we do with homonyms and words of many senses? The word “bank” can be the edge of a river or the institution that keeps our money. Not being allowed two distinct positions, its vector is pulled simultaneously in two opposite directions and ends in a bizarre compromise in the middle of the map — a point that fits neither of the two senses properly.

This obstacle has nothing to do with the quantity of text read but with the basic principle of the system: you cannot reduce a word to a static vector, cut off from the context of the moment. To understand living language, a term’s vector must be fluid, shaped dynamically by the sentence in which it lives at a given moment. This deep need will open the road towards the attention mechanisms and the great language models of today.

There is one more practical difficulty: we have assumed so far that the unit of work is the whole word. But however voluminous a dictionary may be, new words, proper names, typing errors, technical terms or unexpected inflections will keep appearing. In the next lesson we shall discover how artificial intelligence solves this dilemma, breaking words down into flexible Lego pieces.