The Neural Language Model

The trap of the infinite: how many combinations really exist?

The model of the previous lesson rested on a rigid piece of bookkeeping: it kept a huge ledger with every group of words met and counted what followed it. Before we go further, it is worth making a simple common-sense calculation: how large would this register have to be to cover a living language? The answer is overwhelming and makes clear at a glance why the old approach collapsed.

Let us imagine a modest vocabulary of only ten thousand everyday words. The number of possible combinations of just two words is ten thousand multiplied by ten thousand, that is, a hundred million. If we want to take three words into account, we multiply by ten thousand again and arrive at a thousand billion! With every word added to the context, the space of possibilities explodes exponentially into an astronomical abyss.

Even if we scanned a mountain of a thousand billion words — everything mankind has printed since Gutenberg — a text contains a finite number of successive windows. Against the astronomical multitude of combinations the vocabulary allows, our table stays almost entirely empty. This is not a want of diligence on the programmers’ part but a physical limit of the universe: enough text to fill every possible box has never been written and never will be.

The figure zero does not mean impossible

This desert of empty boxes would be no tragedy if the absent sentences were wrong or meaningless. On the contrary! The great majority of perfectly correct, beautiful and logical sentences in English have never been spoken or written by anyone. Think for a moment: the sentence you are reading on this screen right now is, very probably, unique in the history of universal writing.

A blind statistical table cannot make this fine distinction. To it, an absurd sequence such as “the shoes the triangle fly” and a new, poetic metaphor or a natural sentence never met before all receive exactly the same dry verdict: zero. The mathematical smoothing trick of the previous lesson merely shares out a minute and equal crumb among all the unseen variants — an honest admission of ignorance, but one that helps not at all in real conversation.

What a table utterly lacks is the intuition of analogy: the power to say “I have never heard this particular sentence, but it is strikingly like another I know well, and their words are close relations”. To make this leap of thought, words must have a geometry of kinship — something a static table can never offer.

Likeness put to work: the neural network

In a rigid catalogue, the terms “cat” and “dog” are two isolated labels, entirely foreign to each other, as far apart as “cat” and “Wednesday”. Any new observation about a dog brings not a grain of knowledge about a cat.

Here the brilliant idea of the vector map from Lesson 19 comes in. If every word is represented by a bundle of coordinates (a vector), and terms that share similar contexts end up close neighbours in space, then any lesson accumulated for one point radiates of itself over its whole semantic quarter.

This was the revolutionary architecture proposed by Professor Yoshua Bengio in 2003: the words of the sentence are turned into their feature vectors, these are set side by side and sent through a neural network with hidden layers, and at the output the network estimates the chances of every word in the vocabulary. The vectors are not drawn by hand: they are adjusted continuously by gradient descent, learning at the same time as the rest of the network. It is the direct ancestor of all the great language models of today.

The red boxes in the experiment above represent associations deliberately deleted from the reading data. Faced with them, the statistical table stalls and gives zero: it has never met them, so it shrugs. The neural network, by contrast, gives a positive probability full of meaning! Why? Because it noticed that “cat” behaves like “dog”, and about the dog it saw that it sleeps. Without anyone having taught it biology, the model inferred by transfer that cats can sleep too.

Why it works: the pooling of knowledge

The great strength of this approach lies not in the volume of memory but in the fact that the network uses shared parameters for related situations.

In an isolated table, every row is a lonely island: if you learn something about “dog”, the knowledge stays locked up there. In a neural model, any sentence about a puppy shifts its vector slightly on the map, and this shift automatically alters the computations for all the neighbouring terms (wolf, fox, cat). Every example read lights up and enriches the neighbouring words as well, even those not mentioned directly in that sentence!

The effect is salutary above all for rare words. The statistical table abandons them with null scores. The neural model, by contrast, catches the rare term after its two or three appearances, anchors it beside its more popular synonyms, and from then on the new concept inherits on the spot the whole treasury of its neighbours’ knowledge. At bottom, to think and to generalise means to borrow with discernment: to use what you know from a lived experience in order to navigate a new but similar situation.

The obstacle that remains: the prison of the fixed window

Bengio’s model broke the curse of the empty tables, but it soon ran into another difficult engineering barrier.

The context window was still rigid and narrow. If the network is designed to receive exactly three or four words at its input, it is completely blind to everything that happened earlier. Look at a sentence with agreement at a distance: “The girls whom I met yesterday at the library, come from the other quarter, ___”. The word that decides the agreement of the verb is “girls”, ten positions back! For a fixed window of four words, the subject vanished beyond the horizon long ago.

Why can we not simply widen the window to fifty or a hundred words? Because every new position in the window adds a whole set of connections to the first neural layer, swelling the number of parameters exponentially. What is more, the connections for position 1 are entirely separate from those for position 3: what the network learns about a noun at the start of the window does not apply automatically if the same noun appears two positions further to the right!

It is exactly the same waste we met in the analysis of images before convolution appeared. We need a flexible mechanism, able to read sentences of any length and to apply the same principles whatever the position of the word in the text. In the next lesson we shall explore the first great attempt to give networks a continuous memory: the recurrent network, an ingenious idea that was nevertheless to run into a barrier of forgetting all its own.