Convolution

The explosion of connections: why looking globally fails

A modest black-and-white photograph of only 100 by 100 pixels contains ten thousand points of light. If we tried to analyse it with the classic network of the previous lesson, a single neuron looking at all the pixels would need ten thousand weights of its own. And a layer with just a thousand such neurons would instantly swallow ten million dials!

This waste creates an enormous problem: every dial has to be calibrated from examples, which would call for millions of training photographs. But worst of all, independent neurons are blind to spatial relationships: if the network learns what a cat’s ear looks like in the top-left corner, it will have no idea how to recognise it if the cat moves to the bottom-right corner! It would have to relearn the same concept thousands of times, once for every little corner of the screen.

The saving idea: the travelling stamp

Fortunately, human sight and the visual world obey a golden rule: the meaning of an object does not depend on where it happens to be. A vertical line is a vertical line wherever it is drawn, and a car wheel remains a wheel whether it is on the left or on the right of the frame.

We can write this intuition directly into the structure of the network! Instead of allotting huge neurons to every region, we take a minute group of weights — a grid of only $3 \times 3$ numbers, say — and walk it step by step across the whole image, like a magnifying glass or a stamp. This travelling stencil is called a filter (or kernel), and the mathematical operation of sliding it over the pixels is called convolution.

The gain is astounding: the number of dials falls from tens of thousands to just 9 or 25! And if the filter learns to detect a vertical stripe, it will recognise it instantly over the whole surface of the image, because it uses the same numbers everywhere. We have turned an intuition about the physical world into an enormous saving in computation.

Each value in the map on the right says how well the filter matches the piece of image at that place. The light regions are the places where the pattern the filter is looking for actually is. The map is called an activation map and is itself an image, which can be passed on further.

The numbers in the filter can be changed directly, and the map is remade at every keystroke. In a trained network they are not written by hand: they are found by gradient descent, like any other weight.

The hierarchy of sight: from edges to corners and objects

The result of applying a filter over a photograph is a new image, called an activation map. The bright regions of this map show the places where the filter fitted the details of the picture like a glove.

But here is the great secret: because the activation map is itself an image, we can walk a new generation of filters over it! The second layer no longer looks at the raw pixels but at the answers of the first layer, learning to combine simple patterns into more complex ones. For instance, if the first layer found vertical lines and horizontal lines separately, the second layer can detect their intersection: the corner!

The third panel was drawn by nobody: it comes out of the combination of the first two by an ordinary neuron, which adds the two inputs and requires the sum to pass a threshold.

In a deep convolutional network this pyramid rises through dozens of levels: the first layers see only lines and lights; the middle layers assemble corners and circles into textures and shapes; and the top layers recognise headlights, wheels, tails or a dog’s eyes. And all these specialised filters are not programmed by hand: gradient descent sculpts them out of examples by itself!

Spatial summary: downsampling

Once a filter has detected that an eye or a wheel is somewhere in a given region, the exact millimetric position of every pixel is no longer so important. Far more valuable is the fact that the feature is there!

To compress the information, engineers introduce an operation called downsampling (or pooling): they divide the map into little squares of $2 \times 2$ pixels and keep from each square only the largest value, throwing the rest away. The size of the image thus falls fourfold, making the following computations much faster and giving the network a natural tolerance: if the cat takes a small step to one side, the compressed map stays almost unchanged.

This compacting must be dosed with measure. If we throw away too much of the original geometry, the network might confuse an ordinary human face with a surrealist painting in which the eyes sit below the chin and the mouth on the forehead. That is why modern models use downsampling more prudently than in the past.

Architecture: a theory about the world cast into equations

Convolution is not a universal remedy for every kind of problem but an intelligent restriction, created expressly for data with a visual organisation. It assumes firmly that neighbouring pixels are related and that patterns have the same meaning wherever they appear.

If you tried to apply convolution to a population register — where one column gives the age, another the salary and another the town of residence — the method would be a total failure: walking a stamp over data with no spatial continuity is meaningless.

This is the great philosophical lesson of the chapter: the choice of a network’s the architecture is the subtle way in which the engineer pours his own understanding of the world into the model. We do not write rigid rules by hand, but we build a stage on which the data can learn only what the structure of the world permits. Architecture is a theory about reality encoded in silence.

The turning point: from images to words

The first successful convolutional networks were created by the French researcher Yann LeCun at the end of the 1980s, for the automatic reading of the digits on American bank cheques. But the global explosion came in 2012: trained on modern graphics cards, a deep convolutional network shattered every record at a great world competition in image recognition (ImageNet), setting off the contemporary revolution in artificial intelligence.

Convolution conquered machine vision because images have a fixed, two-dimensional geometry. But what do we do when it comes to human language? Text is not a picture: words flow in time, sentences have different lengths, and the meaning of a word may depend on something said ten pages earlier.

In the coming chapters we leave the world of pixels and enter the fascinating universe of language: how do we turn a word into a computable mathematical entity, and how do we make a machine make out meaning?