Layers

A team of neurons: the birth of the first layer

Nothing obliges us to use a single neuron to analyse the input data. We can set a whole team of neurons side by side, each looking at the same measurements but weighing them with its own set of weights. Every member of the team will draw its own line of demarcation in space and offer a nuanced answer between 0 and 1.

Such a group of neurons processing the same data simultaneously forms a layer. Their outputs, taken together, represent a completely new description of the example: in place of the raw measurements we started from, we obtain a list of answers to intermediate questions put by the neurons. It is a historic moment in this course: for the first time, the model no longer works only with features chosen by a human, but builds its own internal representations!

The answers of this first layer can be sent on as input data for another neuron. The intermediate layers — those that neither receive the raw data directly from outside nor produce the final visible answer — are called hidden layers (or hidden layers).

The mystery of XOR, solved in three simple steps

The problem that blocked research in 1969 — the impossibility of separating the four points on the diagonals — is solved spectacularly by a small team of just three neurons: two in the hidden layer and one at the output.

The first hidden neuron draws a line and asks: “Is at least one of the inputs active?”. The second neuron draws another, parallel line and asks: “Are both inputs active?”. The third neuron, the one at the output, listens only to the voices of its two colleagues and gives the final verdict: “Say YES only if the first said YES but the second said NO!”.

Look at the third panel: the resulting boundary is no longer a rigid straight line but a band cut out with real craft! We have overcome the limitation without inventing any complicated magic: we used the same straight cuts, but combined them logically in stages.

Why the S-curve is the soul of the network

Here a fascinating mathematical secret is hidden, one that explains why neural networks work and why an apparently minor detail makes all the difference.

Imagine what would happen if we removed the S-curve (the logistic function) and let the neurons do only simple additions and multiplications. The output of the first layer would be a sum of numbers. The second layer would add those sums in turn. But in algebra, a sum of sums is still just a simple linear sum! However many hundreds of layers we stacked one on another, the whole vast network would collapse mathematically into a single equivalent neuron, and the boundary would become a flat straight line again. Without a non-linear bend between the stages, the depth of the network is a pure optical illusion!

The smooth S-curve — called in general the activation function — is precisely the hinge that bends space at every step and prevents this collapse. Press the button “Remove the S-curve” in the next figure and you will see it with your own eyes: however many neurons you add, the network becomes powerless and behaves like a simple ruler.

Sculpting complex shapes: the enchanted ring

In the interactive figure below we have a spectacular challenge: a circle of blue points surrounded by a sea of green points. No straight line and no simple band can isolate the circle in the middle.

Look at what happens as you raise the number of hidden neurons: with a single neuron, the boundary is a failed straight line. With 3 or 4 neurons, the boundary becomes a triangle or a rhombus — each neuron adding a straight side that “chisels” the surrounding space. When you reach 8 or 12 neurons, the polygon acquires so many facets that it becomes an almost perfect circle, embracing the blue points impeccably!

But what if you now press “Remove the S-curve”? All twelve neurons suddenly lose their power: their lines gather back into a single one, getting dozens of examples wrong again. Put the S-curve back, and the round shape reappears instantly. Non-linearity is the network’s true magic pair of scissors.

From width to depth: deep networks

Mathematicians have proved a spectacular result called the universal approximation theorem: even a single hidden layer, if it is given a large enough number of neurons, can approximate any continuous shape in the world to any desired precision. It would seem that the problem is solved for ever; but in practice there are two enormous traps.

First, the word “enough” may mean billions of neurons set side by side — a vast waste of memory and computation. Second, the theorem guarantees only that those ideal weights exist, without offering us any certainty that the gradient-descent algorithm will ever manage to find them in a reasonable time.

This is why the revolution in artificial intelligence did not consist in creating very “wide” networks, but deep networks (deep learning). When we set the layers one after another in a cascade, the network naturally learns a hierarchy of concepts: the first layers detect small edges and contours; the next combine them into textures and corners; the higher layers assemble those shapes into eyes, noses or wheels; and the last layer recognises a whole person or a whole car. The same expressive power is obtained with far fewer neurons if we arrange them in depth!

In modern deep networks the classic S-curve has often been replaced by an even simpler function called ReLU: it leaves positive numbers unchanged and cuts everything negative down to zero. Although it has an abrupt corner, its slope does not die away at large values, which allows networks with hundreds of layers to be trained smoothly.

But how do we train a network with dozens of layers and billions of dials? How do we find the slope of every dial buried deep in the model’s innards without recomputing everything from scratch? In the next lesson we discover the most elegant engine of artificial intelligence: the algorithm of backpropagation.