The Artificial Neuron

A committee with a weighted vote

The contorted curves of the previous lesson tried to solve everything in a single dimension and broke down quickly. In reality the world is not one-dimensional: to recognise a digit, a face or an animal, we have dozens or thousands of different clues at our disposal.

About a drawing we can measure the density of the ink in various regions, the total width, the height, the number of closed loops or the symmetry. No single clue is enough on its own, but together they paint a clear picture. How can we make them work together? The most natural way is a weighted vote: each measured quantity is multiplied by an importance of its own, and the results are added into a total sum.

These multipliers of importance are called weights. A large positive weight means that the clue argues strongly for the answer “YES”. A negative weight means that the clue argues against (“NO”). A weight close to zero means that the feature is ignored. On top of all this we add a fixed number called the offset (or bias), which sets the neuron’s baseline level of scepticism: how demanding it is before giving an affirmative answer.

The frontier that cuts space in two

The resulting weighted sum is a single number. To take a clear-cut decision (“it is a 7” or “it is not a 7”), we compare that sum with zero: if the result passes zero, the answer is YES; if it stays below zero, the answer is NO.

If we imagine all the examples laid out on a flat surface according to their two measurements, the line of demarcation where the sum is exactly zero forms a straight line. That line is called the decision boundary. The weights decide the tilt of the line, and the offset moves it nearer to or further from the origin.

Notice the subtle difference from the rule in lesson 11: there we cut space with two rigid thresholds parallel to the axes, forming a right-angled corner. An artificial neuron can draw a line tilted at any angle at all! It can therefore balance the clues: if a drawing has a little less ink above but is much narrower, the neuron can elegantly let the two compensate each other. And the greatest advantage is that this formula works identically whether we have two, twenty or ten thousand input measurements.

From the rigid step to the supple slope

A rigid all-or-nothing decision (0 or 1, like an abrupt step) has a fatal defect: it cannot be trained by gradient descent! On the step, the ground is perfectly flat to the left and to the right, and in the middle there is a vertical chasm. The slope is either zero or infinite — it offers the traveller no clue which way to turn the dials.

The ingenious solution is to replace the step with a smooth S-shaped curve called the logistic function (or sigmoid). The neuron’s output ceases to be a mechanical “yes/no” and becomes a subtle degree of confidence between 0 and 1: a value of 0.98 means “very probably yes”, 0.10 means “very unlikely”, and 0.50 means “completely unsure”.

This nuance changes everything: now every minute change in a weight moves the degree of confidence smoothly, offering a continuous slope that the descent can feel under its feet. What is more, as the neuron becomes very sure about an example, the slope slackens, automatically concentrating its learning effort exactly on the unclear examples at the border.

The training button in the previous figure runs gradient descent exactly as in the thirteenth lesson, on the neuron’s weights. The two measured quantities are brought to the same scale beforehand, for the reason shown there.

The descent comes to rest at three mistakes, although there is a line that makes two. Prolonging the training changes nothing. This is not a weakness of the method but the consequence of what was asked of it: it lowers the cost smoothly, not the number of mistakes. The position that gives the least cost is not the same as the position that gives the fewest mistakes — the difference noted at the end of the lesson on the cost function, here with figures.

The impassable wall facing a single neuron

Because the frontier of a single neuron is always a straight cut, its power stops before any pattern that cannot be separated by a line. And the most famous example has only four points!

Look at the figure above: two blue points on one diagonal and two green points on the other. However you turn the ruler and wherever you place it, it is impossible to separate the colours with a single straight line! This configuration is the famous logical operation exclusive-or (XOR), which we met in lesson 6 in the addition of bits.

In 1969 the researchers Marvin Minsky and Seymour Papert proved mathematically that a single neuron cannot solve even the XOR problem. Their book set off an AI winter: many drew the mistaken conclusion that neural networks were a dead end. In reality the proof showed only the limits of a solitary neuron — not those of a team of neurons organised in layers!

Where the name comes from: biology versus mathematics

The model was proposed in 1943 by Warren McCulloch and Walter Pitts as a simplified mathematical sketch of the nerve cells in the brain. In 1958 Frank Rosenblatt added to it the ability to adjust its weights from examples and christened it the perceptron.

It is important to dispel any confusion: the artificial neuron is only an inspiring metaphor, not a faithful copy of the biological brain. The neurons in our heads communicate through pulsing electro-chemical impulses, depend on fine temporal rhythms and learn through processes far richer than the mathematical descent of a numerical slope. To say that artificial networks “think like a brain” is poetry, not a technical description.

But here is the beauty of it: a single neuron draws a straight line. What happens, though, if we set several neurons to draw each its own line, and then give their results to a new generation of neurons? In the next lesson we discover how to build layers and how a simple combination of straight lines can learn any shape in the universe.