The Cost Function
For a computer to search automatically for the best rule, the vague notion of “error” must be turned into a clear, precise number. Choosing that number looks at first like a mere technical detail; in reality it is the very place where we tell the machine what we value and how we see the world.
Terms: parameters model prediction cost function the cost landscape
The friendliest family of rules: the straight line
The last lesson left us with two great curiosities: what shape do we choose for our rule, and how do we help the machine find it without groping in the dark? To understand the mechanism in all its simple splendour, we shall choose the most intuitive family of rules in the world: a straight line drawn among points.
Imagine noting in a diary how long several car journeys took: along the horizontal you put the distance in kilometres, and up the vertical the time in minutes. Naturally, the further you go, the longer the journey takes. The points seem to line up along a straight line, even if there are small departures because of traffic lights or the weather.
A straight line has only two “dials” we can turn: the slope (how many minutes each kilometre travelled adds) and the starting height (how many minutes you lose before setting off, at the lights or leaving the car park). In the language of machine learning, these two adjustable dials are called parameters. The rule fitted with its parameters forms a model, and the time our line estimates for a given distance is a prediction. A model is, in essence, a hypothesis about the world that leaves a few blanks for the data to fill in.
The points in the figure are generated from a real relation settled in advance, over which small delays and fluctuations from real life have been added. The true relation stays hidden until you find the best fit.
How we measure the error: the trap of pluses and minuses
If we lay some line among the points, for every journey measured there will be a mismatch: the difference between the time the line predicts and the real time on the clock. Some real points lie above the line (we took longer than predicted), others below (we arrived sooner).
If we simply added those differences, something dangerous would happen: a plus of 10 minutes would cancel a minus of 10 minutes. A completely crooked and useless line could have a total error of zero. The first obligatory decision, therefore, is to turn every error into a positive value.
The commonest method is squaring every error. It is not merely a calculating trick but a choice with three deep effects: first, any departure becomes a positive number; second, large departures are punished far more severely than small ones (an error of 10 minutes weighs a hundred times more than one of a single minute); and third, the resulting number varies extremely smoothly and continuously as we move the dials.
The average of all these squared errors gives us a single final number, measuring how “dissatisfied” the line is over all the data at once. That score is called the cost function (or loss function). From now on the whole task of machine learning becomes a clear mission: to move the model’s dials until this cost is as small as it can be.
The landscape of the cost: a valley ringed by hills
In the right-hand panel of the figure we can view the same cost in another light — as a genuine relief map. Every point on this map stands for a unique combination of the two dials. The colour shows you how large the error is: light shades mark the regions of small cost (an excellent fit), and dark ones the steep slopes with large errors. The curved lines are contour lines, exactly as on a walker’s map of a mountain.
The yellow point is you at this moment; you can press anywhere on the map to move the dials instantly to that spot. This picture of the the cost landscape (or loss landscape) is the compass of the whole course: to learn is to descend towards the bottom of the valley.
For a straight line and squared errors, this landscape has a dream shape: it is like a perfectly smooth bowl, with a single deep valley and no traps at all. Whatever point you start from, the descent will unfailingly bring you to the same ideal place. When the models grow more complicated, the landscape will acquire thousands of valleys and ridges, and the descent will become a real adventure.
What the cost chooses for you: the trap of accidents
Press the button “Add a freak measurement”. That button simulates a most unusual day: a huge traffic jam on the motorway, or an error in noting the mileage. Watch what your orange line does: it is pulled strongly upwards by that isolated point, spoiling the fit on all the other normal journeys.
Why does that happen? Precisely because of the squaring: a huge distance to the freak point generates an astronomical cost, and the model is forced to sacrifice the accuracy of all the well-behaved points in order to appease that one colossal error.
Now look at the olive-green line in the graph: it was computed using only the plain departures, without squaring them (the direct distance). That line barely flinches when the extreme point appears. Which of them is right? Neither absolutely: the first assumes that any large departure signals something crucial that must not be ignored; the second assumes that huge departures are mere anomalies to be overlooked.
This is a fundamental lesson: the cost function is not a neutral tool but the place where our view of what matters lies hidden. The algorithm will optimise with blind loyalty exactly the measure we give it. If we choose the wrong formula, the computer will learn impeccably… exactly the wrong thing.
What qualities must a friendly cost have?
To guide machine learning, a good cost function must meet three friendly conditions: it must be easy to compute for any position of the dials, it must fall faithfully as the predictions come closer to reality and — most important — it must change smoothly, with no jolts or sudden jumps, when we turn a dial by a very little.
That last requirement explains why the simple count of wrong examples (which we used in lesson 11) cannot help us here. If you merely count how many drawings are wrong, that number is a rigid integer: 3, 4, 5. When you move a dial by a millimetre, the number of mistakes often stays unchanged, so you cannot tell whether you have come closer to the solution or moved away. It is like trying to descend a mountain made only of huge steps and perfectly flat terraces: on the terrace you cannot feel which way the slope goes.
That is why, in artificial intelligence, the final goal we want (“zero mistakes”, say) and the cost we minimise during training are two different things. We look for a supple, continuous function that gives us a slope at every step. But how do we descend that slope when we have not two dials easy to draw on a screen but millions of parameters hidden in the fog? That is the secret we shall unravel in the next lesson.