Learning from Examples
Think how you recognise a friend’s face, or how you ride a bicycle: you can do it impeccably, yet you cannot write a manual of exact steps for it. For such tasks without a recipe, the machine no longer receives rigid step-by-step instructions; it receives examples from which it discovers the patterns by itself. This is the great change of perspective on which all modern artificial intelligence rests.
Terms: tacit knowledge rule with thresholds machine learning labelled examples training family of rules
When the mind knows more than it can explain
In the first two parts of this course we saw how a classical computer works: we give it a meticulous list of instructions and it carries them out blindly at astonishing speed. But what are we to do when the task cannot be broken down into a clear recipe of steps?
If you see a friend in a crowded railway station, you recognise him in a fraction of a second. Just as naturally you tell a cat from a dog, recognise a letter scribbled on a bus ticket, or feel whether a sentence is funny. Yet if someone asked you to set down on paper a list of infallible mathematical rules — what exactly you measure at every millimetre, and in what order — you would find the task almost impossible.
It is not a want of attention or of skill. In 1966 the philosopher Michael Polanyi named this reality of our thinking tacit knowledge, summing it up in a memorable phrase: “We know more than we can tell”. For decades researchers tried to write thousands of “if… then…” rules by hand for such problems. The result was always the same: endless lists of complicated conditions that worked on the few examples they had in mind and collapsed at the first new case.
Let us try anyway: a mechanical rule
To see directly where the classical approach stumbles, let us try to write a manual recipe anyway. We want to teach the machine to tell the digit 7 from the digit 1 on the basis of two simple features measured on the drawing: how much ink gathers in the upper third of the image, and how wide the drawing is relative to its height.
Our handwritten rule sounds clean and reasonable: “We judge it to be a 7 if at least 40% of the ink lies in the upper third and if the width of the drawing is at least a quarter (26%) of its height.”
Where rigid logic breaks
Watch what happens as you experiment: if you draw a hurried 7, upright and with a short top bar, the ink in the upper third drops below the 40% threshold and the rule wrongly decides you drew something else. If you draw a stylised 1, with a long tail at the top and a wide horizontal foot, the rule is fooled and enthusiastically declares it a genuine 7.
We might be tempted to add a new saving condition: “if the middle line is slanted…”. But that new condition will spoil dozens of other cases that used to work. In the real world, the notion of “seven” has no single narrow geometric definition. It is a vast family of related shapes, full of variations that nobody can list in advance. A human concept is not a fixed dictionary definition but a supple web of family resemblances — and such families do not fit into a few rigid instructions.
The same rule, but with adjustable dials
The thresholds we chose — 40% ink at the top and 26% width — were not law but merely two numbers picked by eye. What if we turned them into adjustable dials? In the gallery below we have 40 drawings generated by the page (20 digits 1 and 20 digits 7). A red frame marks the examples the rule judges wrongly at the current setting.
If you press “Try every combination”, the machine does in an instant what would take a person hours: it works systematically through more than 15,000 possible pairs of thresholds and picks the one that makes the fewest mistakes. It all takes a fraction of a second and needs no philosophical theory about what the digit seven means.
Look carefully at the best result, though: even with the finest imaginable pair of thresholds, the rule still gets two or three examples wrong. The trouble was not that we had failed to find the ideal values, but that a rigid rule with only two straight thresholds simply has not the finesse to separate all the real shapes. And if you press “Other examples” a few times, you will notice that we sometimes hit by sheer luck on a set that separates perfectly — a temporary illusion we shall take apart later.
The great reversal of perspective
In traditional programming the flow has one direction only: the human thinks and writes the logical rules, the machine applies them to the data and produces answers. When the rules cannot be put into clear words or formulas, this approach is stuck from the first second.
What we can do instead is turn the question round: we offer the machine a mass of data together with the correct answers we expect, and let the machine discover automatically the rule that links them. The arrows change direction: what was output becomes input, and what was input becomes output. The computer stays the same — circuits, transistors, logic switches — but the question we put to it is entirely new.
ordinary programming
learning from examples
Programs working in this second direction make up the field called machine learning (or machine learning). Data accompanied by the correct answers are called labelled examples, and the process by which the machine calibrates its dials to find the best pattern is called training.
What does it really mean that the machine “finds the rule”?
The phrase “the machine finds the rule” is generous and may create the illusion that the algorithm invents concepts out of nothing, by some magical spark. In reality things are far clearer and more disciplined: the machine invents no new shapes.
It is the human who chooses in advance a family of rules of possible rules — “a rule based on two thresholds that we can adjust”, for instance. That family comes with a set of dials (knobs or parameters). What the machine does is explore those dials and stop them in the position that makes the fewest mistakes on the examples offered. We are automating the search, then, not the invention. The choice of family remains entirely the human’s responsibility: if the family chosen is too simple or ill-suited, not even the cleverest search algorithm will be able to remove the errors.
From this follow the two great questions that will guide our whole exploration: Which families of rules are rich and flexible enough to hold complex concepts from real life, such as a person’s face or the meaning of a word? And How do we search among them when the dials are no longer two but hundreds of billions, and trying every combination in turn would take billions of years? In the next lesson we take the simplest and friendliest family with two dials — a straight line — in order to understand how we measure the error and how we shrink it step by step.