What Can Be Seen Inside
Seen from outside, a neural network looks like an impenetrable black box: an ocean of billions of anonymous decimal numbers. Can we look inside to see whether the model forms real representations of the world — clear concepts that we too can understand? The field of mechanistic interpretability shows that we can. To examine the silicon brain without illusions, science uses a rigorous method in three steps: a reading with a probe, a control group, and a direct surgical intervention.
Terms: activations a probe direction control intervention
The task and the question
The interactive model on this page receives two two-digit numbers and one mathematical operation, and produces the digits of the result. Between the input data and the final answer lie two hidden neural layers, each made of 72 neurons. When the network processes a calculation, the instantaneous numerical values passing through these units are called activations.
These activations are just 72 decimal numbers with no descriptive label. The fascinating question is: has the network discovered human logical concepts by itself — such as “the carry from units to tens” or the tens digit of the result? To find out, researchers introduce a a probe (probe): a small external detector, entirely separate from the model, trained to guess the concept sought by analysing only the internal activations, without touching or changing the model.
The probe we use is deliberately minimal (a linear probe). It has no capacity for complicated reasoning of its own; it can only multiply each of the 72 numbers by a weight and add them up. If so simple a detector manages to guess the concept perfectly, the conclusion is unambiguous: the information was already clearly organised in the innards of the network!
What can be read from the activations
Look at the interactive table: the middle columns are the control group — the indispensable scientific the control , without which the figures would have no value. The first control column measures what can be read directly from the raw input numbers; the second shows what the probe finds in an identical but entirely untrained network, left with random values.
The presence of the controls lets us distinguish three fundamental situations. When all the values are high, the concept was already obvious in the raw input data, so the probe demonstrates no skill of the network. When the last column (the trained model) spectacularly exceeds both controls, we have proof that the model computed internally a new abstraction, absent at the input. And when the final score falls below the controls, it means the network gave up that detail in order to concentrate on other priorities.
A fall in the score does not necessarily mean the information has vanished without trace; it means only that it no longer lies on the surface in a simple linear form. Every probe measures the world through one particular pair of spectacles.
One unit or one direction
If a concept is certainly in the network, where exactly does it live? Our first naive intuition would be to look for the “dedicated neuron” — a single biological or artificial unit that lights up like a lamp whenever that concept appears (the famous theory of the “grandmother neuron”).
The graph above categorically demolishes this illusion. For the concepts the network actively computes, no individual neuron comes near the precision of the whole layer. The concept does not live in an isolated cell but in a distributed direction — a collective harmony, a fine pattern in which dozens of units vibrate together with different intensities and signs.
This discovery explains the phenomenon of superposition: a space of 72 dimensions can house thousands of different concepts at once in the form of quasi-orthogonal directions, which interweave harmoniously without destroying one another. This is why the search for a single neuron for every notion is a dead end, while the search for vector directions brings to light the true geometry of artificial thought.
Read does not mean used
Beware, however, of a logical trap: the fact that a probe can read a feature does not automatically mean the model uses it to take decisions! It might be a mere coincidence, a passive shadow or a by-product of computation with no relevance.
To move from correlation to causation, a surgical intervention is needed (ablation). We take the geometric direction identified by the probe and simply remove it from the internal activations — we “anaesthetise” that component without touching the rest of the network — and then measure the impact on solving the task.
Look at the control row with the random direction: any crude intervention produces a small disturbance, but the great question is whether the targeted direction causes a significantly more severe fall. If removing the direction collapses the model’s accuracy, we are certain the model was actively relying on it. If the impact is identical to that of a random disturbance, it means the feature was merely written down passively, with no functional role.
This is exactly the leading methodology used today to probe frontier language models: identifying meaningful directions, validating them rigorously on control samples, and intervening actively (erasing or amplifying) to steer the network’s behaviour from within.
How far one can go
The principle demonstrated here on a teaching model of 72 units is applied directly to the great language systems with hundreds of billions of parameters, but on a dizzying scale.
At that scale, researchers cannot guess in advance, by hand, all the millions of possible concepts. Special helper networks are used (sparse autoencoders) which automatically decompose the internal activity into a gigantic dictionary of clean directions. In these maps fascinating directions have been discovered: neural representations for the Eiffel Tower, for advanced scientific concepts, for a joking tone or even for the intention to mislead.
What stays unchanged and lasting is the rigour of the scientific method: a faithful reading through probes, a control, and a direct causal intervention. Without a control we risk deceiving ourselves; without intervention we cannot know whether what we see really matters. It is the way we make sense of any mystery we cannot question directly in words.
With this look inside, our journey through the fundamental mechanisms of modern artificial intelligence ends. In the last part of the course we shall add no new pieces to the puzzle. We shall use everything we have learned to reflect on three great philosophical and practical questions: does a machine that merely predicts words build a deep model of the world? what does it really mean for an algorithm to be “intelligent”? and what unsuspected risks appear when a whole society optimises its aims by artificial indicators?