Search Before the Answer
The most powerful tool of a modern model is not a mathematical calculator but a well-ordered library. Instead of asking the network to memorise all the books in the world by heart in its internal parameters, we let it sit an examination with the textbook on the desk: first we search out a few relevant fragments from an external database, lay them before the model, and it answers by reading and synthesising the fresh information. Knowledge thus moves from the rigid memory of the weights straight into the living context of the conversation.
Terms: retrieval a vector database matching by words the right fragment
Why we search first
A classical language model is the prisoner of the moment its training ended: it knows only what it managed to read up to then, in the form of diffuse associations imprinted in billions of weights. It does not know yesterday’s news, cannot quote paragraphs of a private contract precisely, and cannot guarantee the exact source of a statement.
All three of these great shortcomings find their remedy in an ingenious change of strategy: we move the factual knowledge out of the model’s internal memory and straight onto its working sheet (into the context). When the user puts a question, a dedicated system searches an archive of documents, cuts out the most relevant passages and serves them to the model alongside the question. This technique is called generation augmented by retrieval (known for short as RAG — Retrieval-Augmented Generation).
It is the difference between heaven and earth: a traditional examination based purely on cramming, and an examination with the books open on the desk. Still, the open textbook brings a risk of its own: if you do not find the right page, or if you leaf by mistake into the wrong chapter, the answer will be compromised with the same apparent confidence.
How closeness is measured
The demonstration collection on this page holds 36 explanatory fragments from the lessons so far. The aim of the search is that, on receiving a question, the system should automatically find exactly the fragment containing the clarification wanted.
But how do we decide which text matches the question? There are two great philosophies. The first approach — classical lexical search — counts the identical words appearing in both texts, giving more weight to rare, specific words (such as “transistor” or “attention”) than to common connecting words. The second approach — semantic search — projects every sentence into a geometric space of meaning, turning the text into a vector of coordinates and measuring the angle between their directions. A database organised on this geometric principle is called a vector database.
The truly relevant fragment is marked with an orange frame, whatever place it took in the ranking. The number on the right measures the mathematical score of the match, on a scale from 0 to 1.
The context given to the model
Once the search stage is over, the final assembly is simple and transparent. The winning fragments are concatenated in order into a compact block of reference, at the end of which the user’s question is added. This whole assembly is handed to the model as a single continuous text.
When the words of the question change
The real test of maturity for a RAG system is not the elegance with which it glues the texts together but the precision of the retrieval: how often does it manage to catch the essential fragment among the first results returned?
The experiment below runs this check in two different scenarios. First we use questions phrased directly, with the exact technical terms; then we test the same questions rephrased through synonyms or roundabout expressions, deliberately avoiding the obvious keywords.
When the question uses exactly the terms from the dictionary, simple word-matching works perfectly and brings the correct fragment to first place. But as soon as a person uses a metaphor or an unexpected synonym, strictly lexical search becomes powerless.
Here the role of advanced vector databases comes to the fore. In commercial production systems the vectors are not computed by simple local averages but generated by neural models trained specially for search on hundreds of millions of real queries. They understand that “running out of memory” and “exhausting the RAM” speak of the same phenomenon, even though they share no identical word.
What it solves and what it does not
The RAG technique offers enormous practical advantages: it brings the very latest information or private confidential data into the discussion, allows sources to be quoted exactly, and anchors the answer in verifiable documents instead of depending on the model’s vague memory.
But it is crucial to remember what this method cannot solve: it does not cancel the risk of confabulation examined earlier. The model runs through the fragments it receives exactly as it would through any other string of tokens, and it has no infallible filter for telling a truly relevant fragment from one that merely looks apt on the surface. If mistaken or misleading information slips into the context, the network will synthesise it in the same convincing, self-assured tone. An open textbook on the desk is of little use to a pupil who takes as gospel any rumour written on a stray page.
For this reason we cannot solve the problem by stuffing dozens of pages into the context: every fragment added raises the risk of noise and confusion (besides the substantial cost in memory and computation explored earlier).
So far we have examined how performance can be improved by adding tools and archives around the model. In the next lesson we shall lift the bonnet and look inside: how the gigantic neural machine can be compressed and optimised — through quantisation, distillation and mixtures of experts — so that it runs fast and cheaply without losing its skill.