One Space for Image and Text
How can a computer understand that the photograph of a puppy and the sentence “a puppy frisking on the grass” speak of the same thing? Two different neural networks — one for images and the other for words — are taught to place the visual world and language on the same map. When the ideas match, their points become close neighbours, letting us search for images by plain words and recognise new objects with no special training.
Terms: a common space multimodal contrastive learning recognition with no special training
Two languages, one dictionary
In the lessons on language we saw a remarkable idea: words can be turned into vectors — lists of numbers that work like coordinates on a vast geometric map of meaning. There, notions close in sense, such as “cat” and “tomcat”, naturally end up as neighbours. But why stop at text? Could we not place images on the same map as well?
This elegant bridge (the CLIP model) is built from two completely different arms working hand in hand. One arm looks at an image and sums it up in a vector of coordinates; the other arm reads a text and produces a vector of exactly the same length. Their mission is simple and natural: a photograph and its correct description must fall as close together as possible on the map, while unrelated images and texts are pushed apart. The geometric place where the two worlds meet is called a a common space, and a system able to combine sight and word in this way is called a multimodal.
model.
The great practical advantage is that we need no painstaking labelling done by hand. The internet is already full of hundreds of millions of photographs accompanied by captions, titles or natural descriptions — a raw material offered ready-made.
How closeness is learned
Think of two people speaking different languages: one thinks only in images, the other only in sentences. Instead of attempting a direct mechanical translation between them, both learn to express their ideas in a third “universal language of pure meanings” (the vectors of the common space). And there, seeing whether they say the same thing becomes a simple measurement of closeness. contrastive learningHow do the networks learn to make this fine match? The method is called
. At every step the model receives a group of examples — say 24 matched pairs of images and texts. The vectors of all the images and of all the descriptions are computed, forming a chessboard of 24 by 24 comparisons. The rule of the game is clear: on every row and every column the greatest match must be the true pair (on the diagonal), and all the other, wrong associations must be rejected. closer Nobody imposes from the start where on the map each concept should stand. All that is asked is that the matching pair should always be
than the mismatched variants. Out of this simple continuous comparison an organic order is born: we teach no rigid definitions; the model learns by differentiation — “this image resembles this description more than any other”.
The demonstration model on this page works with miniature images of ten by ten pixels in three shades, and with descriptions made of three clear properties: the geometric shape, the colour and the size.
Search by text
The map you see is a flat, two-dimensional projection of a richer mathematical space with sixteen dimensions. What matters is that the points (which stand for images) and the squares (which stand for written descriptions) gradually gravitate together and settle in the same quarters of meaning, although they were processed by completely distinct mechanisms.
Recognition with no special training
train it first recognition with no special training The same mechanism brings an astonishing benefit:
(often called “zero-shot”). Instead of building a separate detector specialised in one kind of object, we phrase all the possible answers as descriptive sentences, compute their positions on the map and check which of them lies nearest the vector of the mystery image. With no extra dedicated training, the system identifies the object at a glance.
The table below examines this performance over all 24 possible descriptions. For a genuine test, we divided the images into two camps: combinations of shape, colour and size present in training, and four fresh combinations the model has never met before.
The last columns show a subtle nuance of how these networks work. Every property taken separately (the shape, the colour or the size) is recognised remarkably well, even on the unseen combinations. What is sometimes lost is precisely the tight link between them.
This is not a fault of our simplified model but a well-known peculiarity of the whole family of multimodal architectures: the “bag of notions” effect. The model knows for certain that there is a circle in the image, that there is the colour yellow and a small size, but it sometimes finds it hard to decide whether it is the circle that is yellow and small, or another figure in the composition.