Multimodal fusion
A Socratic walk-through of multimodal fusion — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #How can one model read a photograph and a sentence with the same machinery?
A photograph is a grid of brightness values. A sentence is a string of symbols with no brightness at all. They have no unit in common, no shared axis, nothing you could sensibly add together. And yet one model takes both and answers a question about the picture.
The tempting assumption is that there must be two systems inside, one for each kind of input, meeting somewhere to compare notes. That is not what happens, and the actual arrangement is more interesting. So let me ask it directly: what would have to be true for a single mechanism to handle both?
Reasoning it through
REASONING #Work backwards from the mechanism. A transformer does not operate on words. It operates on a sequence of vectors — lists of numbers, all the same length. The very first thing that happens to a sentence is that each token is replaced by its embedding vector; from that moment on, no letters remain anywhere in the machine. Attention compares vectors with vectors and knows nothing of language.
So the question softens considerably. It is no longer "how can a model read pixels and words with one mechanism", but "can a photograph be turned into a sequence of vectors of the right length?" And that is an ordinary engineering question.
The answer, in the design that has become standard, is disarmingly plain. Cut the image into a grid of small square patches — the original vision transformer used sixteen-by-sixteen pixel patches — flatten each patch and multiply it by a learned matrix to produce one vector. A patch has now become a token. An image of many patches is a sequence, and the sequence looks, to everything downstream, exactly like a sentence.
Pause on that, because it is the whole idea. The photograph is not translated into words. It is converted into the format that a transformer consumes, which is not words either. Text was never the native input; the vector sequence was, and text had to be converted too. Both modalities pay the same entry fee.
But now a harder question. Suppose I hand you two sequences of vectors, one from a picture of a dog and one from the sentence "a dog on grass". Nothing so far makes those vectors related. Each encoder was free to use the space however it liked. Sameness of format is not sameness of meaning — so what makes the vectors comparable?
Only training. In the contrastive approach that CLIP used in 2021, an image encoder and a text encoder are trained together on paired images and captions with one objective: make a matched pair's vectors close and every mismatched pair's vectors far apart. Nothing tells either encoder what a dog is. The pressure to agree, applied across hundreds of millions of pairs, is what forces both to organise their space around whatever distinctions the captions actually track. In the arrangement used by today's vision-language chat models, the pressure is applied differently — a projection layer is trained to map the vision encoder's outputs into the same space the language model already uses for its token embeddings, so image vectors can simply be placed into the token sequence alongside text ones and the language model attends across both without modification.
And an honest wrinkle, because "shared space" oversells it slightly. Liang and colleagues showed in 2022 that in contrastively trained models the image and text embeddings do not actually intermingle: they sit in separate regions with a measurable gap between them, and the model works by relative geometry within that arrangement rather than by literal co-location. So the space is shared in the sense that distances across it are meaningful, not in the sense that a picture of a dog lands on top of the word.
The analogy
THE ANALOGY #Think of a currency exchange in an airport. Yen and pounds have nothing in common physically — different metal, different paper, different sizes. They become comparable only because both are converted into a common denomination, and the rate is not a property of either currency; it is set by a market that observed a great many trades between them.
an exchange rate is a single number applied uniformly, whereas the mapping from an image to the shared space is a learned function with millions of parameters — so a mismatch is not a bad rate to be corrected, but a whole region of the space where the training data never taught the two sides to agree.
Clarifying the model
THE MODEL #Three refinements.
First, the misconception to retire: the model is not converting the image into a description and then reading the description. If it were, everything not mentioned in the description would be lost, and you could not ask a follow-up question about a corner of the photo nobody captioned. The image tokens remain available throughout, and attention can go back to them for whatever the current question needs.
Second, "the same machinery" is true of the transformer and false of the front end. The patch encoder and the token embedder are different components doing different jobs. What is genuinely shared is everything after the entrance — the attention layers, which have no idea which of their inputs came from a lens.
Third, the limitation this exposes. Because alignment comes entirely from paired data, the model's cross-modal ability is bounded by what those pairs contained. Captions describe what people bother to say about images. Fine spatial relations, counts of objects, and text rendered inside a picture are all things captions frequently omit or state loosely — and those are, not coincidentally, among the known weak spots of these models. The failure is not in the fusion mechanism; it is in what the pairing pressure ever asked the space to encode.
A picture of it
THE PICTURE #How to readRead the two boxes on the left as separate front doors, each with its own way of turning its raw material into fixed-length vectors. The two arrows into the shared space are where the modalities stop differing — and note that only those arrows are trained to agree, which is why the gap listed inside the shared space matters. The final link shows the sequence handed to attention as one mixed run of vectors, with nothing marking which came from a lens.
What became clearer
WHAT CLEARED #There is no translation between a photograph and a sentence. Both are converted into the only thing a transformer ever consumed — a sequence of vectors — and the surprising part is not that a picture can be tokenised but that text always had to be. What makes the two comparable is neither format nor architecture but training on paired examples, which is why the model's cross-modal strength is exactly as good as the pairings it learned from, and no better.
Where to go next
ONWARD #- How the same conversion is done for audio and video, and why time introduces a problem images do not have.
- Why models often generate images through a separate system rather than emitting patch vectors directly.
- What the modality gap implies for retrieval that searches images with a text query.
Key terms
TERMS #| Term | What it means |
|---|---|
| Embedding | a fixed-length vector standing in for a token, patch, or other unit of input. |
| Patch embedding | turning a small square crop of an image into a single vector, the image equivalent of a token embedding. |
| Contrastive training | learning by pulling matched pairs together in the space and pushing mismatched pairs apart. |
| Modality gap | the observed separation between image and text regions of a jointly trained embedding space. |
Every term the collection defines is gathered in the glossary.