Multimodal Frontier Models

It is April 2022. Language models can write fluent prose from a text prompt, but they are blind — GPT-3 has never processed a pixel, and a decade of image-captioning research still leans on task-specific pipelines built for one benchmark at a time. Over the next twenty months, a DeepMind team, OpenAI, a two-person academic project, and a thousand-person Google effort each answer the same question — how do you give a language model eyes — and by the end of 2023 the answers converge on a single model that reads, writes, and sees without task-specific glue.

Where We Were

By early 2022 two separate lines of work had solved two separate halves of the problem, and neither had solved both. In-context few-shot learning — showing a language model a handful of examples in its prompt and letting it infer the task, introduced with GPT-3 in May 2020 — had made task-specific fine-tuning optional for text. But GPT-3 was text-only: it had no mechanism for taking an image as part of its input at all.

The vision side had its own advance. CLIP (CLIP, bridging text and image), released in January 2021, trained a two-tower model to align images and text captions with a contrastive objective, and the resulting embeddings transferred to zero-shot image classification with no labeled examples. That was a genuine breakthrough in representation — an image and its caption end up near each other in a shared space — but CLIP has no generative half. It can score how well a caption matches an image; it cannot answer an open-ended question about the image, describe what is happening in it, or hold a conversation about it. Text-to-image generation, exemplified by DALL-E in February 2021 (owned by image-generation/12), ran the pairing the other direction — text in, pixels out — which is a different problem again from reading an image and writing text about it.

Before either of these, visual question answering had existed for years as its own subfield: object detectors extracted region features, a fusion module combined them with a question embedding, and a classifier picked an answer from a fixed vocabulary. Every new dataset needed a new labeled training run and a new task head. There was no version of GPT-3’s trick — paste a few examples in a prompt and generalize — for anything involving pixels. That was the gap standing open in April 2022: a model that could look at an image or a video, read arbitrary text about it, and produce open-ended answers, ideally with the same few-shot flexibility that had just made text-only in-context learning work.

Text got few-shot learning and vision got aligned representations, but nothing yet did both at once until Flamingo bridged the two. PNG ↓

The Breakthrough

Flamingo: bridge, don’t retrain. “Flamingo: a Visual Language Model for Few-Shot Learning” (Alayrac et al., 2022), submitted in April 2022, answered the gap with a deliberately conservative architecture: take two models that already work — a vision encoder and a language model — freeze both, and train only a small bridge between them. The vision side was NFNet-F6, a convolutional network of 435 million parameters pretrained with a contrastive image-text objective in the style CLIP established. The language side was a frozen decoder-only transformer from the Chinchilla family (the compute-optimal scaling work published one month earlier), at 70 billion parameters. Neither model’s weights moved during Flamingo training.

The bridge had two trainable pieces. A Perceiver Resampler (194 million parameters) took the variable-length grid of features the vision encoder produced — more for a video, fewer for a single image — and cross-attended them into a fixed set of 64 visual tokens, regardless of input size. Those 64 tokens were then injected into the frozen language model through new Gated Cross-Attention Dense layers, inserted before every seventh transformer block starting from the first, adding 10 billion trainable parameters on top of the frozen 70 billion. The whole assembly, plus vision encoder and resampler, totaled roughly 80 billion parameters for the largest Flamingo variant. Training data mixed paired image-text and video-text datasets with webpages where images and text appear interleaved in their natural reading order — the same distributional trick that let GPT-3 learn to follow in-context examples, now with pictures in the context.

Only the resampler and the gated cross-attention layers ever train; both the 435-million-parameter vision encoder and the 70-billion-parameter language model stay exactly as they were. PNG ↓

The result behaved like GPT-3 for pixels: shown a handful of image-question-answer triples in its prompt, Flamingo generalized to new images with no gradient update. It set a new few-shot state of the art on 16 of the vision-language benchmarks studied, reaching 67.6% accuracy on VQAv2 and 57.8% on OK-VQA at 32 shots — the OK-VQA number surpassing the previous fine-tuned state of the art, despite Flamingo seeing no task-specific training data at all. On video, it reached 31.0% on MSRVTT-QA and 52.3% on MSVD-QA, and produced captions scoring 113.8 CIDEr on COCO and 86.8 CIDEr on YouCook2, all in the 32-shot setting.

GPT-4: capable, but undisclosed. Eleven months later, in March 2023, OpenAI submitted the “GPT-4 Technical Report” (OpenAI, 2023), describing “a large-scale, multimodal model which can accept image and text inputs and produce text outputs.” The report is unusual for an anchor paper in this chapter: it discloses capabilities in detail — GPT-4 passes a simulated bar exam around the top 10% of test takers, and the team could predict aspects of its performance from smaller models trained on as little as one-thousandth the compute — but says nothing about how the multimodal input is fused, how large the model is, or what it was trained on. The report states this directly: “given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, [or] training method.” GPT-4 is confirmed multimodal and confirmed capable; its mechanism is not public.

LLaVA: the same idea, a hundredfold cheaper. One month after GPT-4, “Visual Instruction Tuning” (Liu et al., 2023) — an academic project out of University of Wisconsin–Madison and Microsoft Research, submitted in April 2023 — showed that Flamingo’s expensive bridge was not the only way to connect a frozen vision encoder to a frozen language model. LLaVA used a frozen CLIP ViT-L/14 encoder and a single trainable linear layer: a projection matrix that maps CLIP’s visual features directly into the word-embedding space of Vicuna, a LLaMA-derived instruction-tuned language model. No resampler, no cross-attention layers threaded through the LM — just enough of a linear map that visual features and word embeddings live in the same space, after which they are concatenated into one token sequence and trained with the ordinary next-token loss.

The harder problem was training data: nobody had a large corpus of multimodal instruction-following conversations. LLaVA’s authors solved this by feeding GPT-4 symbolic text descriptions of images — captions and bounding-box coordinates — and prompting it to synthesize conversations, detailed descriptions, and reasoning questions as if it could see the image directly. Fine-tuned on this GPT-4-generated data, LLaVA reached an 85.1% relative score compared with GPT-4 on a held-out synthetic multimodal instruction benchmark, and, fine-tuned further on ScienceQA, the combination of LLaVA and GPT-4 reached a new state-of-the-art accuracy of 92.53%. A method that had needed a DeepMind-scale training budget in April 2022 was reproduced by a small academic team, on public compute, twelve months later.

Gemini: no bridge at all. By December 2023, Google’s “Gemini: A Family of Highly Capable Multimodal Models” (Gemini Team, 2023) dropped the bridging pattern entirely. Where Flamingo and LLaVA start from separately pretrained vision and language components and connect them after the fact, Gemini was pretrained jointly on interleaved text, image, audio, and video data from the start — natively multimodal rather than stitched together. It shipped in three sizes, Ultra, Pro, and Nano, spanning data-center reasoning down to on-device deployment. Gemini Ultra reached 90.04% on MMLU, the first model reported to exceed the paper’s human-expert baseline of 89.8%, and advanced the state of the art on 30 of 32 benchmarks evaluated, including all 20 multimodal benchmarks examined. Three anchor papers in twenty months had moved from bridging two frozen models, to a cheap linear bridge anyone could train, to no bridge at all.

The Mathematics

Flamingo’s only genuinely new mechanism is how a frozen language model is made to attend to visual tokens without disturbing what it already knows how to do. The attention operation itself — queries, keys, values, softmax — is the mechanism attention and the transformer already define; what Flamingo adds is where that attention sits and how its contribution is scaled in:

\mathbf{h}' = \mathbf{h} + \tanh(\alpha) \cdot \mathrm{CrossAttn}\!\left(\mathbf{h},\, \mathbf{X}_{\mathrm{vis}}\right)

Take the frozen language model’s hidden state $\mathbf{h}$ at some layer, and let it query the 64 visual tokens $\mathbf{X}_{\mathrm{vis}}$ that the Perceiver Resampler produced, using ordinary cross-attention. Before adding that visual contribution back into the residual stream, scale it by $\tanh(\alpha)$ , where $\alpha$ is a learnable scalar initialized to zero. At the start of training, $\tanh(0) = 0$ , so the new layer contributes nothing and the model behaves exactly like the untouched frozen language model — training then gradually opens the gate as the visual signal proves useful. This is what buys stability: a 70-billion-parameter language model with strong existing behavior does not get knocked off course by a randomly initialized new layer dropped into the middle of it.

Symbol	Meaning
$\mathbf{h},\ \mathbf{h}'$	Language-model hidden state before and after the gated cross-attention layer
$\mathbf{X}_{\mathrm{vis}}$	The 64 visual tokens produced by the Perceiver Resampler
$\alpha$	Layer-specific learnable gate, initialized to 0
$\mathrm{CrossAttn}(\cdot,\cdot)$	Standard cross-attention: queries from $\mathbf{h}$ , keys and values from $\mathbf{X}_{\mathrm{vis}}$

LLaVA’s bridge is smaller than Flamingo’s by design — no gating, no new attention layers, just an embedding-space alignment:

\mathbf{H}_v = \mathbf{W}\, \mathbf{Z}_v

The two equations bracket a real design trade-off. Flamingo’s 10-billion-parameter gated cross-attention stack gives the frozen language model a dedicated mechanism for attending to visual detail at every few layers, which is part of why it holds up on fine-grained VQA; LLaVA’s single projection matrix is orders of magnitude cheaper to train — closer in spirit to the parameter-efficient adapters PEFT and LoRA formalizes for text-only fine-tuning — but leans entirely on the frozen LLM’s own attention to make sense of visual tokens once they arrive. Neither approach touches the vision encoder’s weights at all; both treat “connect a frozen expert model to a frozen language model” as the whole problem, which is exactly the pattern Gemini’s native joint pretraining, with no frozen components and no bridge to speak of, was built to avoid.

What It Enabled

Approach	Vision fusion mechanism	New trainable parameters	Base language model	Jointly pretrained?
Flamingo (Apr 2022)	Gated cross-attention + Perceiver Resampler	~10.2B	Frozen Chinchilla 70B	No
LLaVA (Apr 2023)	Linear projection into embedding space	A few million	Frozen Vicuna 7B/13B	No
GPT-4 (Mar 2023)	Undisclosed	Undisclosed	Undisclosed	Not stated
Gemini 1.0 (Dec 2023)	Native, no bridge	N/A — single training run	Gemini Ultra / Pro / Nano	Yes

The era did not stop at Gemini. Two more releases, both owned by this chapter on the master timeline, closed out the 2022–2024 window without publishing new architecture papers. In March 2024, Anthropic’s Claude 3 family — Opus, Sonnet, and Haiku — added vision input across all three tiers alongside a 200,000-token context window (the mechanics of extending context that far are long context’s territory, not this chapter’s). In May 2024, OpenAI’s GPT-4o — “o” for omni — unified text, vision, and audio processing in one model, responding to spoken audio in as little as 232 milliseconds and 320 milliseconds on average, and was priced at half of GPT-4 Turbo in the API.

Release	Date	What changed	Verified detail
Claude 3 family	Mar 2024	Vision added across Opus/Sonnet/Haiku	200K-token context window at launch
GPT-4o	May 2024	Single model for text, vision, and audio in real time	~320ms average voice response latency; half the API price of GPT-4 Turbo

Two threads run forward from here. The bridging pattern that Flamingo and LLaVA established did not disappear once native multimodal pretraining arrived — it is cheaper, and it remains the default way open-weight communities add vision to an existing language model, the same way instruction tuning made capable chat models reproducible outside frontier labs. The native-pretraining pattern that Gemini established, by contrast, is what frontier labs converged on for their flagship models: GPT-4o and later Gemini and Claude releases all train on multimodal data from the start rather than bridging afterward. Both threads feed directly into later chapters: perceiving a screenshot or a UI is a prerequisite for the tool-using systems agents covers, and evaluating what a multimodal model actually knows, as opposed to what it merely restates from training data, is the subject taken up in measuring minds and the frontier.

The bridging lineage never disappears, it stays the cheap default for open models, while frontier labs converge on the native-pretraining branch for their flagship releases. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Flamingo: a Visual Language Model for Few-Shot Learning	Alayrac et al.	2022	arXiv:2204.14198	First frozen-vision-plus-frozen-LM bridge; gated cross-attention preserves the LM at init; few-shot SOTA on 16 benchmarks
GPT-4 Technical Report	OpenAI	2023	arXiv:2303.08774	First frontier model with disclosed multimodal capability but withheld architecture; ~top-10% simulated bar exam
Visual Instruction Tuning	Liu et al.	2023	arXiv:2304.08485	LLaVA: open replication via one linear projection and GPT-4-generated instruction data; 85.1% relative to GPT-4
Gemini: A Family of Highly Capable Multimodal Models	Gemini Team, Google	2023	arXiv:2312.11805	First natively multimodal frontier model; 90.04% MMLU, first to exceed the paper’s human-expert baseline

PM Lens

The question I actually ask about a “multimodal” model card is whether the vision arrived by bridging or by native pretraining, because that predicts what the model is good for. A bridged model — the Flamingo/LLaVA lineage — is cheap to iterate on and fine, in my experience, for tasks close to captioning and simple visual QA; I would not trust it with dense OCR on a cluttered ad creative or fine layout comparison. A natively pretrained model — Gemini, GPT-4o, Claude 3’s vision tier — handles that kind of fine-grained visual reasoning noticeably better, which is exactly what the “no bridge at all” architecture buys. That distinction changes my build-versus-buy math: when I evaluated vision APIs for creative-QA tooling, GPT-4o’s real price advantage over GPT-4 Turbo mattered less than whether it could reliably read small brand-safety text baked into a banner image, which pushed the decision toward the natively multimodal tier every time. I now read “multimodal” on a spec sheet the way I read “encoder-decoder” a few years ago — as an architecture claim to verify, not a capability to take on faith.

Search