Flow Matching and Transformer Backbones

It is September 2022. Latent diffusion has just shipped in public, class-conditional diffusion models have already been shown to beat GANs on ImageNet, and the field’s sampler research is deep into shaving steps off a Markov chain that was never designed to be short. In the space of four weeks, two papers — written independently, using different notation, arriving at almost the same equation — propose that the chain be discarded altogether: forget the noising process, forget the score, and just teach a network to point along a straight line from noise to data. Three months later a third paper removes the convolutional U-Net that had carried every image diffusion model since 2020 and replaces it with a plain transformer. By March 2024, a fourth paper fuses all of it — straight-line training and a transformer backbone — into the architecture behind Stable Diffusion 3.

Where We Were

By mid-2022, image diffusion had a working recipe and a growing pile of patches on top of it. Denoising diffusion probabilistic models, introduced in June 2020, defined a forward process that corrupts data with Gaussian noise over many steps and a network that learns to reverse it (Diffusion Foundations). The reverse-time SDE and probability-flow ODE view, established in late 2020, had shown that this reverse process could be run as a deterministic ODE solve rather than a stochastic chain, cutting sampling from a thousand steps to a few dozen (Scores, SDEs, and Faster Sampling). Classifier-free guidance, published in July 2022, gave practitioners a knob to trade diversity for fidelity to a text prompt (Guidance). And Stable Diffusion’s public release that August had proved a latent-space U-Net could run on a single consumer GPU.

None of this changed two facts about how these models were built. First, every one of them was still, underneath the ODE reframing, tied to a diffusion process specifically — a forward corruption defined by adding Gaussian noise according to a schedule, with the sampling ODE’s shape inherited from that schedule rather than chosen for convenience. That coupling had a training-time cost too: fitting a diffusion model meant committing to a noise schedule, a variance parameterization, and a loss weighting across timesteps, each of which had accumulated its own ablation literature since 2020 without a principled way to choose among the options other than trial and error. Second, the network doing the work was a convolutional U-Net, adapted from image segmentation architectures, with skip connections between a downsampling and upsampling path and, in the strongest ImageNet models, self-attention layers inserted at a handful of resolutions to give the convolutions some global context. That architecture produced excellent images, but nobody had shown that making a U-Net bigger produced the kind of clean, predictable quality gains that the language-modeling community had already documented for transformers under Kaplan-style scaling laws (Scaling Laws). Diffusion’s sample-quality gains through 2021 and 2022 had come from architecture tweaks, better noise schedules, and guidance — not from a demonstrated compute-scaling curve.

Two years of real progress, all built on the same noising process and the same convolutional backbone. Neither assumption had yet been tested. PNG ↓

Two separate questions were sitting unaddressed. Did the generative path from noise to data have to be a diffusion process at all, or was diffusion just one convenient way to build some deterministic path — and could a simpler, straighter path be chosen instead? And did the network computing that path have to be a U-Net, or would a transformer, given the same compute, do better?

The Breakthrough

Flow matching (Lipman et al., 2022) answered the first question by stepping outside diffusion entirely. Rather than deriving a generative ODE from a specific noising process and then estimating its score, the paper starts from the ODE itself: pick any smooth “probability path” — a family of distributions $p_t$ that interpolates from a simple prior at $t=0$ to the data distribution at $t=1$ — and directly regress a neural network onto the velocity field that generates that path. The paper shows this regression can be done with a simple, simulation-free loss, and that the specific choice of an optimal-transport path (a straight-line interpolation between a noise sample and a data sample) trains faster and samples faster than paths inherited from diffusion, while a Gaussian-diffusion-style path is also expressible in the same framework, making flow matching a strict generalization rather than a competitor.

Rectified flow (Liu, Gong & Liu, 2022), submitted four weeks earlier by a separate group, arrived at essentially the same conditional objective from a different direction — transport theory rather than continuous normalizing flows — and added an operational idea flow matching’s original paper does not: a network trained to follow straight-line paths between paired noise and data samples can be used to generate new pairs by simulating its own ODE, and refitting a fresh network to those self-generated pairs produces straighter paths still. The paper calls this iterative refitting reflow. Straighter paths need fewer steps to integrate accurately, so a few rounds of reflow, followed by distillation onto a single Euler step, turn a model that needed dozens of steps into one that needs one. Both papers also share a practical consequence that is easy to miss in the equations: because the target velocity no longer depends on a noise schedule, the loss no longer needs the per-timestep variance and weighting terms diffusion training carries, so a flow-matching implementation has noticeably fewer moving parts to get wrong than a comparable DDPM-style training pipeline.

Scalable diffusion models with transformers (Peebles & Xie, 2022) — DiT — answered the second question, and did so independently of both flow-matching papers, arriving eleven weeks after flow matching. The recipe is architecturally blunt: take a latent-diffusion model’s noised latent, cut it into fixed-size patches the way a Vision Transformer cuts an image into patches, drop the sequence of patch tokens through a stack of standard transformer blocks conditioned on the timestep and class label, then reassemble the output tokens back into a latent. There is no U-Net, no downsampling path, no skip connection — the spatial resolution of the token grid is fixed by the patch size and never changes inside the network. Trained on the DDPM objective exactly as before, only the backbone changes. The result is a direct, if unglamorous, empirical claim: DiT-XL/2 reaches a state-of-the-art FID of 2.27 on class-conditional ImageNet at 256×256, and — the finding the paper is actually built around — models with more Gflops, whether from more transformer depth and width or from smaller patches producing more tokens, consistently produce lower FID. Diffusion modeling had acquired the scaling curve the U-Net era never quite delivered.

No downsampling path and no skip connections; the token grid's resolution is fixed by the patch size for the whole forward pass. PNG ↓

Scaling rectified flow transformers for high-resolution image synthesis (Esser et al., 2024) — the Stable Diffusion 3 paper — is where these two independent lines meet. SD3 trains with a rectified-flow-style objective, improved with noise-sampling schedules biased toward the perceptually important middle of the trajectory, and runs it on a transformer backbone descended from DiT’s patchify-and-attend design, called MM-DiT. MM-DiT’s specific contribution is to give image and text tokens separate weight matrices for their own modality while still attending jointly: the two token sequences are concatenated before self-attention, so a single attention operation (the transformer’s self-attention mechanism, owned by Chapter 5 of Language Models) mixes information across both, while each modality keeps its own linear projections into that shared attention, letting image and text representations exchange information without forcing one modality’s features through weights tuned for the other. Esser et al. trained a family of these models from 450M to 8B parameters and reported the validation loss decreasing smoothly with both size and training compute, with no sign of saturation at the largest scale studied — the DiT scaling result, reproduced at production scale on a text-to-image system. Five months later, Black Forest Labs — founded by several of the SD3 paper’s own authors — released FLUX.1, a family of openly available rectified-flow transformer models scaled to 12B parameters, adding rotary position embeddings and parallel attention layers for hardware efficiency (Black Forest Labs, 2024). FLUX.1 shipped in three tiers with different licenses rather than one model — [pro] served only through Black Forest Labs’ own API, [dev] open-weighted for non-commercial use, and [schnell] a guidance-distilled, Apache-licensed variant built for few-step, low-latency generation — which meant the same architectural lineage reached hobbyists, startups building products, and Black Forest Labs’ own paid API simultaneously, on different commercial terms.

The Mathematics

Flow matching’s starting point is a marginal vector field $u_t(x)$ that generates the probability path $p_t$ — the field a network would need to learn to reproduce that path exactly. Directly regressing onto it is intractable, because $u_t(x)$ has to be marginalized over the entire data distribution. The paper’s central result is that this problem can be sidestepped:

\mathcal{L}_{\mathrm{CFM}}(\theta) = \mathbb{E}_{t \sim U[0,1],\; x_1 \sim q(x_1),\; x_0 \sim p_0}\left\| v_\theta(x_t, t) - (x_1 - x_0) \right\|^2, \qquad x_t = (1-t)\,x_0 + t\,x_1

$x_0$ is a sample from the simple prior (Gaussian noise), $x_1$ is a real data sample, and $x_t$ is just the point a fraction $t$ of the way along the straight line between them. $v_\theta$ is the network, trained to predict the velocity of that line — which, because the line is straight, is the constant vector $x_1 - x_0$ regardless of $t$ . What this buys you: minimizing this fully tractable, simulation-free loss produces the same gradient as minimizing the intractable marginal loss against $u_t(x)$ , so training is exactly as simple as DDPM’s noise-prediction loss, but the target path is no longer constrained to whatever a diffusion schedule happens to produce — it can be the shortest path between the two points. The paper’s own optimal-transport loss carries a small floor variance $\sigma_{\min}$ , with target $x_1 - (1-\sigma_{\min})x_0$ ; the clean form above is that loss in the $\sigma_{\min} \to 0$ limit, which is exactly the objective rectified flow writes below.

Symbol	Meaning
$t$	position along the path, from $0$ (noise) to $1$ (data)
$x_0$	a sample drawn from the prior distribution $p_0$
$x_1$	a sample drawn from the data distribution $q$
$x_t$	the linear interpolation between $x_0$ and $x_1$ at time $t$
$v_\theta$	the network’s predicted velocity at $(x_t, t)$
$u_t(x)$	the true marginal velocity field the network approximates

Rectified flow writes the same linear-interpolation objective with its own notation and treats it as an ODE to be solved at sample time:

Z_t = t\,Z_1 + (1-t)\,Z_0, \qquad \frac{dZ_t}{dt} = v(Z_t, t)

Symbol	Meaning
$t$	position along the path, from $0$ (noise) to $1$ (data)
$Z_0$	a sample drawn from the source, noise distribution
$Z_1$	a sample drawn from the target, data distribution
$Z_t$	the linear interpolation between $Z_0$ and $Z_1$ , and the ODE’s state at time $t$
$v$	the learned velocity field driving the ODE

A single round of this training does not make the learned field’s integrated trajectories perfectly straight, because the network averages over many crossing pairs. Rectified flow’s fix is to refit on self-generated data:

v^{(k+1)} = \arg\min_{v} \int_0^1 \mathbb{E}\left[\left\| \left(\hat Z_1^{(k)} - Z_0\right) - v\!\left(Z_t^{(k)}, t\right) \right\|^2\right] dt, \qquad \hat Z_1^{(k)} = Z_0 + \int_0^1 v^{(k)}(Z_t^{(k)}, t)\, dt

Instead of pairing a noise sample with an unrelated real data sample, this reflow step simulates the current velocity field $v^{(k)}$ all the way from $Z_0$ to get an endpoint $\hat Z_1^{(k)}$ , then trains the next model $v^{(k+1)}$ to reproduce the straight line between that specific pair. What it buys you: each reflow round provably reduces a measure of path curvature, so after one or two rounds the ODE is close enough to straight-line that a single Euler step — one network evaluation — approximates what previously took dozens of solver steps. Liu, Gong & Liu report an FID of 2.58 on CIFAR-10 for the first rectified flow fully integrated with an adaptive solver, and an FID of 4.85 with a single Euler step after two rounds of reflow plus distillation, ahead of the strongest prior one-step baseline they compare against.

Symbol	Meaning
$k$	the reflow round index
$v^{(k)}$	the velocity field learned in round $k$
$Z_0$	a noise sample drawn from the source distribution
$\hat Z_1^{(k)}$	the endpoint reached by integrating $v^{(k)}$ ‘s ODE from $Z_0$ to $t=1$
$Z_t^{(k)}$	the linear interpolation between $Z_0$ and $\hat Z_1^{(k)}$ at time $t$
$v^{(k+1)}$	the next round’s velocity field, fit to reproduce that straight line

The reflow-to-distillation route to single-step sampling sits alongside the consistency-model and adversarial-distillation techniques covered in Few-Step Generation; flow matching’s contribution is a training objective that makes the underlying trajectory straight before any distillation step is applied, rather than distilling a curved one.

What It Enabled

The backbone and objective shift did not replace diffusion’s conceptual apparatus so much as generalize it, and the two changes — the training objective and the network architecture — turned out to be independent and compatible, which is why SD3 and FLUX.1 could combine both at once.

Era	Backbone	Training objective	Typical sampling steps	Flagship models
2020–2022	Convolutional U-Net	Denoising score matching (DDPM loss) on a fixed diffusion schedule	20–1,000	ADM, Stable Diffusion 1.x
2022–2024	Transformer (DiT / MM-DiT)	Conditional flow matching / rectified flow regression on a straight-line path	1–50	DiT, SD3, FLUX.1

Model or result	Configuration	Reported result	Source
DiT-XL/2	Transformer backbone, class-conditional ImageNet 256×256	FID 2.27, state of the art at publication	Peebles & Xie, 2022
Rectified flow, 1-RF	Fully solved with an adaptive RK45 solver	FID 2.58 on CIFAR-10	Liu, Gong & Liu, 2022
Rectified flow, 2-RF + distillation	Single Euler step (1 network call)	FID 4.85 on CIFAR-10	Liu, Gong & Liu, 2022
SD3 / MM-DiT family	450M to 8B parameters	Validation loss decreases smoothly with scale, no saturation observed	Esser et al., 2024
FLUX.1	Rectified-flow transformer, RoPE, parallel attention	12B parameters, three tiers: pro (API-only), dev and schnell (open weights)	Black Forest Labs, 2024

The training objective and the network backbone were solved independently and turned out to be compatible, which is why SD3 could combine both at once. PNG ↓

The transformer backbone travels beyond still images: Sora’s spacetime-patch tokenization applies the same patchify-and-attend pattern to video, an application covered in Sora and Spacetime Patches. Within image generation, the flow-matching-plus-transformer combination is the substrate that later frontier systems build on, discussed in The Current Frontier.

The step-count drop compounds with the architecture change rather than substituting for it. A DDPM-era U-Net serving fifty sequential denoising calls and a distilled rectified-flow transformer serving one call are not just different models — they sit on different points of a cost curve that a product team has to price against, because every step is a full forward pass through the network regardless of how large that network is. Reducing steps from fifty to one is roughly a fifty-fold reduction in the compute spent per image, before accounting for any difference in per-step cost between a U-Net and a transformer of comparable quality. That arithmetic is what turned flow matching from an architecture-research result into a deployment decision.

Key Papers

Paper	Authors	Year	Link	Why it matters
Flow Matching for Generative Modeling	Lipman, Chen, Ben-Hamu, Nickel & Le	2022	arXiv:2210.02747	Generalizes diffusion’s ODE to any probability path, trained with a simulation-free regression loss
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow	Liu, Gong & Liu	2022	arXiv:2209.03003	Independently derives the same linear-path objective and adds reflow, the iterative straightening procedure behind one-step sampling
Scalable Diffusion Models with Transformers	Peebles & Xie	2022	arXiv:2212.09748	Replaces the U-Net with a plain transformer and shows compute scales predictably with FID
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis	Esser et al.	2024	arXiv:2403.03206	Combines rectified flow and a DiT-descended MM-DiT backbone at production scale, up to 8B parameters
Announcing Black Forest Labs and FLUX.1	Black Forest Labs	2024	bfl.ai	Open-weight 12B rectified-flow transformer that put the SD3 lineage into wide public use

What I track here is how fast the objective-plus-backbone combination became a commodity. Within a year of SD3’s paper, FLUX.1 shipped the same recipe as open weights, and by the time I’m writing this, hosted inference for it is a line item rather than a research question — fal.ai currently prices FLUX.1 [dev] at $0.025 per megapixel and the distilled [schnell] variant at $0.003 per megapixel, cheap enough to run generation inside a product feature rather than as a standalone tool. The lesson for a PM evaluating any “new” generative architecture now is to ask two separate questions, because this chapter is proof they move independently: is the training objective actually different (flow matching versus diffusion), and is the backbone actually different (transformer versus U-Net)? A launch announcement that blurs the two is usually reselling one of them.

Search