Evaluating and Monetizing Motion

It is December 2018. Text-to-video generation is five years from Sora and does not yet exist as a shippable product, but a six-author team spanning Google Brain and two European academic labs publishes a measurement problem’s solution before the field it measures has grown up: the Fréchet Video Distance, a single number meant to catch temporal flaws that no per-frame image metric can see. Five years later, once video diffusion models are actually good enough to sell by the second, that same measurement problem resurfaces at a different scale — one number turns out not to be enough, a benchmark has to check whether a dropped glass falls the way glass actually falls, and the bill for generating and grading a clip becomes one of the most expensive line items in AI.

Where We Were

Image generation had already solved a version of this problem once. The Inception Score, and then the Fréchet Inception Distance built on top of it, gave GAN researchers a number that compared a batch of generated images against real ones — FID’s math, and its later CLIP-based successor, are taught in full in Evaluation, Provenance, and Safety, and nothing about that derivation needs repeating here. Video inherited the same instinct but not, at first, a working metric. Early video generators — VGAN in 2016 and MoCoGAN in 2017, covered in GAN Video — were judged mostly the way early image GANs were: a grid of frames, a visual spot-check, and borrowed video-compression metrics like PSNR and SSIM that measure how close a reconstructed frame is to a known reference frame. None of those tools were built to ask a generative model’s actual question — does this synthesized motion look like real motion — because none of them look at more than one frame at a time, or compare a distribution of generated clips to a distribution of real ones the way FID compares images.

Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly closed that gap directly. Fréchet Video Distance, submitted in December 2018, adapted FID’s exact recipe to video by swapping the feature extractor: rather than a classifier trained on single ImageNet frames, FVD passes real and generated clips through an Inflated 3D Convnet (I3D), a network built for action recognition and pretrained on the Kinetics-400 and Kinetics-600 datasets, whose convolutions run across time as well as space and therefore encode motion, not just per-frame appearance (Unterthiner et al., 2018). That one substitution — video-native features in, otherwise identical Fréchet-distance math — gave the field its first automated number that could, in principle, penalize a model for getting the motion wrong even when every individual frame looked sharp.

FVD did not stay a niche fix. By the time video diffusion arrived in 2022 — Video Diffusion Models and the cascaded systems that followed, covered in Video Diffusion Models — FVD had become close to the field’s only reported automatic number, the way FID had been for GANs five years earlier. That created a narrower problem than the one FVD was built to solve: a metric validated on constrained benchmarks — robot-arm footage, human action clips, a StarCraft II environment the original paper introduced specifically because the human-video datasets available in 2018 were not challenging enough — was now the primary yardstick for open-domain, text-conditioned generation the 2018 paper never tested against, on model families the 2018 authors had no way to anticipate.

A metric validated on constrained 2018 benchmarks became the primary yardstick for open-domain models its authors never tested against. PNG ↓

The Breakthrough

FVD: the same distance, motion-aware features. The mechanics FVD adds to FID are narrow but consequential. I3D was chosen, the authors reasoned, because action recognition is a temporal extension of image classification — telling a video’s action apart from another requires the network to represent how appearance changes over time, not just what a single frame contains — so features pulled from deep inside an I3D network carry exactly the temporal signal a frame-by-frame Inception feature discards. The paper backed the metric with a large-scale human study, and reported that FVD correlated with human judgments of video quality more reliably than the frame-based alternatives available at the time. It was also candid about where that correlation broke down: the authors reported that FVD differences smaller than roughly 50 points showed agreement with human raters close to random, and that the Gaussian assumption underlying the Fréchet-distance math — treating a batch of feature vectors as if it were drawn from a single multivariate normal distribution — is “seldom an accurate representation” of the true feature distribution, merely a workable approximation for embeddings well-behaved enough to make it useful. Both caveats survived, mostly unread, into five years of results tables that reported FVD to three significant figures as if the number carried no error bars at all.

Only the feature extractor changes from FID. I3D's convolutions run across time, so identical frames with different motion still land apart in feature space. PNG ↓

EvalCrafter and VBench: the one-number premise breaks in the same month. By late 2023, dozens of text-to-video models had shipped, and two independent teams argued, six weeks apart, that a single distance could not tell a product team what it needed to know. Yaofang Liu and nine coauthors, submitting EvalCrafter in mid-October, built a 700-prompt suite sourced from real user queries and scored every generated clip on eighteen separate objective metrics grouped into four categories — visual quality, text-video alignment, motion quality, and temporal consistency — arguing explicitly that a metric like FVD, trained on datasets far smaller and narrower than the models now being evaluated, could not be trusted to summarize all four at once (Liu et al., 2023). Ziqi Huang and fifteen coauthors, submitting VBench at the end of November, went further still: rather than eighteen metrics reported alongside one another, VBench decomposes “video generation quality” into sixteen disentangled dimensions, each graded by its own specialized evaluator model, and states the field’s problem in one line — existing metrics such as the Inception Score and FVD “are inconsistent with human judgement,” and collapsing evaluation into a single number “obscures an individual model’s strengths and weaknesses” instead of showing where a model actually fails (Huang et al., 2023).

VBench’s sixteen dimensions split into two top-level groups. Seven Video Quality dimensions — subject consistency, background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic quality, and imaging quality — ask whether the footage looks and moves well on its own terms, with no reference to what was asked for. Nine Video-Condition Consistency dimensions — object class, multiple objects, human action, color, spatial relationship, scene, appearance style, temporal style, and overall consistency — ask whether the footage actually matches the prompt. Around one hundred prompts were purpose-written per dimension, and each dimension is scored by a tool suited to what it measures rather than one shared classifier: RAFT optical flow for how much genuine motion a clip contains, a video frame-interpolation model’s reconstruction error for how smoothly that motion unfolds, CLIP-style scorers for style and object-class matching. The paper validates every one of the sixteen scorers against its own collected set of human preference annotations, dimension by dimension, rather than asserting the correlation once and moving on the way the original FVD study largely had to.

Sixteen specialised evaluators replace one shared classifier, so a leaderboard can show where a model fails rather than only how far it falls short. PNG ↓

Physical commonsense: a third layer, once smoothness stopped being enough. VBench’s dimensions ask whether motion looks plausible to a viewer skimming a clip; neither VBench nor FVD asks whether that motion obeys physics a viewer might not consciously check. Hritik Bansal and nine coauthors closed that gap with VideoPhy, submitted in June 2024: a benchmark of captions describing real material interactions — solid-solid, solid-fluid, and beyond — scored by whether the generated video both matches the caption and respects physical commonsense, judged jointly rather than as two separate passes (Bansal et al., 2024). The best model they tested, CogVideoX-5B, satisfied both conditions at once on only 39.6% of prompts — meaning the majority of its outputs failed caption adherence, physical plausibility, or both. Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos pushed the same question further with Physics-IQ, submitted in January 2025: 396 real, professionally filmed clips covering fluid dynamics, optics, solid mechanics, magnetism, and thermodynamics, each used to condition a model that must predict five seconds of continuation, scored against how much a real camera’s natural variation differs from itself (Motamed et al., 2025). Across Sora, Runway, Pika, Lumiere, Stable Video Diffusion, and VideoPoet, the best performer — VideoPoet’s multiframe variant — reached a Physics-IQ score of 24.1%, against the 100% a real camera’s own footage scores by definition; the paper’s headline conclusion was that visual realism and physical understanding are unrelated properties of a generated clip, not two faces of the same capability.

The Mathematics

FVD reuses FID’s Fréchet-distance formula without modification — only the features going into it change:

\mathrm{FVD} = \lVert \mu_r - \mu_g \rVert_2^2 + \mathrm{Tr}\Big( \Sigma_r + \Sigma_g - 2\left(\Sigma_r \Sigma_g\right)^{1/2} \Big)

$\mu_r$ and $\Sigma_r$ are the mean and covariance of I3D features taken over a large batch of real video clips; $\mu_g$ and $\Sigma_g$ are the same two statistics computed over a batch of generated clips, both drawn from the same Kinetics-pretrained network. The first term measures how far apart the two feature-space centers sit; the trace term $\mathrm{Tr}(\cdot)$ measures how differently shaped the two feature clouds are, accounting for correlations between feature dimensions rather than treating each one independently. This is the identical equation Evaluation, Provenance, and Safety derives for FID; what buys the video-specific behavior is entirely upstream, in which network produced $\mu$ and $\Sigma$ in the first place — I3D’s convolutions run across the time axis, so two clips with identical individual frames but different motion still land at different points in feature space, which a frame-by-frame Inception feature could never distinguish.

Symbol	Meaning
$\mu_r, \Sigma_r$	mean and covariance of real-clip I3D features
$\mu_g, \Sigma_g$	mean and covariance of generated-clip I3D features
$\mathrm{Tr}(\cdot)$	matrix trace, sum of diagonal entries
$\mathrm{FVD}$	the resulting distance, lower means closer to real motion statistics

VBench’s sixteen dimensions use different underlying measurements — pixel differences, optical-flow magnitude, interpolation error — so each one is rescaled onto a shared direction before it can sit on the same leaderboard. Temporal flickering, for instance, starts as a raw per-frame pixel-difference score and is rescaled so that higher always means better, the convention every other dimension follows:

S_{\mathrm{flicker\text{-}norm}} = \frac{255 - S_{\mathrm{flicker}}}{255}

$S_{\mathrm{flicker}}$ is the raw score VBench’s flicker detector assigns a clip, on a 0-to-255 scale where a higher raw value means more visible flickering between consecutive frames — worse, not better. Subtracting from 255 and dividing by 255 flips and rescales that raw score into $S_{\mathrm{flicker\text{-}norm}}$ , a value in $[0,1]$ where higher now means smoother, matching the direction every other dimension already reports in. What this buys VBench: sixteen dimensions computed by sixteen different tools — RAFT optical flow for dynamic degree, a frame-interpolation model’s reconstruction error for motion smoothness, CLIP-style scorers for style dimensions — can be read off the same leaderboard and averaged into the two top-level groups without one dimension’s scale silently outweighing another’s.

Neither generating the clip nor grading it is free, and both bills land on the same accelerator budget. EvalCrafter’s suite runs eight models across 700 prompts through eighteen separate scoring metrics — CLIP-style alignment, an aesthetic predictor, an optical-flow network, and more — and VBench’s sixteen dimensions each call their own specialized model, meaning a single evaluation pass costs several additional forward passes on top of the video diffusion model’s own iterative sampling. That is the same shift The Economics of Test-Time Compute documents for language models: once inference compute itself becomes a line item rather than a rounding error, evaluation compute — the passes spent grading a model rather than serving a user — needs its own line too, and for video, where a single generation already costs cents to dollars per second, that line adds up fastest.

What It Enabled

None of the five methods in this chapter replaced the others; a team shipping a video model in 2026 typically runs all of them, at different points in a release cycle, against the same checkpoint.

Metric or suite	Submitted	What it measures	Compares to real footage	Known limitation
FVD	2018-12	Distributional distance on I3D motion features	Yes	Differences under ~50 points are close to random against human judgment; Gaussian assumption is an approximation
EvalCrafter	2023-10	18 metrics across visual quality, alignment, motion, consistency	Partially, per metric	No single aggregate score; 700 prompts is a fixed, English-only suite
VBench	2023-11	16 disentangled dimensions, quality and condition-consistency	Yes, per dimension, human-validated	No official weighting into one Total Score in the original release
VideoPhy	2024-06	Joint caption adherence and physical commonsense	No, human-judged	Binary joint pass/fail; best model cleared it only 39.6% of the time
Physics-IQ	2025-01	Physical variance against 396 real filmed clips	Yes, real camera footage is the 100% baseline	Visual-conditioned continuation only, not open-domain text-to-video

What that stack of benchmarks enabled, downstream, was the thing the chapter title puts second: a market willing to charge per second for output, because a checkpoint’s quality could finally be regression-tested between releases instead of eyeballed. A VBench leaderboard entry, or an internal FVD number tracked release over release, plays the same role for a video team that FID plays for an image checkpoint — not proof the model is ready to ship, but a gate that catches an obvious regression before a customer does. That gate matters more for video than it ever did for images, because the unit economics are not comparable. A text token, a generated image, and a second of generated video are priced on wildly different scales even at the same moment in the market:

Modality	Product	Priced unit	Rate
Text	OpenAI gpt-5.4-mini	per 1M output tokens	$4.50
Image	fal.ai, FLUX.1 [schnell]	per megapixel	$0.003
Image	OpenAI gpt-image-1, high quality	per image	$0.19
Video	OpenAI Sora 2, 720p	per second	$0.10
Video	OpenAI Sora 2 Pro, 1080p	per second	$0.70
Video	Google Veo 3, 720p or 1080p, with audio	per second	$0.40
Video	Runway Gen-4.5 API	per second	$0.12

A single high-quality image and a single second of video sit in the same rough price band; the difference is that a usable clip needs many of those seconds in a row, and — per VideoPhy and Physics-IQ — a meaningful share of them will fail on caption adherence or basic physics and need regenerating. A ten-second demo clip on Sora 2 Pro at 1080p runs $7 before a single frame is judged good enough to ship, which is why the token-count arithmetic behind that price — the spacetime patches a diffusion transformer actually has to attend over, covered in Sora and Spacetime Patches — and the distillation and quantization work aimed squarely at cutting that per-second cost, covered in Making Video Cheap, are the two chapters this one’s economics half connects to most directly. The competitive picture those prices sit inside — which vendor trades quality for cost at which tier — is the subject of The Commercial Race.

A market willing to charge per second for output exists because a checkpoint's quality could finally be regression-tested between releases instead of eyeballed. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Towards Accurate Generative Models of Video: A New Metric & Challenges	Unterthiner, van Steenkiste, Kurach, Marinier, Michalski, and Gelly	2018	arXiv:1812.01717	Introduced FVD, extending FID’s Fréchet-distance math to motion-aware I3D features, with a human study and explicit caveats about its own reliability
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models	Liu, Cun, Liu, Wang, Zhang, Chen, Liu, Zeng, Chan, and Shan	2023	arXiv:2310.11440	Scored text-to-video output on 18 metrics across four categories, arguing a single distance metric could not summarize modern generation quality
VBench: Comprehensive Benchmark Suite for Video Generative Models	Huang, He, Yu, Zhang, Si, Jiang, Zhang, Wu, Jin, Chanpaisit, Wang, Chen, Wang, Lin, Qiao, and Liu	2023	arXiv:2311.17982	Decomposed evaluation into 16 human-validated dimensions across quality and prompt-condition consistency, becoming the field’s default leaderboard
VideoPhy: Evaluating Physical Commonsense for Video Generation	Bansal, Lin, Xie, Zong, Yarom, Bitton, Jiang, Sun, Chang, and Grover	2024	arXiv:2406.03520	Showed the best available model satisfied caption adherence and physical commonsense jointly only 39.6% of the time
Do generative video models understand physical principles?	Motamed, Culp, Swersky, Jaini, and Geirhos	2025	arXiv:2501.09038	Introduced Physics-IQ, a real-filmed physics benchmark, and showed the best model reached only 24.1% of the real-footage baseline

PM Lens

The number that changes how I scope a video-generation feature isn’t a benchmark score, it’s the per-second bill. Sora 2 prices 720p output at $0.10 a second and its Pro tier at up to $0.70 a second for 1080p; Veo 3 charges $0.40 a second with audio included; Runway’s Gen-4.5 API works out to about $0.12 a second. A ten-second hero clip for a product demo can cost anywhere from $1 to $7 depending on which of those three I pick, before I’ve generated a single usable take — and VideoPhy’s finding that even the best open model gets caption adherence and physics right together only 39.6% of the time means I budget for retries, not just for seconds. I’ve started treating a VBench dimension score the way I treat FID for image checkpoints: a regression gate between releases, not proof the model is ready to ship. None of these numbers has ever told me whether a client will accept the physics of a dropped glass — that judgment still costs a human reviewer’s time, on top of whatever the API bills.

Search