Image-to-Video and Motion Control

It is August 2023. Text-to-video diffusion models can now produce a few seconds of plausible motion from a written prompt, but a prompt is a blunt way to ask for a specific still image to start moving, for a camera to pan left rather than right, or for one object in a busy scene to move while everything else holds still. Over the next five months, three papers — one on trajectories drawn by hand, one on turning a single photograph into the first frame of a video, one on unifying camera and object control into swappable modules — each answer a different piece of the same question: given a picture that already exists, how should it move.

Where We Were

By mid-2023, video diffusion had a text-conditioning problem shaped like image diffusion’s before ControlNet arrived. Chapter 5 covers how Video Diffusion Models and its 2022 successors — Make-A-Video, Imagen Video, Phenaki — factorized spatial and temporal attention to generate short clips directly from a text prompt, and Chapter 6 covers how Video LDM and Stable Video Diffusion’s own architecture bolt temporal layers onto a frozen or lightly fine-tuned image latent diffusion model (Chapter 13 of the image-generation book owns the perceptual-compression math underneath both). None of these systems took a specific existing image as their starting point in a well-posed way. A prompt like “a photograph of this exact cat, but walking” has no way to attach “this exact cat” to a diffusion model that only reads text.

Two adjacent lines of work had chipped at the edges of the problem without solving it. Runway’s Gen-1, covered in Chapter 7, could restyle an existing video while preserving its structure — but it needed a full driving video as input, not a single still photograph, so it answered “how should this footage look” rather than “how should this picture move.” And the diffusion forward-and-reverse process itself (Chapter 8 of the image-generation book owns the DDPM math every video diffusion chapter in this book builds on) gave no special role to a starting frame at all — the reverse process denoises every frame in a clip from pure noise, symmetrically, with no built-in notion of “frame one is fixed and given, frames two through N are the question.”

Where a still image was used to seed a video at all, it was typically as a soft, secondary hint — a CLIP embedding blended into the same conditioning slot as the text prompt — rather than as a hard constraint the output had to match pixel-for-pixel on frame one. And whether a video was generated from text or from an image, the direction the camera moved and which of several visible objects should move were left entirely to the model’s own learned defaults. There was no dial for either one.

Each system gets closer to a moving picture without ever letting a user fix the start frame or the motion. The coral card is the gap DragNUWA, SVD and MotionCtrl each attack a piece of. PNG ↓

The Breakthrough

“DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory” (Yin et al., 2023), submitted August 16, 2023, is the earliest of the three and attacks the motion side of the problem directly. It accepts three optional inputs at once — a text prompt, a first-frame image, and a hand-drawn or dragged trajectory — and its central move is a way to make trajectory control learnable from ordinary video without any trajectory labels at all. A Trajectory Sampler extracts training trajectories on the fly from each clip’s own optical flow, computed with the Unimatch estimator: it lays down anchor points at a fixed spatial interval with small random jitter, then samples between one and eight of them per clip with probability weighted by how much motion passes through each point, and tracks each sampled point frame to frame to build a sparse path. A Gaussian filter (kernel size 99, standard deviation 10) then blurs each sparse path into a smooth trajectory map the network can consume as a dense per-pixel input, the same shape as an optical-flow field. A Multiscale Fusion module folds this trajectory map, together with the input image and text, into the UNet’s features at several resolutions at once, and an Adaptive Training curriculum trains first on the dense optical-flow field itself — which gives the model a strong, easy signal to learn visual consistency from — before switching to the sparse, user-friendly trajectories the sampler produces, so that the model does not have to learn “sparse points mean motion” and “motion should look realistic” simultaneously from scratch. DragNUWA trained on 10 million clips from WebVid and 75,000 BLIP2-captioned high-resolution clips from VideoHD, in two resolutions: a 320×192 “LD” variant and a 576×320 “HD” variant.

“Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets” (Blattmann et al., 2023), submitted November 25, 2023, answers a different half of the question: given that a still image should seed a video, what training recipe and what dataset actually produce a model that generates the right several seconds of motion from it. SVD’s own architecture is the Video LDM lineage Chapter 6 covers in full — temporal layers inserted into a 2D image latent-diffusion backbone; what this chapter is about is the training regime the paper builds around that backbone and the image-to-video mode it produces. The paper identifies three necessary stages — image pretraining, video pretraining, and a final high-quality video fine-tune — and argues that a systematically curated pretraining set matters as much as either stage’s architecture. Its curated Large Video Dataset, LVD, holds 577 million annotated clips; after filtering out clips with excessive text overlay, static motion, or other quality issues, the filtered LVD-F used for actual training holds 152 million clips. The resulting image-to-video model conditions on a single input frame in two separate ways at once: the frame’s own latent, after a small amount of extra noise is mixed in (a “noise augmentation” step that makes the model more tolerant of imperfect inputs), is duplicated across the output’s frame axis and concatenated channel-wise with the noisy video latent the UNet denoises, while the same frame is also encoded with a frozen CLIP image encoder and fed through cross-attention in the slot the text prompt normally occupies. The model ships in 14-frame and 25-frame variants at up to 576×1024 resolution, and the paper reports that human raters preferred its 25-frame image-to-video output over both Runway’s Gen-2 and Pika Labs in pairwise preference comparisons. Separately, the paper trains small camera-motion LoRA modules — “horizontally moving,” “zooming,” and “static” categories, inserted into the temporal attention blocks — on a small dataset annotated with camera-motion metadata, and shows the same base model extends to multi-view 3D generation, a capability Chapter 17 picks up.

“MotionCtrl: A Unified and Flexible Motion Controller for Video Generation” (Wang et al., 2023), submitted December 6, 2023, closes the loop by treating camera motion and object motion as two genuinely separate problems that had been getting tangled together, and gives each its own dedicated, swappable module rather than one control signal standing in for both. A Camera Motion Control Module (CMCM) reads a sequence of camera poses — each pose a 3×3 rotation matrix plus a 3×1 translation vector, 12 numbers per frame — and injects them into the second self-attention layer of the base model’s temporal transformers; only that one attention layer and a small added projection are trainable, everything else in the base model stays frozen. An Object Motion Control Module (OMCM) instead reads a per-frame optical-flow-style displacement field describing how a chosen object should move, extracted through several downsampling convolutions into multi-scale features, and injects those features only into the convolutional encoder — following the same precedent ControlNet and T2I-Adapter set, the paper restricts the signal to the encoder specifically to balance video quality against how strongly the object’s motion gets controlled. The two modules train and run independently: CMCM trains on an augmented version of RealEstate10K (62,992 clips with camera-pose ground truth and BLIP2-generated captions, roughly 50,000 iterations), OMCM trains on an augmented WebVid (about 243,000 clips with trajectories extracted by ParticleSfM’s motion segmentation, first 20,000 iterations on dense flow then 20,000 more on sparse trajectories, with the base model and CMCM held frozen throughout). Because each module is a small, separately trained addition rather than a retraining of the base network, MotionCtrl deploys, largely unmodified, on LVDM and VideoCrafter1 for text-to-video and on AnimateDiff — camera control, object control, both together, or neither, are all just a matter of which modules are attached.

DragNUWA, Stable Video Diffusion and MotionCtrl all fit this same shape, they just fill different input slots and freeze different parts of the UNet. PNG ↓

The Mathematics

Stable Video Diffusion’s image-to-video conditioning extends the standard denoising objective (Chapter 8 of the image-generation book owns the DDPM loss this builds on) with two parallel paths for the fixed first frame:

\mathcal{L}_{\text{I2V}} = \mathbb{E}_{z_0,\, t,\, \epsilon \sim \mathcal{N}(0,1)}\Big[\big\Vert \epsilon - \epsilon_\theta\big(\mathrm{concat}(z_t, \hat{z}^{(1)}),\, t,\, \mathrm{CLIP}(x^{(1)})\big) \big\Vert_2^2\Big]

$z_t$ is the noisy video latent at timestep $t$ , exactly as in ordinary diffusion training. $\hat{z}^{(1)}$ is the conditioning frame’s own latent, after a small amount of extra noise has been mixed in and after it has been copied across every frame position, then joined to $z_t$ channel-wise before the UNet sees either — this is what pins frame one in place spatially. $\mathrm{CLIP}(x^{(1)})$ is the same conditioning frame encoded by a frozen CLIP image encoder and read through cross-attention, in the same architectural slot a text embedding would otherwise occupy — this is what gives the model a semantic, not just pixel-level, sense of what the frame contains. Training the network to still predict the true noise $\epsilon$ with both of these attached is what buys image-to-video generation: the same denoising machinery every diffusion chapter in this book uses, with the conditioning frame wired in twice, once as literal pixels and once as meaning.

Symbol	Meaning
$z_0, z_t$	the clean and noised video latent at timestep $t$
$\hat{z}^{(1)}$	the noise-augmented latent of the conditioning frame, duplicated across all frames
$x^{(1)}$	the conditioning frame’s pixels
$\mathrm{CLIP}(x^{(1)})$	the conditioning frame’s CLIP image embedding
$\epsilon, \epsilon_\theta$	the true added noise and the network’s prediction of it
$t$	the diffusion timestep

DragNUWA’s Multiscale Fusion injects the trajectory map into the UNet the way ControlNet’s zero convolutions inject spatial conditioning into image diffusion (Chapter 15 of the image-generation book owns that mechanism) — as a learned, per-scale rescaling of the existing feature map rather than a simple addition:

h^{(l)} \leftarrow h^{(l)} \odot \big(1 + \gamma^{(l)}(g^{(l)})\big) + \beta^{(l)}(g^{(l)})

$h^{(l)}$ is the UNet’s own feature map at some internal resolution $l$ ; $g^{(l)}$ is the trajectory map, downsampled to match that same resolution. $\gamma^{(l)}$ and $\beta^{(l)}$ are small, zero-initialized convolutions that turn the trajectory map into a per-pixel scale and shift — at the very start of training both are zero, so $h^{(l)}$ passes through unchanged and the pretrained model’s behavior is preserved, exactly the same initialization trick Chapter 15 of the image-generation book uses for ControlNet’s zero convolutions. As training proceeds, the scale and shift move away from zero and the trajectory begins to reshape the feature map directly, at every resolution the UNet operates at, rather than only where the map happens to line up pixel-for-pixel with a single layer.

Symbol	Meaning
$h^{(l)}$	the UNet’s feature map at scale $l$
$g^{(l)}$	the trajectory map, downsampled to scale $l$
$\gamma^{(l)}(\cdot), \beta^{(l)}(\cdot)$	zero-initialized convolutions producing a scale and a shift from $g^{(l)}$
$\odot$	elementwise multiplication

MotionCtrl’s Camera Motion Control Module injects a pose sequence into the base model’s temporal self-attention by concatenation and projection, not by attention or by a residual add:

h' = \mathrm{FC}\big(\mathrm{Concat}\big(h,\, \mathrm{Expand}(RT)\big)\big)

$h$ is the output of the first self-attention layer inside the base model’s temporal transformer block, before any camera information is added. $RT = \{RT_0, \dots, RT_{L-1}\}$ is the sequence of camera poses for a video of length $L$ , each $RT_i$ a 12-number rotation-and-translation pair; $\mathrm{Expand}$ broadcasts that sequence spatially so it has the same height-and-width shape as $h$ . Concatenating the two along the channel dimension and passing the result through a fully connected layer $\mathrm{FC}$ projects back to $h$ ‘s original shape — this projection, $h'$ , is what the second self-attention layer then reads, replacing what $h$ alone would have been rather than adding a correction on top of it. Only this $\mathrm{FC}$ and the second self-attention layer are trainable, so the rest of the base video model, whether that is LVDM, VideoCrafter1, or AnimateDiff, is never touched. What it buys you: a single small module that reads an explicit camera path and steers a pretrained video model’s global motion toward it, independent of whatever module, if any, is separately steering an object’s local motion.

Symbol	Meaning
$h, h'$	the first temporal self-attention layer’s output, and the projection fed to the second
$RT = \{RT_0, \dots, RT_{L-1}\}$	the camera pose sequence, rotation plus translation per frame
$\mathrm{Expand}(\cdot)$	broadcasting the pose sequence to spatial dimensions
$\mathrm{FC}(\cdot)$	the trainable fully-connected projection, the only new weights in this module

What It Enabled

The three papers do not compete so much as stack — SVD supplies the image-conditioned foundation model, and DragNUWA-style trajectory control or MotionCtrl-style pose control can, in principle, sit on top of it as an add-on module, the same layered pattern Chapter 15 of the image-generation book describes for ControlNet, T2I-Adapter, and IP-Adapter stacking on a frozen Stable Diffusion checkpoint:

	DragNUWA	Stable Video Diffusion	MotionCtrl
Submitted	Aug 16, 2023	Nov 25, 2023	Dec 6, 2023
Primary contribution	Trajectory-conditioned motion from optical flow, no labels needed	Curated three-stage training recipe and dual image conditioning	Separate, swappable camera and object motion modules
Motion signal	Sparse dragged points, smoothed into a flow-like map	None directly — motion follows from the conditioning frame and text	Camera pose sequence (CMCM) and object flow field (OMCM), independently
Base model touched	Trained end-to-end with the conditioning	Own dedicated latent video diffusion backbone	Frozen — LVDM, VideoCrafter1, or AnimateDiff
Training data	10M WebVid clips + 75K VideoHD clips	577M-clip LVD, filtered to 152M	62,992 RealEstate10K clips (camera) + ~243K WebVid clips (object)

The same underlying capability was moving from research into shipping consumer features on its own commercial track: Runway added Motion Brush and an updated Director Mode to Gen-2 in November 2023, letting a user paint a region of a single image and assign it a direction of motion, or set a camera’s move directly, without knowing any of the mechanisms above exist — a product feature that shipped independently of, and in fact slightly ahead of, Stable Video Diffusion’s own paper landing that same month. Chapter 10 picks up that race in full as Gen-3, Kling, and Veo turn motion and camera control into a standard, expected feature of a commercial video model rather than a research demo. SVD’s own camera-motion LoRAs and multi-view extension also feed forward into the 3D-consistency work Chapter 17 covers, where a camera trajectory is not just a creative control but a way to probe a scene’s underlying 3D structure. And the general pattern here — read an explicit, structured control signal (a trajectory, a pose) and steer a pretrained generative model’s output to match it — reappears, generalized to arbitrary actions rather than only camera and object motion, in the action-conditioned world models Chapter 15 and Chapter 16 cover.

The three papers stack rather than compete, so a product or a later chapter can draw on any combination of them. PNG ↓

The limits were visible from the start. DragNUWA’s own paper reports no quantitative benchmark against prior trajectory-control baselines, only qualitative ablations showing all three of its conditioning signals are needed together; MotionCtrl’s camera and object modules are independently trained and only loosely validated when combined; and SVD’s image conditioning, however carefully trained, still inherits every generated video’s tendency to drift from the conditioning frame’s fine detail as more frames unroll — a temporal-consistency problem Chapter 12 returns to directly.

Key Papers

Paper	Authors	Year	Link	Why it matters
DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory	Yin et al.	2023	arXiv:2308.08089	Learns trajectory-based motion control from unlabeled video by sampling training trajectories directly from optical flow
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets	Blattmann et al.	2023	arXiv:2311.15127	Establishes the curated-data, three-stage recipe and dual latent-plus-CLIP conditioning that became the standard open image-to-video setup
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation	Wang et al.	2023	arXiv:2312.03641	Separates camera and object motion into independent, frozen-base-model add-on modules deployable across multiple backbones

PM Lens

The thing I watch for in this space is how fast a research control signal turns into a UI affordance a non-technical user can operate without knowing what a trajectory sampler is. Runway shipped Motion Brush and an upgraded Director Mode into Gen-2 the same month Stable Video Diffusion’s paper landed — a paint stroke and a camera slider standing in for everything this chapter’s equations do underneath. When I scope a “let users control the shot” feature now, I ask which of camera motion and object motion the request actually needs, because MotionCtrl’s split shows they are genuinely separable engineering problems with separate training data and separate failure modes — a camera-only feature is a much smaller ask than a general per-object motion editor, and conflating them in a spec produces a slower, more expensive build than either one alone would need.

Search