GAN Video

It is September 2016. The generative adversarial network has spent two years reshaping still-image synthesis, and a group at MIT asks the obvious next question: if a discriminator can teach a generator what a convincing photograph looks like, can a discriminator teach a generator what a convincing second of video looks like? The answer arrives in three widening steps over the following three years — a tiny 64-pixel clip, a generator that separates what moves from how it moves, and finally an attempt to do both at the scale of a real internet video dataset.

Where We Were

By late 2016 the adversarial game was already the leading recipe for sharp still images. The GAN objective — a generator and a discriminator locked in a minimax game with no explicit density and no Markov chain — is covered in full in image-generation, Chapter 3; this chapter assumes that objective and asks what changes when the sample being judged has a time axis instead of just two spatial axes.

Video research had already tried the alternative path — regression rather than adversarial judgment — and run into a well-documented wall. Recurrent networks trained to predict future frames by minimizing pixel-wise error, surveyed in early unsupervised video representation work from February 2015, produce blurry, washed-out predictions whenever the future is genuinely uncertain: a mean-squared-error loss is minimized by averaging over every plausible continuation, and the average of several sharp, different futures is a blur. Why that specific failure happens, and why it is a property of the loss rather than of the network, is owned by Chapter 1 — this chapter is the point where GAN training becomes one of the two mainstream fixes.

Every step plays the same minimax game image generation already had. What changes over three years is how much of a clip the discriminator can see at once. PNG ↓

The case for trying the adversarial route in video was the same case that had already worked for images: a discriminator judging an entire generated clip has no reason to reward the safe, blurry average of several possible futures, because a blurry clip is easy for a discriminator to flag as fake. The generator is pushed toward committing to one sharp, plausible continuation instead of hedging across all of them. What nobody had yet worked out, going into 2016, was how to build a discriminator — and a generator — that could look at motion at all, rather than just judging one frame at a time.

The Breakthrough

Generating Videos with Scene Dynamics (Vondrick et al., 2016) is the first paper to point the adversarial game at raw video. Rather than feed noise through one generator network, it splits the generator into two streams that share early weights but specialize toward the end. A foreground stream applies spatio-temporal convolutions — convolutions that slide across space and across time — to model the parts of the scene that move. A background stream applies ordinary spatial convolutions to a single frame and then replicates that static frame across every time step, on the observation that in most short clips the camera and the setting barely change while the subject does. A third output, a per-pixel mask produced by a branch that shares weights with the foreground stream, decides at every location and every frame how much of the final pixel comes from the moving foreground versus the static background. The generator, discriminator, and training procedure otherwise follow the plain adversarial objective from image-generation, Chapter 3: a 3-D-convolutional discriminator judges whole clips as real or generated, and gradients flow back through both streams and the mask.

The discriminator only has to flag a blurry clip as fake to push both streams toward one sharp, committed guess. PNG ↓

Trained on large amounts of unlabeled internet video, the model — nicknamed VGAN — generated clips only 64 pixels square and one second long at full frame rate (32 frames), a scale chosen because adversarial video training was, at the time, barely stable even at that size. The paper reported three results, all from a human preference study and two follow-on experiments: workers preferred the two-stream generator over a same-architecture autoencoder baseline 82% of the time on average, and preferred it over a one-stream ablation of the model itself 53% of the time, evidence that the foreground/background split was doing real work rather than just adding parameters; the same generator, given a single static image as a conditioning input, could paint in a plausible short future for that image, rarely correct but often plausibly in motion; and, perhaps most surprisingly, the internal features the discriminator learned while telling real motion from generated motion turned out to be useful for recognizing human actions with minimal labeled supervision — fine-tuned on UCF-101, the discriminator’s features reached 52.1% accuracy, edging out hand-crafted STIP features (43.9%) and a contemporary unsupervised video-representation method, Shuffle and Learn (50.2%), though still well short of the 91.4% a fully supervised ImageNet-pretrained network reached on the same task. It was an early hint that a model trained only to distinguish real from fake video had, along the way, learned something like a representation of what motion in the world normally looks like. A sparsity penalty on the mask, discussed under the Mathematics section below, was needed in practice to stop the model from routing everything through the more flexible foreground stream and ignoring the background stream entirely.

MoCoGAN: Decomposing Motion and Content for Video Generation (Tulyakov et al., 2017) took a different cut at the same problem: instead of separating a video into foreground and background pixels, it separates the latent code that generates a video into a content part and a motion part. The intuition is that a short clip of a person waving has one thing that stays fixed across the whole clip — who the person is, what they are wearing, the room they are in — and one thing that changes frame to frame — the trajectory of the wave. MoCoGAN fixes a single content vector for an entire clip and drives a separate recurrent network, at every time step, with fresh noise to produce a motion vector that evolves over time; the frame generator then maps the concatenation of the fixed content vector and the current motion vector to a single image, one frame at a time. Two discriminators supervise training: an image discriminator that judges individual sampled frames for realism, and a video discriminator that judges a sampled sequence of consecutive frames for realistic motion. Crucially, the recurrent motion network is driven by an independent noise draw at every step rather than by its own previous output alone, which the authors found necessary to keep the model from settling into one repetitive motion pattern regardless of the noise it was given.

Because content and motion are separate inputs, MoCoGAN could, after training, be asked to hold content fixed while sampling different motion — the same face making different expressions — or hold motion fixed while sampling different content, a capability image-domain conditional GANs (covered as the conditioning mechanism in image-generation, Chapter 3) never had a temporal axis to demonstrate. The paper evaluated this on four datasets of increasing difficulty: a synthetic shape-motion dataset of moving circles and squares, the MUG facial-expression database, a Tai-Chi clip collection scraped from YouTube, and the Weizmann human-action dataset — none at internet scale, but each chosen to isolate a specific kind of motion the decomposition needed to capture.

Adversarial Video Generation on Complex Datasets (Clark et al., 2019) is the paper that asked whether any of this scaled. Its model, DVD-GAN — Dual Video Discriminator GAN — borrowed its generator and much of its discriminator design directly from BigGAN (Brock et al., 2018), the large-scale class-conditional image GAN covered in image-generation’s Chapter 6: class-conditional batch normalization, residual blocks, and spectral normalization all carried over from images into video. The paper’s own innovation was splitting judgment of a clip into two specialized discriminators rather than one: a spatial discriminator that samples a handful of individual frames at full resolution and judges each one’s image quality, and a temporal discriminator that looks at the whole clip — every frame, but spatially downsampled — and judges whether the motion across time is coherent. Because the spatial discriminator only ever sees a few full-resolution frames rather than the whole clip, and the temporal discriminator only ever sees downsampled frames, the combination processes far fewer pixels than a single discriminator looking at every frame at full resolution would — a design the authors reported cut per-clip pixel processing by 58% for a 48-frame, 128-pixel clip, which is what made training on a dataset the size of Kinetics-600 tractable at all.

DVD-GAN trained on Kinetics-600, a DeepMind action-recognition dataset of roughly 500,000 video clips across 600 action classes, at resolutions up to 256 pixels square and clip lengths up to 48 frames, using between 32 and 512 replicas on Google’s TPU pods for 12 to 96 hours depending on configuration. It reported a new state of the art in Fréchet Inception Distance for frame prediction on Kinetics-600 and in Inception Score for unconditional video synthesis on UCF-101 — the first time an adversarial video model had been shown to hold state-of-the-art numbers on datasets that size, rather than on small, curated clip collections.

The Mathematics

VGAN’s contribution is not a new training objective — it trains with the same minimax game as image-generation, Chapter 3 — but a new generator architecture, which is worth writing out because the composition rule is the whole idea:

G(z) = m(z) \odot f(z) + \big(1 - m(z)\big) \odot b(z)

$z$ is the input noise vector. $f(z)$ is the moving-foreground stream’s output video, built with spatio-temporal convolutions. $b(z)$ is the static-background stream’s output, a single generated frame replicated across every time step. $m(z)$ is a per-pixel, per-frame mask with values between 0 and 1, produced by a branch that shares weights with the foreground stream; $\odot$ is element-wise multiplication. At every pixel and frame, the mask decides how much of the final image comes from the foreground stream versus the background stream. What it buys you: a generator that can keep the background perfectly static — which is what most of a short real clip actually does — while spending its limited capacity on the part of the scene that moves, instead of having to re-synthesize every pixel of every frame from scratch.

Symbol	Meaning
$z$	input noise vector
$f(z)$	foreground stream output — a full video, spatio-temporal convolutions
$b(z)$	background stream output — one frame, spatial convolutions only, replicated over time
$m(z)$	mask, values in $[0,1]$ , one per pixel per frame
$\odot$	element-wise multiplication

Training the mask with only the adversarial loss let the model route almost everything through the more expressive foreground stream and ignore the background stream, which defeated the point of separating them. The paper added a small $\ell_1$ penalty to push the mask toward using the background stream wherever possible:

\mathcal{L}_{\mathrm{mask}} = \lambda \lVert m(z) \rVert_1, \quad \lambda = 0.1

MoCoGAN’s core idea is a factorization of the latent code rather than of the pixels. A clip’s content is drawn once and held fixed; its motion is produced step by step by a recurrent network driven by fresh noise at each step:

z^{(k)} = \big[z_C;\ z_M^{(k)}\big], \qquad z_C \sim \mathcal{N}(0, I_{d_C}), \qquad z_M^{(k)} = R_M\big(z_M^{(k-1)}, \epsilon^{(k)}\big),\ \ \epsilon^{(k)} \sim \mathcal{N}(0, I_{d_E})

$z_C$ is a content vector drawn once from a fixed Gaussian prior and reused, unchanged, for every frame $k$ of a clip. $z_M^{(k)}$ is the motion vector at frame $k$ , produced by a recurrent network $R_M$ (a GRU in the paper) from the previous motion vector and a fresh noise draw $\epsilon^{(k)}$ — motion is a stochastic process unfolding over time, content is not. The two are concatenated, written $[z_C; z_M^{(k)}]$ , and passed to the frame generator to produce frame $k$ . What it buys you: after training, holding $z_C$ fixed while resampling the motion process yields the same subject performing different motions, and holding the motion process fixed while resampling $z_C$ yields different subjects performing the same motion — control that a single undifferentiated latent code has no way to offer.

Symbol	Meaning
$z^{(k)}$	full latent code for frame $k$ , the concatenation of content and motion
$z_C$	content vector, drawn once from $\mathcal{N}(0, I_{d_C})$ and held fixed for the whole clip
$z_M^{(k)}$	motion vector at frame $k$
$R_M$	recurrent network (GRU) producing the motion vector at each step
$\epsilon^{(k)}$	fresh Gaussian noise fed to $R_M$ at step $k$ , drawn from $\mathcal{N}(0, I_{d_E})$

Two discriminators supervise the result, one judging single frames and one judging short sequences:

\mathbb{E}_v\big[-\log D_I(S_1(v))\big] + \mathbb{E}_{\tilde v}\big[-\log\big(1 - D_I(S_1(\tilde v))\big)\big] + \mathbb{E}_v\big[-\log D_V(S_T(v))\big] + \mathbb{E}_{\tilde v}\big[-\log\big(1 - D_V(S_T(\tilde v))\big)\big]

$v$ is a real video, $\tilde v$ is a generated one. $S_1$ samples one frame from a clip; $S_T$ samples $T$ consecutive frames. $D_I$ is the image discriminator, judging the single sampled frame from $S_1$ ; $D_V$ is the video discriminator, judging the $T$ -frame sequence from $S_T$ for motion realism. Each of the four terms is a standard adversarial log-loss term, exactly the kind covered in image-generation, Chapter 3, just applied twice — once to single frames, once to short sequences. What it buys you: a generator that has to satisfy two separate critics simultaneously, one that only cares whether any given frame looks like a real image, and one that only cares whether motion across a handful of frames looks like real motion — splitting “does this look right” from “does this move right” into two trainable signals instead of asking one discriminator to judge both at once.

Symbol	Meaning
$v$ , $\tilde v$	a real video clip; a generated video clip
$S_1$ , $S_T$	operators sampling one frame, or $T$ consecutive frames, from a clip
$D_I$	image discriminator, judges a single sampled frame
$D_V$	video discriminator, judges a $T$ -frame sequence
$\mathbb{E}$	expectation over the given distribution of clips

DVD-GAN, like its parent architecture BigGAN, replaced the log-loss GAN objective with a hinge-loss variant, a margin-based objective rather than a probability-based one:

\mathcal{L}_D = \mathbb{E}_{x \sim p_{\mathrm{data}}}\big[\max(0,\, 1 - D(x))\big] + \mathbb{E}_{z \sim p(z)}\big[\max(0,\, 1 + D(G(z)))\big], \qquad \mathcal{L}_G = -\mathbb{E}_{z \sim p(z)}\big[D(G(z))\big]

$D(x)$ is now an unbounded score rather than a probability; the discriminator is trained to push that score above 1 for real data and below $-1$ for generated data, and the $\max(0, \cdot)$ hinge means it pays no further penalty once a sample is safely past that margin — unlike the log-loss objective, which keeps rewarding a discriminator for pushing scores toward extremes forever. The generator simply tries to maximize the score the discriminator assigns to its own samples. What it buys you: a training signal that, empirically, proved more stable at BigGAN’s scale, and which DVD-GAN inherited unchanged; in the model itself, $D(x)$ is the sum of a spatial discriminator’s score over $k$ sampled full-resolution frames and a temporal discriminator’s score over the whole clip downsampled by average pooling, so a video is only rated well if it passes both a per-frame image check and a whole-clip motion check.

Symbol	Meaning
$\mathcal{L}_D$ , $\mathcal{L}_G$	discriminator loss, generator loss
$x \sim p_{\mathrm{data}}$	a real video clip, drawn from the data distribution
$z \sim p(z)$	a noise vector, drawn from the generator’s prior
$D(\cdot)$ , $G(\cdot)$	the discriminator’s unbounded score; the generator’s video output
$\max(0, \cdot)$	the hinge — zero penalty once a sample is past its margin

What It Enabled

The three papers describe a single axis of progress — scale — measured in resolution, clip length, and dataset size, with the underlying adversarial recipe from image-generation, Chapter 3 held constant throughout:

	VGAN, 2016	MoCoGAN, 2017	DVD-GAN, 2019
Resolution	64 × 64	64 × 64	up to 256 × 256
Clip length	32 frames	16 frames	up to 48 frames
Discriminator design	single 3D-conv discriminator	image + video discriminator	spatial + temporal discriminator
Largest training set	unlabeled internet video	~1,000–4,500 clips per dataset	Kinetics-600, ~500,000 clips
Compute	single-GPU scale	single-GPU scale	32–512 TPU replicas
Headline result	plausible one-second clips, transferable action features	independent control of content and motion	state-of-the-art FID and Inception Score at Kinetics-600 scale

None of the three models solved video generation outright, and by design their ambitions stayed modest — a second of motion, a handful of controllable datasets, one action-recognition benchmark. The adversarial line continued past this chapter in two directions. Scaling further under the same paradigm ran into the same instability and mode-collapse pressures that had already forced image-generation, Chapter 4’s three years of GAN-training fixes onto still images, now compounded by a second axis of variation; video generation’s own path to a learned world model, rather than a discriminator-judged clip, is the separate lineage covered starting at Chapter 3. The dominant paradigm shift, though, mirrored what had already started happening to images: diffusion models, introduced for images in 2020 and covered in image-generation, Chapter 8, were adapted to video beginning with Video Diffusion Models in April 2022, covered in Chapter 5 — the point at which the field’s frontier for video fidelity stopped being adversarial. The one place adversarial video generation kept a durable advantage was latency: a GAN generator is one forward pass, not a multi-step sampler, which is why lip-sync and real-time face-video work such as Wav2Lip in August 2020, covered in Chapter 13, stayed in the GAN family well after diffusion had taken over open-ended text-to-video.

Key Papers

Paper	Authors	Year	Link	Why it matters
Generating Videos with Scene Dynamics	Vondrick, Pirsiavash, and Torralba	2016	arXiv:1609.02612	First adversarial video generator; introduces the two-stream foreground/background/mask decomposition and shows discriminator features transfer to action recognition
MoCoGAN: Decomposing Motion and Content for Video Generation	Tulyakov, Liu, Yang, and Kautz	2017	arXiv:1707.04993	Factorizes the latent code into a fixed content vector and a recurrently-generated motion vector, giving independent control over subject and motion
Adversarial Video Generation on Complex Datasets	Clark, Donahue, and Simonyan	2019	arXiv:1907.06571	Scales adversarial video to Kinetics-600 via a BigGAN-derived generator and a spatial/temporal dual-discriminator split, the largest adversarial video result before diffusion took over

PM Lens

I read this lineage as a case study in when “just scale it” works and when it doesn’t. VGAN to MoCoGAN to DVD-GAN is a straight line of more compute, bigger datasets, and a smarter discriminator split — and it did produce real gains, state-of-the-art numbers on Kinetics-600 among them. But none of these models ever generated anything a product team could ship as open-ended text-to-video; the ceiling wasn’t ambition, it was that adversarial training gets harder to stabilize as you add a second axis of variation on top of images, exactly what image-generation’s Chapter 4 already had to spend three years fixing for stills alone. What did ship, and still ships today, is the one place a single forward pass beats a multi-step sampler on cost: real-time lip-sync and face-video tools, the direct descendants of the GAN-family work in this chapter, still underpin a chunk of the AI-dubbing and avatar tooling I evaluate for localization use cases, precisely because a GAN generator is cheap enough to run live where a video diffusion model is not.

Search