The Third Dimension

It is March 2020. Multi-view stereo and structure-from-motion pipelines have spent two decades turning a handful of photographs into an explicit point cloud or mesh — faithful to the geometry the cameras could triangulate, blind to whatever the cameras never quite captured. A six-author paper proposes something stranger: throw the mesh away, and train a small neural network to answer, for any point in space and any viewing direction, how much light arrives there. Over the following three and a half years that idea for representing a scene is stripped of its need for real photographs, then stripped of its need for a neural network at all, arriving at an explicit, real-time representation that puts a version of it on an ordinary phone.

Where We Were

By 2020, view synthesis — the problem of generating a photograph from a camera position no photograph was actually taken from — had two working but incompatible families of solutions. The first, built on decades of multi-view stereo and structure-from-motion, triangulates matching features across a set of calibrated photographs into an explicit 3D representation: a point cloud, a voxel grid, or a textured mesh. That representation is discrete by construction — a fixed set of points or voxels, decided once at reconstruction time — and it struggles with anything a static, opaque surface model does not capture well: reflections, translucency, thin structures like foliage, or fine geometric detail smaller than the reconstruction’s resolution. The second family, image-based rendering, skips reconstructing 3D geometry at all and instead blends or warps nearby captured photographs directly into a novel view. That sidesteps the discretization problem but pays for it with density: it needs many closely spaced photographs to interpolate between, and it extrapolates badly to viewpoints the capture did not anticipate.

Neither family treated the scene itself as something differentiable — a function whose parameters could simply be optimized by gradient descent against a rendering loss, the way every other generative model in this book’s lineage already was. That gap is what a continuous, per-scene function optimized directly from posed photographs could close, if volume rendering could be made differentiable enough to backpropagate through.

Each step removes a requirement the last one still needed: first real photographs, then the network itself. PNG ↓

The Breakthrough

NeRF, representing scenes as neural radiance fields for view synthesis (Mildenhall et al., 2020), answers the differentiability question directly: represent a scene as a fully connected, non-convolutional network whose input is a single continuous 5D coordinate — a 3D spatial location and a 2D viewing direction — and whose output is a volume density and a view-dependent emitted color at that point. To render a pixel, march a ray from the camera through the volume, query the network at points sampled along the ray, and composite the resulting densities and colors with classical volume rendering, which the paper notes is “naturally differentiable” — so the only supervision the whole pipeline needs is a set of photographs with known camera poses, the same calibration a structure-from-motion pipeline already computes as a byproduct. Each output pixel gets there by comparing the network’s rendered color against the corresponding pixel in a real captured photograph and backpropagating the difference through the volume-rendering integral and into the network weights — one network, trained from scratch, per scene.

Two details make that MLP good enough to compete with classical reconstruction rather than merely a demonstration. Each input coordinate passes through a fixed positional encoding before the network’s first layer — mapping a scalar $p$ to $\gamma(p) = (\sin(2^0\pi p), \cos(2^0\pi p), \ldots, \sin(2^{L-1}\pi p), \cos(2^{L-1}\pi p))$ , applied with $L=10$ to each of the three position coordinates and $L=4$ to each of the two direction components — which lets a network that is otherwise biased toward smooth, low-frequency functions represent the sharp edges and fine texture real scenes have. And the architecture routes viewing direction in only after density is decided: the position alone, encoded and passed through eight fully connected layers of 256 channels, produces a density and a 256-dimensional feature; only that feature, concatenated with the encoded viewing direction, passes through one more 128-channel layer to produce the final view-dependent color — so a surface’s density is a property of where it is, and only its appearance is allowed to change with how it is viewed, matching how real materials behave.

Every step is differentiable, so a photograph is the only supervision the whole pipeline needs; the gradient never has to leave the network. PNG ↓

Optimizing one of these networks to convergence — typically 100,000 to 300,000 iterations on a single NVIDIA V100 GPU — takes about one to two days, and rendering a single novel frame afterward, at 256 network queries per ray, takes roughly 30 seconds on the same V100. Both numbers matter for what happens next: a representation this slow to train and this slow to render was never going to reach real time by refining the same recipe. It first had to be pushed in the opposite direction, toward needing less input, before it was pushed toward speed.

That opposite direction arrived two and a half years later. DreamFusion, text-to-3D using 2D diffusion (Poole et al., 2022), asks whether a NeRF can be optimized with no photographs of the object at all — only a text prompt and a pretrained 2D text-to-image diffusion model standing in for what “photographs of that object” would even look like. The paper’s own framing is a DeepDream-like procedure: render the NeRF from a randomly sampled camera and light position, add noise to that rendering, and ask a frozen diffusion model — Imagen’s 64×64 base model, with no modifications — how far its predicted noise, conditioned on the text prompt, differs from the noise actually added. That mismatch, backpropagated through the renderer rather than through the diffusion model’s own network, is the entire training signal. No 3D data, no multi-view captures, and no gradient ever has to pass through the diffusion model’s weights, which the paper shows can be skipped without hurting the result and without the cost of differentiating through it.

Fifteen thousand iterations of the same loop, and only the renderer's weights ever move; the diffusion model stays frozen throughout. PNG ↓

The optimized representation itself is a mip-NeRF 360 MLP — five ResNet blocks of 128 hidden units with Swish activations and layer normalization between them — trained for 15,000 iterations, about 1.5 hours, on a four-chip TPUv4 machine, at a rendering resolution of only 64×64 to match the diffusion prior’s own resolution. One number stands out against everything 2D diffusion sampling had established: DreamFusion turns its classifier-free guidance weight up to $\omega=100$ , far past the $\omega \in [5,30]$ range 2D image samplers use, and the paper reports this works without the visible artifacts a 2D sampler shows at similarly extreme guidance — because the NeRF’s rendered colors are already bounded by a sigmoid activation, where a 2D pixel sampler needs post-hoc clipping to stay in range. (Classifier-free guidance itself is introduced in full in Guidance; DreamFusion’s contribution here is the number, not a new mechanism.)

DreamFusion proved a NeRF could be conjured from language alone, but it inherited NeRF’s other problem in full: an MLP is still what gets optimized, and an MLP is still slow to render. By mid-2023, a wave of methods had chased that specific weakness — hash-grid feature lookups instead of a pure MLP, explicit voxel grids with far less network left to evaluate — trading some quality for large speedups, but none reaching real-time display at the resolutions unbounded, complete scenes actually need. 3D Gaussian splatting for real-time radiance field rendering (Kerbl et al., 2023) answers by discarding the network entirely. The scene becomes an explicit cloud of a few million 3D Gaussians — each one a position, an anisotropic covariance describing its size and orientation, an opacity, and a color — initialized from the same sparse point cloud a structure-from-motion calibration step already produces before any optimization begins. Training alternates gradient steps on those per-Gaussian parameters with an interleaved density-control pass that clones or splits Gaussians in under-reconstructed regions and prunes ones that have become too transparent or too large to matter, letting the number and placement of primitives adapt rather than staying fixed from initialization. What replaces the ray-marching MLP is a custom, tile-based, visibility-aware rasterizer, written to run efficiently on the GPU rather than assembled from generic autodiff operations, that projects each Gaussian onto the image plane and composites the overlapping ones in depth order — the paper’s stated result is training times competitive with the fastest prior methods and, for the first time on unbounded, complete scenes at 1080p, real-time display at 30 frames per second or more.

The Mathematics

NeRF’s rendering step turns a set of sampled densities and colors along one camera ray into a single pixel color by discretizing the volume rendering integral into a sum over $N$ sample points:

\hat{C}(\mathbf{r}) = \sum_{i=1}^{N} T_i \left(1 - e^{-\sigma_i \delta_i}\right) c_i, \qquad T_i = \exp\left(-\sum_{j=1}^{i-1} \sigma_j \delta_j\right)

$\mathbf{r}$ is the camera ray being rendered, sampled at $N$ points along its length. At each sample $i$ , the network has already produced a density $\sigma_i$ and a color $c_i$ ; $\delta_i$ is the distance to the next sample. $T_i$ is the accumulated transmittance — how much light survives from the camera to sample $i$ without being absorbed by any denser material in between — and $1 - e^{-\sigma_i\delta_i}$ is the chance that sample itself absorbs and re-emits light. Multiplying transmittance, local absorption, and local color, then summing over every sample, is a discrete approximation of the classical volume rendering integral. What this buys you: because every term in the sum is a smooth, differentiable function of $\sigma_i$ and $c_i$ , the pixel color $\hat{C}(\mathbf{r})$ can be compared against a real photograph and the error backpropagated straight through the rendering equation into the network weights — the property that makes training from photographs alone possible at all.

Symbol	Meaning
$\hat{C}(\mathbf{r})$	the rendered color for camera ray $\mathbf{r}$
$N$	number of points sampled along the ray
$\sigma_i$	volume density predicted at sample $i$
$c_i$	view-dependent color predicted at sample $i$
$\delta_i$	distance between sample $i$ and the next sample
$T_i$	accumulated transmittance from the camera to sample $i$

DreamFusion’s score distillation sampling (SDS) loss replaces NeRF’s photometric loss with a gradient computed entirely from a frozen 2D diffusion model’s noise prediction:

\nabla_\theta \mathcal{L}_{\mathrm{SDS}} = \mathbb{E}_{t,\epsilon}\left[w(t)\left(\hat\epsilon_\phi(\mathbf{z}_t; y, t) - \epsilon\right)\frac{\partial \mathbf{x}}{\partial \theta}\right], \qquad \mathbf{x} = g(\theta)

$\theta$ is the NeRF’s own weights, the only thing being optimized. $\mathbf{x} = g(\theta)$ is a rendered image from a randomly sampled camera view — the NeRF’s output, treated here as a “parametric image generator.” That rendering is noised for $t$ steps to produce $\mathbf{z}_t$ , the same forward process Diffusion Foundations defines in full; $\hat\epsilon_\phi(\mathbf{z}_t; y, t)$ is the frozen, text-conditioned diffusion model’s guess at the noise that was added, and $\epsilon$ is the noise that actually was added. $w(t)$ is a timestep-dependent weighting term. What this buys you: the difference between predicted and true noise says how much the rendering looks like something the diffusion model would have generated for prompt $y$ ; multiplying by $\partial \mathbf{x}/\partial\theta$ carries that signal back through the renderer into the NeRF’s weights, without ever needing to backpropagate through the diffusion model itself — the term that would normally require differentiating the diffusion network’s every layer never appears in this gradient.

Symbol	Meaning
$\theta$	the NeRF’s parameters, the only thing optimized
$\mathbf{x} = g(\theta)$	a rendered view of the NeRF at the current parameters
$\mathbf{z}_t$	that rendering after $t$ steps of forward noising
$y$	the text prompt conditioning the diffusion model
$\hat\epsilon_\phi$	the frozen diffusion model’s predicted noise
$\epsilon$	the noise actually added during the forward process
$w(t)$	a timestep-dependent loss weight

A 3D Gaussian is a smooth density function centered at a mean position, with its shape controlled by a covariance matrix built from a separate scale and rotation, so the two stay independently optimizable:

G(\mathbf{x}) = \exp\left(-\frac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1} (\mathbf{x}-\mu)\right), \qquad \Sigma = R S S^\top R^\top

$G(\mathbf{x})$ is the Gaussian’s value at any 3D point $\mathbf{x}$ — highest at the center $\mu$ , falling off smoothly with distance, the same bell-curve density a Gaussian distribution has in one dimension, generalized to three. $\Sigma$ , the covariance matrix, controls the ellipsoid’s size and orientation; factoring it as $RSS^\top R^\top$ , a scaling matrix $S$ and a rotation matrix $R$ built from a learned quaternion, keeps every optimization step producing a valid, positive-definite covariance automatically, rather than requiring the optimizer to respect that constraint on the raw matrix entries. What this buys you: a handful of numbers per primitive — a position, a 3-vector scale, a 4-number rotation, an opacity, and a color — replace the millions of weights a NeRF’s MLP needed to represent the same scene, and every one of those numbers is directly, individually adjustable by gradient descent.

Symbol	Meaning
$G(\mathbf{x})$	the Gaussian’s density value at 3D point $\mathbf{x}$
$\mathbf{x}$	a 3D point at which the Gaussian is evaluated
$\mu$	the Gaussian’s center, a learned 3D position
$\Sigma$	the covariance matrix controlling size and orientation
$R$	a rotation matrix, built from a learned quaternion
$S$	a diagonal scaling matrix, the learned per-axis extents

Rendering composites the Gaussians that project onto a given pixel, sorted by depth, with the same transmittance-and-absorption structure NeRF’s volume rendering already established:

C = \sum_{i=1}^{N} c_i\, \alpha_i \prod_{j=1}^{i-1} (1-\alpha_j)

$C$ is the final composited pixel color. $N$ is the number of Gaussians overlapping this pixel, ordered front to back; $c_i$ is Gaussian $i$ ‘s color and $\alpha_i$ its opacity at this pixel, computed from $G(\mathbf{x})$ and its own opacity parameter. The product $\prod_{j=1}^{i-1}(1-\alpha_j)$ is how much of the pixel is still “unclaimed” by the time Gaussian $i$ is reached — every Gaussian closer to the camera has first claim on the light. What this buys you is the same differentiable alpha-blending NeRF’s transmittance term $T_i$ provides, computed over a short, explicit, depth-sorted list of Gaussians instead of hundreds of network queries per ray — the change that turns rendering from a 30-second MLP evaluation into a tile-based rasterization pass fast enough to run at 30 frames per second or more.

Symbol	Meaning
$C$	the composited color at this pixel
$N$	number of Gaussians overlapping this pixel
$c_i$	color contributed by Gaussian $i$
$\alpha_i$	opacity contributed by Gaussian $i$ at this pixel
$j, i$	depth-sorted indices, nearest to camera first

3D Gaussian splatting’s real-time result is a systems achievement as much as a representational one: the paper reports training a scene in 6 minutes 25 seconds at 7,000 iterations or 41 minutes 33 seconds at 30,000 iterations on an NVIDIA A6000, and rendering the result at 134 frames per second at 1080p — against 48 hours to train and 0.06 frames per second to render a comparable Mip-NeRF360 model on the same class of hardware. That gap opens up because the rasterizer is hand-written CUDA rather than a generic sequence of autodiff’d tensor operations, the same move that first turned graphics chips into general-purpose parallel processors in the other direction, covered in GPUs Learn to Think: a GPU built to rasterize triangles is, here, rasterizing Gaussians instead, with a purpose-built kernel back at the center of the pipeline rather than a matrix multiply.

What It Enabled

	Explicit mesh or point cloud, multi-view stereo	NeRF, implicit MLP, 2020	3D Gaussian splatting, explicit primitives, 2023
Representation	Fixed set of points or polygons, decided at reconstruction time	Continuous function, one MLP per scene	Explicit but continuously optimizable primitives, one set per scene
Optimizable by gradient descent	No	Yes, but through hundreds of network queries per pixel	Yes, through a fast differentiable rasterizer
View-dependent effects (reflections, translucency)	Poorly, if at all	Yes, learned per point	Yes, via per-Gaussian, view-dependent color
Typical training time	Not learned; computed directly from calibrated photos	1–2 days on a single V100	Minutes to tens of minutes on a single GPU
Typical rendering rate	Fast, but fixed once reconstructed	Roughly 30 seconds per frame on a V100	30+ fps at 1080p, real time

Reported on the Mip-NeRF360 benchmark dataset at each method’s best-quality setting, the same comparison the 3D Gaussian splatting paper ran against its own predecessors:

Method	PSNR	Train time	Render rate
Plenoxels	23.08	25m49s	6.79 fps
Instant-NGP, big model	25.59	7m30s	9.43 fps
Mip-NeRF360	27.69	48h	0.06 fps
3D Gaussian splatting, 30K iterations	27.21	41m33s	134 fps

The pattern across those four rows is the chapter’s real argument: the highest-quality prior method (Mip-NeRF360 itself) was three orders of magnitude too slow to render in real time, and the fastest prior methods (Plenoxels, Instant-NGP) bought their speed with a visible quality cost. 3D Gaussian splatting is the first entry in the table within half a PSNR point of the best quality on offer while rendering over a thousand times faster than that same best-quality method — not a small improvement traded for a small one, but a representation that stopped forcing the tradeoff.

None of the three papers in this chapter is itself a video-generation method, but each answers a question the rest of this book keeps needing an answer to: what does a scene look like from a position or a moment nobody photographed? NeRF and its explicit successor make that question answerable for a static, real scene from real photographs; DreamFusion makes it answerable for an imagined object from nothing but a caption, generalizing the score-distillation idea beyond the 3D case it was invented for. That is the same underlying question — “what is on the other side of this cut, this camera move, this occlusion” — that the learned-simulator framing of Predicting the Next Frame first asked for whole environments rather than single objects, and that the next two chapters push toward video models that behave like interactive worlds rather than fixed clips: whether a generative video model can be steered frame by frame like a game engine (Video Models Become Game Engines), and whether a world model can serve as a general-purpose foundation model rather than a one-off simulator (World Models as Foundation Models). A NeRF’s optimized weights are also a useful point of comparison for what “latent” means elsewhere in this book: they are a compressed, continuous representation of exactly one scene, learned from scratch every time, in contrast to the single shared, amortized latent space a VAE or a latent-diffusion model like Stable Diffusion learns once across an entire dataset (Latent Diffusion and Stable Diffusion) — the same idea of operating in a learned continuous space, applied at opposite ends of the “one model, one scene” versus “one model, every scene” spectrum.

The explicit, real time representation is what later chapters build worlds on top of, not the implicit MLP that started the lineage. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis	Mildenhall et al.	2020	arXiv:2003.08934	Represented a 3D scene as a continuous, differentiable MLP optimized directly from posed photographs, replacing explicit reconstruction with a rendering loss
DreamFusion: Text-to-3D using 2D Diffusion	Poole et al.	2022	arXiv:2209.14988	Introduced score distillation sampling, optimizing a NeRF from a text prompt alone using a frozen 2D diffusion model as the only supervision, with no 3D training data
3D Gaussian Splatting for Real-Time Radiance Field Rendering	Kerbl et al.	2023	arXiv:2308.04079	Replaced the MLP with an explicit cloud of anisotropic 3D Gaussians and a custom rasterizer, reaching real-time rendering at 1080p for the first time on unbounded scenes

PM Lens

The number I keep coming back to from this chapter is the lag between a paper and a product. NeRF shipped in March 2020 and took roughly four years to show up as a smooth-enough consumer feature. 3D Gaussian splatting shipped on arXiv on August 8, 2023, and Polycam — a phone-based 3D capture app — added Gaussian splat export within about seven weeks, folding it into a Pro tier that runs $12.99 to $26.99 a month depending on plan and billing cycle. Same underlying research pattern, same three-author-team-to-open-benchmark pipeline, wildly different time-to-shelf, and the difference is almost entirely the training-time and rendering-speed numbers in this chapter’s own tables: a representation that trains in tens of minutes and renders in a browser is a feature; one that takes two days per scene on a datacenter GPU is a research demo. When a paper crosses my desk claiming a “10x” or “100x” speedup on some existing pipeline, this is the chapter I think of before I believe it changes the product roadmap — check whether the speedup lands the total cost inside what a phone, or a free tier, can actually run.

Search