PEFT and LoRA

It is February 2019. BERT has made “pretrain then fine-tune” the default recipe in NLP, but the recipe has a hidden cost: fine-tuning copies the entire model, once per task, with a full set of gradients and optimizer state to match. A team of eight researchers proposes freezing the pretrained network almost entirely and inserting a small bottleneck module between its layers instead — the first widely cited attempt to make adaptation cheap rather than merely effective.

Where We Were

The transfer-learning recipe this book’s language-models volume traces in detail — pretrain a large network, then fine-tune all of its weights on a downstream task — treats every task as a reason to duplicate the model. For BERT-base’s 110 million parameters this was a nuisance; for GPT-2’s 1.5 billion, announced the same month as the adapters paper, it was a real storage and serving problem; by GPT-3’s 175 billion in May 2020, full fine-tuning per task had become impractical for most teams outside the largest labs.

The cost was not only disk space. This book’s chapter on optimizers shows that Adam-family optimizers keep two extra tensors — the first and second moment estimates — the same size as the parameters they are updating. Full fine-tuning a model with Adam means holding gradients and optimizer state for every one of its weights in accelerator memory at once, on top of the weights themselves. A lab that wanted to serve ten fine-tuned variants of the same base model needed either ten full copies on disk or ten full training runs’ worth of memory to produce them, with no way to share the pretrained weights across tasks at serving time.

Two questions followed from this cost. First: how much of a pretrained network actually needs to change to adapt it to a new task? Second: could the update itself be represented far more compactly than a full new copy of every weight matrix? The gap between “fine-tune everything” and “these two open questions” is where parameter-efficient fine-tuning (PEFT) begins.

Full fine-tuning stays the default until the models get too big to copy; two parameter-efficient methods arrive within two years, each paying for its savings somewhere else. PNG ↓

The Breakthrough

Adapters (Houlsby et al., 2019) answered the first question directly: insert a small bottleneck module — project down to a much smaller dimension, apply a nonlinearity, project back up, add the result back to the original activation — after every attention and feed-forward sublayer, and train only those bottlenecks while freezing the rest of the pretrained network. On GLUE, the paper reports landing “within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task” across 26 diverse text-classification tasks. The pretrained backbone stayed shared across every task; only the small adapter modules differed.

Adapters left a residual cost, though: the extra bottleneck layers sit directly in the forward pass, so every inference call pays for two additional matrix multiplications per sublayer, however small. Prefix-tuning (Li & Liang, 2021) took a different route to the same goal, borrowing an idea from prompting rather than from module insertion: keep the language model’s parameters entirely frozen, and instead learn a short sequence of continuous vectors — “virtual tokens” — prepended to the keys and values that every attention layer attends to. The paper reports learning “only 0.1% of the parameters,” with performance “comparable” to full fine-tuning in the full-data setting and better than full fine-tuning in low-data settings, tested on GPT-2 for table-to-text generation and BART for summarization. Prefix-tuning added no new layers to the forward pass, but its learned prefix consumed part of the model’s finite context window at every step, a cost that grew with the number of virtual tokens.

By mid-2021 the field had two working PEFT methods, each solving part of the problem and each paying for it somewhere else: Adapters cost inference latency; prefix-tuning cost context length. LoRA (Hu et al., 2021) removed both costs by changing where the update lives rather than how it is computed. Instead of inserting a new module into the forward pass or prepending anything to the input, LoRA represents the update to an existing weight matrix as the product of two small matrices, added in parallel to the frozen original weight. Because the update is additive and linear, it can be computed alongside the frozen matrix during training and merged directly into it afterward — at inference time there is, per the paper, “no additional inference latency” at all. LoRA’s headline numbers: on GPT-3 175B, it “reduce[s] the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times” relative to full fine-tuning with Adam, while performing “on-par or better than fine-tuning” on RoBERTa, DeBERTa, GPT-2, and GPT-3.

The update to a weight is a separate, additive path; the frozen weight and the low-rank update run in parallel, then sum, with no added inference latency. PNG ↓

LoRA still assumed the frozen base weights sat in accelerator memory at full precision. QLoRA (Dettmers et al., 2023) removed that remaining assumption by compressing the frozen base model itself: quantize it to a 4-bit format called NF4 — the bit-level mechanics of low-bit quantization are this book’s quantization chapter — apply a second round of quantization to the quantization constants themselves (“double quantization”), and manage GPU memory spikes during training with paged optimizers that page optimizer state out to CPU memory on demand. LoRA adapters are then trained on top of this frozen, quantized base in ordinary 16-bit precision. The combination, per the paper, “reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU” while preserving the task performance of full 16-bit fine-tuning; the resulting Guanaco model family, fine-tuned in this way from LLaMA, reached “99.3% of the performance level of ChatGPT” on the Vicuna benchmark after “only requiring 24 hours of finetuning on a single GPU.”

Freezing the base at 4-bit is what fits a 65B model on one GPU; only the LoRA adapters ever receive a gradient. PNG ↓

The Mathematics

Adapters insert a bottleneck between a frozen sublayer’s output and the residual stream. For an activation $h \in \mathbb{R}^d$ :

h' = h + f\!\left(h \, W_{\text{down}}\right) W_{\text{up}}

$h$ is the activation coming out of a frozen attention or feed-forward sublayer, and $h'$ is what gets passed on to the rest of the network. $W_{\text{down}} \in \mathbb{R}^{d \times m}$ projects that activation down into a much smaller bottleneck dimension $m$ , a nonlinearity $f$ is applied, and $W_{\text{up}} \in \mathbb{R}^{m \times d}$ projects it back up to the original size $d$ before adding it back to $h$ . Because $m \ll d$ , the two adapter matrices together hold far fewer parameters than the sublayer they sit next to — this is the “3.6% parameters per task” the paper reports — and because they are trained while everything else is frozen, one pretrained backbone can support many task-specific adapter sets without ever being copied.

Symbol	Meaning
$h$	Activation entering the adapter, from a frozen sublayer
$h'$	Activation leaving the adapter, passed to the residual stream
$W_{\text{down}}$	Down-projection matrix, $d \times m$
$W_{\text{up}}$	Up-projection matrix, $m \times d$
$f$	Nonlinearity inside the bottleneck
$d$	Original hidden dimension
$m$	Bottleneck dimension, $m \ll d$

Prefix-tuning leaves every weight matrix untouched and instead prepends learned vectors to the keys and values an attention layer sees. For a layer with query $Q$ , key $K$ , and value $V$ , and a learned prefix of key/value vectors $P_k, P_v$ of length $l$ :

\mathrm{Attn}\left(Q, [P_k; K], [P_v; V]\right) = \mathrm{softmax}\!\left(\frac{Q \, [P_k; K]^{\top}}{\sqrt{d_k}}\right) [P_v; V]

Symbol	Meaning
$Q, K, V$	Query, key, and value matrices from the frozen attention layer
$P_k, P_v$	Learned prefix key and value vectors, the only trainable parameters
$l$	Prefix length, in virtual tokens
$d_k$	Key dimension, used to scale the dot product

LoRA replaces both the extra module and the extra context tokens with a low-rank update added directly to an existing weight matrix. For a frozen weight $W_0 \in \mathbb{R}^{d \times k}$ and input $x$ :

h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} B A x

$W_0$ never changes during training — it is the pretrained weight, frozen exactly as in Adapters and prefix-tuning. The update $\Delta W$ is decomposed into two small matrices, $A \in \mathbb{R}^{r \times k}$ and $B \in \mathbb{R}^{d \times r}$ , with rank $r$ far smaller than either $d$ or $k$ — the paper’s rank ablations on GPT-3 show $r = 1$ already sufficient for adapting the query and value projections on several tasks. $A$ is initialized from a random Gaussian and $B$ is initialized to all zeros, so $\Delta W = BA$ is exactly zero at the start of training and the model behaves identically to the frozen pretrained network until gradients start moving it. The scaling factor $\alpha/r$ keeps the update’s magnitude comparable across different choices of rank. Because $\Delta W x$ is computed by two small matrix multiplications added to the frozen path, $B$ and $A$ can be folded directly into $W_0$ after training — $W_0 + BA$ becomes a single ordinary weight matrix — which is why LoRA adds no inference latency, unlike an Adapter’s extra forward-pass layers.

Symbol	Meaning
$W_0$	Frozen pretrained weight matrix, $d \times k$
$x$	Input activation to the layer
$\Delta W$	The trained update, represented as a low-rank product
$A, B$	Low-rank factor matrices, $r \times k$ and $d \times r$
$r$	Rank of the update, $r \ll \min(d, k)$
$\alpha$	Scaling constant for the update magnitude
$d, k$	Output and input dimensions of the original weight matrix

QLoRA changes none of this decomposition; it changes what $W_0$ costs to hold in memory. The total training memory is approximately:

M_{\text{total}} \approx \frac{N \cdot b_{\text{base}}}{8} + \frac{N_{\text{LoRA}} \cdot b_{\text{adapter}}}{8} (1 + s)

$N$ is the number of parameters in the frozen base model and $b_{\text{base}}$ is the number of bits used to store each one — 4 for QLoRA’s NF4 format, versus 16 or 32 for an ordinary fine-tune, and the byte-level mechanics of that 4-bit format belong to this book’s quantization chapter. Because $W_0$ is frozen, it needs no gradient and no optimizer state at all — the $(1+s)$ optimizer-state multiplier from Adam or AdamW applies only to the second term, the tiny LoRA adapter parameters $N_{\text{LoRA}}$ , stored at a higher precision $b_{\text{adapter}}$ (16 bits) so training stays numerically stable. Because $N_{\text{LoRA}} \ll N$ , that second term is small in absolute terms even with Adam’s $2\times$ state overhead attached to it, so shrinking $b_{\text{base}}$ from 16 bits to 4 bits shrinks the dominant first term by roughly $4\times$ — the arithmetic behind fitting a 65-billion-parameter model onto a single 48 GB GPU.

Symbol	Meaning
$M_{\text{total}}$	Approximate total memory for weights and adapter optimizer state
$N$	Number of parameters in the frozen base model
$b_{\text{base}}$	Bits per base parameter, 4 for NF4
$N_{\text{LoRA}}$	Number of trainable LoRA adapter parameters
$b_{\text{adapter}}$	Bits per adapter parameter, typically 16
$s$	Optimizer-state multiplier, 2 for Adam’s two moment tensors

What It Enabled

The practical effect of this four-paper progression was to turn “fine-tune a large pretrained model” from a full-cluster job into something that fits, for the smaller end of the model range, on a single accelerator. QLoRA’s own headline result — a 65-billion-parameter model fine-tuned on one 48 GB GPU, reaching 99.3% of ChatGPT’s performance on the Vicuna benchmark after 24 hours of training — is the clearest evidence of how far the gap between “adapt a frontier-scale model” and “adapt it on hardware a single researcher can access” had closed.

Method	Year	Trainable parameters	Added inference cost	Frozen-weight precision
Full fine-tuning	—	100%	none	full (16/32-bit)
Adapters	2019	~3.6% per task	extra forward-pass layers	full
Prefix-tuning	2021	~0.1%	consumes context window	full
LoRA	2021	far below 1%, merged into $W_0$	none, merges into frozen weight	full
QLoRA	2023	same as LoRA	none	4-bit NF4

LoRA’s mergeable, additive structure had a second consequence beyond raw memory savings: because $BA$ is a small, self-contained update, it can be stored, shipped, and swapped independently of the multi-gigabyte base model it modifies. That property underlies the open-weights fine-tuning ecosystem this book’s sister volume covers around Llama and the open-weight explosion and the practice of instruction-tuning open models on commodity hardware — a LoRA adapter for a 7B-parameter model is typically megabytes, not gigabytes, because it stores only $A$ and $B$ , not a second copy of $W_0$ . The same additive structure crossed over into diffusion image models almost immediately, where it underlies the subject- and style-personalization techniques this book’s sister volume covers in personalization and LoRA — the math is identical, only the frozen base model differs. Serving many such adapters against one shared frozen base, in turn, is one of the problems this book’s inference-engines chapter returns to when it covers how a single deployment can multiplex requests across dozens of fine-tuned variants without holding dozens of full model copies in memory.

Two 2019 fixes converge on LoRA, which removes both their costs, and QLoRA then removes the memory floor underneath it. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Parameter-Efficient Transfer Learning for NLP	Houlsby et al.	2019	arXiv:1902.00751	Introduced bottleneck adapter modules; within 0.4% of full fine-tuning at 3.6% of the parameters per task
Prefix-Tuning: Optimizing Continuous Prompts for Generation	Li & Liang	2021	arXiv:2101.00190	Learned continuous prefixes prepended to attention keys/values; 0.1% of parameters, no new layers
LoRA: Low-Rank Adaptation of Large Language Models	Hu et al.	2021	arXiv:2106.09685	Low-rank additive update merged into frozen weights; 10,000x fewer trainable params, no inference latency
QLoRA: Efficient Finetuning of Quantized LLMs	Dettmers et al.	2023	arXiv:2305.14314	4-bit NF4 frozen base plus LoRA adapters; 65B-parameter fine-tuning on a single 48GB GPU

PM Lens

The question I actually get asked is not “what is LoRA” — it is “why can our team fine-tune a 70B open-weight model on the GPUs we already have, when the vendor’s own training guide assumes a cluster.” The honest answer is that we are not touching 70 billion parameters at all; we are training a handful of low-rank matrices that get merged back in afterward, on top of a base model that QLoRA-style 4-bit loading has already shrunk to fit. That distinction matters in two concrete ways I keep having to explain: first, a LoRA adapter is small enough to email — megabytes, not the tens of gigabytes of the base model — which is why a single deployment can serve a dozen fine-tuned variants of the same open model side by side. Second, “we fine-tuned it” and “we retrained it” are not the same claim, and a roadmap slide that blurs them will eventually meet an engineer asking which weights actually moved.

Search