LoRA (Low-Rank Adaptation)

Each linear layer in a transformer multiplies its input by a weight matrix W₀ of size d × k. Full fine-tuning changes every entry of W₀. LoRA (Hu et al., 2021) keeps W₀ frozen and learns the change as the product of two thin matrices, ΔW = BA, where B is d × r, A is r × k, and the rank r is much smaller than d or k.

The layer's forward pass becomes h = W₀x + BAx. A starts as random Gaussian values and B starts at zero, so at the first step the adapter adds nothing and the model behaves exactly like the base model. The update is scaled by α/r, which the authors note is roughly equivalent to adjusting the learning rate.

Only A and B receive gradients, so optimizer state is kept for a tiny fraction of the parameters. For GPT-3 175B the paper reports 10,000 times fewer trainable parameters and a 3 times lower GPU memory requirement than fine-tuning with Adam, with quality on par or better on the models it tested. The original experiments applied LoRA to the attention query and value projections (Wq and Wv).

At deployment the update can be merged once, W = W₀ + BA, and served as ordinary weights, so there is no extra inference latency, unlike adapter layers that add computation. Alternatively the small A and B matrices are kept separate and swapped per task on top of one shared base model.

Worked numbers for one 4096 × 4096 projection matrix with rank r = 8, then the forward pass in a few lines of PyTorch.

Full matrix W0:   4096 × 4096 = 16,777,216 parameters
LoRA B (d × r):   4096 × 8    =     32,768
LoRA A (r × k):      8 × 4096 =     32,768
Trainable total:                    65,536  (0.39% of the matrix)

import torch

d, k, r, alpha = 4096, 4096, 8, 16
W0 = torch.randn(d, k)                       # pretrained weight, frozen
A = (torch.randn(r, k) * 0.01).requires_grad_()  # trainable, Gaussian init
B = torch.zeros(d, r, requires_grad=True)        # trainable, zero init

def lora_forward(x):                         # x has shape (k,)
    return W0 @ x + (alpha / r) * (B @ (A @ x))

Because B is zero at the start, lora_forward(x) equals W0 @ x until training moves B. After training, W0 + (alpha / r) * B @ A can be computed once and the adapter disappears from the serving path.

What is the difference between LoRA and full fine-tuning?

Both adapt a pretrained model to a task. Full fine-tuning updates every weight and produces a complete new copy of the model. LoRA freezes the original weights and trains only a low-rank update, so the result is a small adapter file that sits on top of the unchanged base model. The LoRA paper reports quality on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3, but that is a result on those benchmarks, not a guarantee for every task.

How is QLoRA different from LoRA?

QLoRA keeps the LoRA adapters but stores the frozen base model in 4-bit precision and backpropagates through it into the adapters. It adds a 4-bit NormalFloat data type, double quantization of the quantization constants, and paged optimizers to handle memory spikes. The QLoRA paper reports fine-tuning a 65B-parameter model on a single 48 GB GPU while preserving 16-bit fine-tuning task performance.

Is LoRA related to LoRa radio or LoRaWAN?

No. LoRa (Long Range) is a low-power wireless radio technology used by IoT devices, and LoRaWAN is a network protocol built on it. LoRA in machine learning stands for Low-Rank Adaptation. The two share only the spelling.