Training · After pretraining

LoRA

low-rank adapters beside frozen weights

here
rank 4 Wqkv × 12 blocks
trainable
147,456 0.11%
model
GPT-2 small

All steps

  1. Freeze W, learn A · B

    Fine-tuning changes a weight matrix by some ΔW. LoRA keeps W frozen and learns ΔW as a product A · B of two thin matrices, with r columns between them. B starts at zero, so training starts from the unchanged model.

    W′ = W + A · B · rank r

  2. How few weights train

    Here: rank 4 on the attention input matrix of each of GPT-2’s 12 blocks, 147,456 trainable weights, 0.11% of the model. Only those need gradients and optimizer state.

    147,456 of 137M weights

  3. A real run

    A real run: teach GPT-2 small that things sit “on the moon”, from two sentences. Both LoRA and full fine-tuning drive the loss near zero; LoRA takes more steps. On an unseen sentence, “The bird sang on the … moon” goes from 0.75% to 100%: it learned the pattern, and overdid it.

    loss 4.85 → 0.12 (LoRA) · 0.12 (full)

  4. Is a real change low-rank?

    LoRA bets that the change a fine-tune needs is low-rank. For this narrow task it is: rank 4 holds 70% of the full fine-tune’s change to one matrix. Qwen3’s broad post-training is not: rank 16 holds only 9.5% of its change to a query matrix.

    narrow tasks: low rank · broad training: not

  5. Merge it, or swap it

    After training, A · B can be added into W once, so the model runs exactly as fast as before. Or the adapters stay separate: one frozen base can serve many small adapters, one per customer or task, swapped per request.

    W′ = W + A·B, or many A·B on one W

Code

class LoRALinear(nn.Module):
    def __init__(self, base, r=4):
        self.base = base.requires_grad_(False)                          # W stays frozen
        self.A = nn.Parameter(torch.randn(base.in_features, r) * 0.02)
        self.B = nn.Parameter(torch.zeros(r, base.out_features))       # ΔW = A·B starts at 0
    def forward(self, x):
        return self.base(x) + (x @ self.A) @ self.B
# after training: merge once, no extra cost
base.weight.data += (A @ B).T

Go deeper