LoRA: The Low-rank Update
Intermediate
The problem, from first principles
LoRA (Hu et al., 2021) freezes the pretrained weight and learns only a low-rank update, the product of two thin matrices. Fine-tuning then trains a small fraction of the parameters, and the update can be merged into the weight after training.
From theory to code
Implement lora_delta(A, B, alpha, r), the scaled low-rank update.
Constraints
Bstarts at zero so training begins at the pretrained model.
Hints
Hint 1
Multiply the up-projection by the down-projection, then scale by alpha over r.
Keywords
research-paperssystemsfine-tuningadapters
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.