Compare: Full Fine-Tune vs. LoRA, on Parameter Count and Memory
The problem, from first principles
rl-alignment-lora-low-rank-adapters built LoRA's forward and backward passes, this question quantifies exactly WHY anyone would bother: how many fewer parameters LoRA actually trains compared to a full fine-tune of the same layer, and, a second, often-overlooked source of savings: how much less optimizer memory that translates into, since a frozen parameter needs no optimizer state at all.
From theory to code
Implement full_finetune_parameter_count, lora_parameter_count, optimizer_state_bytes, and compare_full_finetune_vs_lora, bundling all four numbers together for a single layer.
Constraints
full_finetune_parameter_count(in_features, out_features)returnsin_features * out_features + out_features(weight matrix plus bias).lora_parameter_count(in_features, out_features, rank)returnsrank * in_features + out_features * rank.optimizer_state_bytes(num_trainable_params)returnsnum_trainable_params * 8: Adam's fp32 momentum plus fp32 variance, 4 bytes each, PER TRAINABLE parameter only.compare_full_finetune_vs_lorabundles both parameter counts and both optimizer-memory figures, plus their ratio, into one dict.
Hints
Open one at a time. Each gives away a little more than the last.
Hint 1
Optimizer state is only ever computed for TRAINABLE parameters: a full fine-tune's optimizer memory scales with full_finetune_parameter_count, while LoRA's scales with the much smaller lora_parameter_count, since the frozen base weight needs zero bytes of optimizer state, not just zero gradient computation.
Hint 2
compare_full_finetune_vs_lora doesn't need any new math beyond calling the three other functions with the right arguments and packaging the results: the interesting part is just seeing all four numbers side by side.
Keywords
No results yet
Run your code to check it against the examples, or submit to run every test.