LoRA vs full fine-tuning
How much of a model needs to learn?
A direct PyTorch implementation of low-rank adaptation, compared with full fine-tuning on EuroSAT RGB. A focused experiment in parameter efficiency and the mechanics beneath adapter libraries.
View sourceThe problem
Full fine-tuning updates every model parameter. That can be an expensive way to adapt a pretrained transformer to a smaller downstream task. This experiment asks how a small trainable update compares with full adaptation under a shared training setup.
Freeze the backbone and replace selected query/value projections with a custom LoRALinear layer. Train low-rank A and B matrices plus the classifier, then compare ranks 16, 32 and 64 against a full-fine-tuning baseline. Typed configuration and separate data, model and training modules keep the runs reproducible.
An individual deep-learning experiment using google/vit-base-patch16-224 on ten land-use classes. The same stratified split is reused across the four configurations.
The mechanism
A direct PyTorch implementation.
Wnew = W + (α / r) BA
The frozen weight keeps its pretrained representation. B and A learn a low-rank correction scaled by α/r. The implementation adapts query/value projections in the final three transformer blocks and also trains the classifier.
Engineering challenges
The custom layer retains the pretrained weight, initializes A with Kaiming uniform and B with zeros, and adds the scaled low-rank output in forward(). This makes the starting update zero and the adaptation mechanism directly inspectable.
The default data path creates one seed-42 stratified 80/20 split reused across all configurations. Five epochs and batch size 16 define the reported run budget. Configuration-specific checkpoints and result records keep experiments separate.
Model construction supports both encoder.layer attention query/value paths and vit.layers q_proj/v_proj paths. The target remains the query and value projections in the final three transformer blocks.
Implementation & trade-offs
LoRA targets Q/V projections in blocks 9, 10 and 11. The classifier also remains trainable, and reported trainable parameter counts include it.
Trade-offThis keeps the adapter small. Applying LoRA to embeddings, MLPs or more layers could change the tradeoff and was not ablated in the reported comparison.
Ranks 16, 32 and 64 vary the capacity of the low-rank update. All three retain less than 1% trainable parameters in the reported table.
Trade-offThe validation scores are not monotonic with rank. A single run cannot distinguish seed variation from a reliable difference between configurations.
Report validation accuracy, trainable counts and the shared experimental setup alongside the original repository figure.
Trade-offThe dataset and single seed limit generalization. Trainable-parameter savings alone do not establish proportional wall-clock speed, memory or inference-latency gains.
Results & evaluation
Historical repository results.
Methods and limitations travel with the numbers.
Select a run below to inspect its trade-off. The horizontal axis uses a logarithmic scale.
| Select a run | Trainable | Accuracy |
|---|---|---|
| 100.00% | 98.74% | |
| 0.18% | 97.98% | |
| 0.35% | 97.94% | |
| 0.69% | 98.13% |
5 epochs · batch size 16 · seed 42 · stratified 80/20 train/validation split. Counts include the trainable classifier. Single-run values have no confidence intervals.
Limits & lessons
Parameter efficiency is a measurable engineering tradeoff. Implementing the update directly makes it possible to explain which weights change, count them correctly and separate an encouraging point estimate from a reproducible empirical conclusion.
Inspect the work
Selected implementation and documentation reviewed for this case study.
