LoRA, a lightweight way to fine-tune AI, may need a much higher learning rate than training every weight — why?

This explores whether LoRA, which trains small add-on matrices while the original weights stay frozen, needs a different learning rate than full fine-tuning, which updates every weight. One caveat first: the retrieved notes say little about learning rates themselves. None of their summaries gives LoRA-to-full-fine-tuning learning-rate ratios or tuning curves. The Thinking Machines Lab work in this corpus is the main outside source on this question. That research is widely reported to find that LoRA's best learning rate is roughly ten times higher than full fine-tuning's, but the notes here don't record that number. What the corpus does have is the context that makes the gap make sense: when LoRA and full fine-tuning end up equivalent, and why.

The main finding is that LoRA can match full fine-tuning on both sample efficiency and final performance. The conditions are that it's applied to all weight matrices, not just the attention layers, and that the dataset is smaller than what the adapter can absorb Can LoRA match full fine-tuning performance in practice?. That reframes the learning-rate question. If the two methods can reach the same place, the hyperparameters are mostly how you correct for LoRA updating far fewer numbers. A small adapter needs bigger steps to move the model as far as full fine-tuning does. That's the intuition behind LoRA's higher optimal rate.

The surprising part is how little capacity some kinds of training need. For reinforcement learning, LoRA at rank 1, the smallest adapter possible, matches full fine-tuning. The explanation is that each RL episode delivers only about one bit of reward signal, while supervised training delivers signal on every token Why does LoRA rank 1 work for reinforcement learning?. That fits separate evidence that RL naturally changes only 5–30% of a model's parameters, and that those changes are nearly full-rank and consistent across runs Does reinforcement learning update only a small fraction of parameters?. So the right learning rate likely depends on the training signal as well as on LoRA versus full fine-tuning. A low-information signal like RL reward puts much less strain on a small adapter than dense supervised data does.

There's also a reason to prefer smaller, more contained updates beyond tuning convenience. Fine-tuning that drifts far from the base model can damage stored knowledge Can decoding-time tuning preserve knowledge better than weight fine-tuning? and reduce the model's ability to learn later tasks Does staying close to the base model preserve learning ability?. A 1.5B model trained only with LoRA matched larger fully fine-tuned RL models on reasoning, which suggests that much of post-training adjusts output format rather than adding knowledge Can small models reason well by just learning output format?. QLoRA extends the efficiency argument: it fine-tunes huge models through frozen 4-bit weights on a single GPU without losing accuracy Can we finetune huge language models on a single GPU?.

The takeaway: a higher LoRA learning rate isn't a quirk. It follows from asking a tiny adapter to make the same change a full update would. How tiny the adapter can be depends on how much information the training signal carries, and RL carries very little. If you want exact learning-rate numbers, this corpus doesn't have them yet. Start with the Thinking Machines notes and their source.