LoRA, the cheap way to fine-tune AI models, can suddenly fall behind full training — what decides when that happens?

This explores whether there's a dataset size past which LoRA stops keeping up with full fine-tuning. The short answer: the corpus doesn't give a single number, and it suggests that asking for one is a slightly mistaken question. LoRA doesn't break down at a fixed number of examples. It breaks down when the amount of *information* you're trying to put into the model is more than the adapter can hold. Thinking Machines Lab found that LoRA matches full fine-tuning on instruction and reasoning datasets as long as two things are true: the adapter is applied to all weight matrices, not just the attention layers, and the dataset stays below LoRA's capacity threshold Can LoRA match full fine-tuning performance in practice?. That threshold moves depending on the adapter's rank (how big the add-on matrices are) and on what kind of learning signal you're feeding it.

The clearest evidence that capacity is about information, not example count, comes from reinforcement learning. The same lab found that LoRA at rank 1, the smallest adapter possible, matches full fine-tuning for RL Why does LoRA rank 1 work for reinforcement learning?. Their explanation is that an RL episode gives roughly one bit of feedback (did it work or not), while supervised fine-tuning gives feedback on every single token. So RL takes in about 1000 times less information per token, and a tiny adapter can absorb a very large number of RL episodes before it fills up. Supervised training on the same number of examples would saturate it much sooner. 'How many examples before LoRA degrades' therefore depends heavily on whether you're doing SFT or RL.

This lines up with a second thread in the corpus: much of fine-tuning doesn't teach new knowledge at all. A 1.5B model trained only with LoRA matched larger fully fine-tuned RL models on reasoning, which suggests the training mostly taught it how to *organize its output*, not new facts Can small models reason well by just learning output format?. LIMA showed something similar for alignment: 1,000 carefully chosen examples got close to models trained on far more data, because post-training mostly unlocks abilities the base model already has Can careful curation replace massive alignment datasets?. If your goal is format, style, or bringing out existing skills, you may never get near LoRA's ceiling. If your goal is to load in a lot of new domain knowledge, you will reach it sooner.

The practical twist is that hitting the capacity ceiling is often not your biggest problem. Across more than 1,000 QLoRA runs, a 9k-example dataset beat a 450k-example one on chatbot benchmarks, because the smaller dataset matched the target task better Does dataset size matter more than dataset fit?. QLoRA also showed that adapters trained on top of 4-bit compressed models can match full-precision accuracy Can we finetune huge language models on a single GPU?, so the low-cost setup isn't what limits quality. Before worrying about when LoRA runs out of room, it's usually worth asking whether more data is helping at all. The corpus doesn't contain a measured curve showing exactly where SFT performance drops off at a given rank. If you need that number, you'd have to measure it for your own task, but the frame of information versus capacity tells you what to vary: rank, which layers get adapters, and how much new knowledge the data really carries.