Does a cheap fine-tuning shortcut work equally well on any model or task, or does something else decide if it keeps up?

This explores whether LoRA, the cheap fine-tuning method that trains only small add-on weights, works equally well everywhere or whether how well it does depends on the model's design or the kind of task you're teaching it. The corpus doesn't have direct comparisons of LoRA across different model families or across fields like medicine and law. What it does have points somewhere more interesting: whether LoRA keeps up with full fine-tuning depends on how much information the training has to push into the model.

The clearest evidence comes from Thinking Machines Lab. They found that LoRA matches full fine-tuning on instruction and reasoning data, in both learning speed and final quality, if two conditions hold Can LoRA match full fine-tuning performance in practice?. First, the adapters have to cover all of the model's weight matrices, not just the attention layers, which many early LoRA setups targeted by default. That is the part that touches architecture: it's less about which model you use and more about where in the model you put the adapters. Second, the dataset has to stay below what the small adapter can hold. Past that point, LoRA falls behind. So the 'scale' in your question really has a ceiling, and the ceiling is capacity, not domain.

The surprising part is how much that ceiling moves with the training method. In reinforcement learning, LoRA at rank 1 (the smallest adapter possible) matched full fine-tuning Why does LoRA rank 1 work for reinforcement learning?. The explanation is about information. Supervised training gives feedback on every token, so it carries a lot of information per example. RL gives about one reward per attempt, roughly a thousand times less information per token. A tiny adapter is plenty to absorb that. So the same LoRA setup can be too small for supervised fine-tuning on a big dataset and more than big enough for RL on the same model.

This connects to a finding that changes how LoRA's success looks. A 1.5B model trained only with LoRA matched larger models that had full RL training on reasoning tasks Can small models reason well by just learning output format?. The authors' reading is that RL mostly teaches the model how to organize its output, not new facts. That would explain why a small adapter is enough: changing format is a small adjustment, while adding knowledge is a big one. A better test of 'domain dependence', then, is whether a task needs new knowledge or a new way of using knowledge the model already has. A domain like medicine could land on either side, depending on what you ask the model to learn.

On architecture itself, the corpus is thin for LoRA specifically. Nearby work shows that architecture choices such as hidden size and the balance between attention and MLP layers change scaling behavior in measurable ways Can architecture choices improve inference efficiency without sacrificing accuracy?. Recommender research found that fitting the design to the problem beats adding raw capacity What architectural choices actually improve recommender system performance?. Neither tests LoRA, so treat them as hints for where to look, not answers. If you want to predict whether LoRA will work for your case, the corpus suggests asking 'how much is my training signal trying to teach?' before asking 'which model or which domain?'