Fine-tuning can change a model in two ways: tweak a few weights freely, or spread small tweaks across many weights — which one actually happens?

This explores two different ways a fine-tuning update can be 'small': changing only a few weights while leaving the change high-dimensional, or changing many weights but forcing the change to be low-dimensional, as LoRA does. The corpus has no head-to-head experiment comparing the two. It does have enough to show that they are different kinds of small, and the difference is surprising.

The key finding is that reinforcement learning makes the first kind of change on its own, with no one asking it to. Across seven RL algorithms and ten model families, RL ends up touching only 5–30% of a model's parameters. No regularization pushes it there. The updates that do happen are nearly full-rank, so they aren't squeezed into a few directions the way LoRA assumes. They are also nearly identical across random seeds, which means the model keeps returning to the same subnetwork rather than picking weights at random Does reinforcement learning update only a small fraction of parameters?. The picture is of a model that already 'knows' which weights matter for the task and edits only those, but edits them in rich, high-dimensional ways.

On that picture LoRA should struggle, since its whole premise is that useful updates live in a low-rank space. Yet Thinking Machines Lab found LoRA matches full fine-tuning on instruction and reasoning data, with two conditions: it must cover all weight matrices, not just attention, and the dataset must stay below LoRA's capacity Can LoRA match full fine-tuning performance in practice?. One way to fit these together is that the two kinds of small don't conflict. RL feedback carries relatively little information per example, so even a full-rank update may hold little enough that a well-spread low-rank adapter can approximate it. The 'cover all matrices' condition also fits the sparsity finding: if the relevant weights are scattered through the network, an adapter that only touches attention will miss many of them. Read the two notes together for that reason. Neither one tests this connection directly.

A third option sidesteps the question entirely. Representation fine-tuning (ReFT) leaves the weights frozen and learns small edits to the model's hidden activations. Its low-rank variant beats LoRA while using 10–50× fewer parameters Can editing hidden representations beat weight updates for finetuning?. That suggests 'which weights, and at what rank' may be the wrong question for many tasks. The capability may already be in the model, and the job is to steer it.

A useful parallel comes from outside fine-tuning. Sparse attention can match dense attention at frontier scale, but only when the sparsity is learned during pretraining rather than bolted on afterward Can sparse attention match dense models without retrofitting?. The RL result has the same shape: sparsity that the training process discovers for itself looks different from, and possibly better than, a structure you impose in advance. A rank limit is a guess about what the update will look like. Emergent sparsity is the model showing you.