Why does cheap AI fine-tuning flop if you only tweak the attention layers and skip the rest of the network?

This explores why LoRA, which trains small add-on matrices instead of retraining the whole model, underperforms when it is attached only to attention layers. The clearest evidence in the corpus is about coverage. Thinking Machines Lab found that LoRA can match full fine-tuning in sample efficiency and final performance, but only when it covers all the weight matrices, including the feed-forward (MLP) blocks that sit between attention layers, and only while the dataset stays below LoRA's capacity Can LoRA match full fine-tuning performance in practice?. Read the other way, that finding tells you the attention-only version is the one that falls short. The problem isn't LoRA as a technique. It's where you put it.

The corpus gives an indirect answer to why placement matters so much: attention mostly decides *where the model looks*, not *what it knows*. Several notes treat attention as a routing system. Up to 40% of attention heads can be swapped for short executable programs without hurting question-answering Can executable programs replace transformer attention heads?. A handful of enormous, input-independent activations act as hidden bias terms that steer attention toward particular tokens Do hidden massive activations act as attention bias terms?. Much of attention's failure behavior comes down to noise, meaning focus spread onto irrelevant context. Cancelling that noise improves hallucination, in-context learning and activation outliers all at once Can subtracting attention maps remove noise from context? Can fixing attention noise solve multiple LLM failures at once?. If attention is mainly the switchboard, then fine-tuning only attention lets you re-route information but not change much of what gets computed or stored. The corpus doesn't test this mechanism directly, so treat it as a strong inference, not a proven result.

The flip side is surprising: how much LoRA needs depends on what you're teaching. For reinforcement learning, LoRA at rank 1, the smallest possible add-on, matches full fine-tuning. The reason is that an RL episode carries only a few bits of feedback, about 1000× less information per token than supervised training Why does LoRA rank 1 work for reinforcement learning?. In a related finding, a 1.5B model trained with LoRA alone matched larger fully trained RL models on reasoning, apparently because RL mostly teaches how to lay out the output, not new facts Can small models reason well by just learning output format?. So the attention-only shortfall should matter most when you're trying to add knowledge or rich behavior through supervised data, and least when you're only nudging style or format.

One caution against concluding that attention doesn't matter: changing *how* attention works can be transformative. Switching a decoder-only model from one-directional to two-directional attention turns it into a top text encoder. The bottleneck there was the attention mask, not model size Why do decoder-only models underperform as text encoders?. The takeaway: attention is powerful when you change its structure, but on its own it's a thin place to fit new learning. The corpus doesn't include a head-to-head study of attention-only versus MLP-only LoRA, so the case above is assembled from adjacent evidence, not one decisive experiment.