Training an AI on deliberately wrong instructions barely hurts its performance — so what exactly did it learn?

This explores the gap between what a model *appears* to do at the surface and the training process that produced it — and the corpus's recurring lesson is that surface behavior systematically misleads you about its own origin. The cleanest case is instruction tuning: models trained on semantically empty or deliberately *wrong* instructions perform about as well as models trained on correct ones (43% vs. a 42.6% baseline). The surface story — 'the model learned to understand the task' — is false; what actually transferred was knowledge of the output space, not task comprehension Does instruction tuning teach task understanding or output format?. The mechanism (right-looking answers) and the regime (learning a format distribution) are different things, and you only catch the difference by perturbing the training, not by reading the outputs.

The same split runs through reinforcement learning. A model that suddenly 'reasons better' after RL looks like it acquired a new skill, but the training regime tells a different story: RLVR mostly acts as a catalyst that surfaces strategies already latent in the pretrained prior, with updates that are structurally sparse and bounded by what pretraining already contained How does RL training reshape reasoning and what gets lost?. Mechanically, the change is even stranger than it looks — RL concentrates in a sparse but full-rank subnetwork, updating only 5–30% of parameters, and its primary lever is *suppressing* wrong trajectories rather than amplifying right ones What actually changes inside a model during RL training?. So the surface gain ('new reasoning') is produced by a regime of negative reinforcement on a fixed capability set — a mechanism almost opposite to the intuitive read.

But the distinction isn't a clean 'surface always lies.' The regime determines *when* a surface behavior is genuinely new versus merely activated. Capability creation turns out to be domain-conditional: for standard reasoning, RL activates latent abilities; for complex multi-step planning, the same training generates genuinely novel strategies a base model can't reach even with heavy sampling Does reinforcement learning create new reasoning abilities or activate existing ones?. And the regime has internal structure of its own — RL training moves through a predictable two-phase arc, first consolidating execution correctness, then shifting the bottleneck to strategic planning Does RL training follow a predictable two-phase learning sequence?. Which surface skill you see depends on which phase, and even which *format*, the regime happened to amplify — RL tends to collapse onto a single dominant pretraining format, often selected by model scale rather than performance Does RL training collapse format diversity in pretrained models?.

This is exactly why you can't infer the regime from the mechanism by inspection — and why the corpus insists on causal, not just representational, analysis. Identifying a feature that correlates with a behavior tells you a surface mechanism exists; only intervening on it tells you the training actually wired it that way, which is why complete mechanistic claims need both representational *and* causal methods Can LLM understanding rely on just representation or causation alone?. The deeper warning is that identical benchmark scores can sit on top of radically different internal structures, and pushing one surface capability reliably degrades another out of view What really happens inside a language model?. Two models with the same mechanism on the test can carry entirely different regimes underneath.

The payoff for a curious reader: the field is converging on 'learning mechanics' — borrowing from statistical physics — as the frame that explains surface behavior through *training dynamics* rather than static architecture, treating the trajectory of learning as the real object of study Can deep learning theory unify around training dynamics?. Even something as basic as training *order* leaves a mechanical fingerprint: train structured tasks before creative ones and you prevent an entropy collapse that would otherwise quietly damage open-ended ability Does training order reshape how models handle different task types?. The behavior you see at the surface is a shadow cast by the regime — and the whole research program is about learning to read the shadow back to its source.