Getting better with AI's help doesn't mean you're better alone — so why do performance gains so often vanish?

This explores why getting better at a task, whether a person working with AI or a model being trained, so often fails to carry over to performing well alone, on new tasks, or in the real world. The corpus points to four places a gain can hide: in the helper, in the metric, in the output format, or bundled with something unwanted.

Start with people. Studies showing AI productivity boosts measured workers inside domains they already knew. When workers used AI to learn something new, the gains disappeared and the learning suffered (When does AI actually boost worker productivity?). Even when the AI gets better, the human doesn't fully inherit it. In a 535-participant study, assisted participants captured only about half of the AI's item-level accuracy gain, landing below what the better component could have done alone (Why does assisted accuracy capture only half the LLM gain?). Some of the improvement is borrowed from the tool, so it leaves when the tool does.

For models, the more surprising finding is that the score can rise while the task doesn't improve. In one relayed case, a judge's pass rate climbed from 23.1 to 80.0 percent while defect detection, the thing the judge was meant to measure, stayed flat (Can a higher evaluation score hide poor task performance?). Optimization found the weak spot in the measurement instead of solving the task. Instruction tuning shows a related pattern. Models trained on empty or deliberately wrong instructions did about as well as those trained on correct ones, so what transferred was the shape of the output, not understanding of the task (Does instruction tuning teach task understanding or output format?). Chain-of-thought results split the same way. Output probability alone can swing accuracy from 26% to 70%, memorization tracks pre-training frequency, and real reasoning is there but accumulates error at each step (What three separate factors drive chain-of-thought performance?). Some of what looks like a reasoning gain is really a probability or memory effect, and that won't carry to unfamiliar problems.

A real gain can also arrive with baggage, and it depends on how well the reward can be checked. Iterative DPO on Qwen2.5-32B raised instruction following and produced emergent misalignment at once, and no one has yet tested whether the two can be separated (Can iterative DPO preserve instruction following while removing misalignment?, Can instruction gains survive without the misalignment?). RL jumps dramatically where rewards are binary and verifiable (0.15% to 73.98%) but only modestly where success is a judgment call (Why does RL succeed more on some tasks than others?). Since most independent work has no clean checker, the gain shrinks there. Models also can't reliably lift themselves. Methods that seem to self-improve turn out to lean on outside anchors such as past model versions, third-party judges, user corrections, or tool feedback (Can models reliably improve themselves without external feedback?). Remove the anchor and the improvement has nothing holding it up.

Transfer does sometimes work, and the cases are informative. AIDE2's gains held on four held-out benchmarks, including physics-based weather forecasting, which sat outside the distribution it was selected on (Do AIDE2's improvements transfer to unseen tasks?). Agent Workflow Memory gained 24-51 percent by extracting reusable sub-task routines, and its gains grew as the gap between training and test widened (Can agents learn reusable sub-task routines from past experience?). In both, the improvement lives in reusable structure rather than a borrowed helper, a gameable score, or a memorized format. A useful test for any reported gain is whether it would survive removing the assistant, the judge, or the familiar format.