AI can rewrite its own setup until it aces practice tasks — so why does it often flop on new ones?

This explores why a harness that an automated loop has rewritten to score well on its training tasks often shows little benefit on new ones. The corpus's most direct answer is that most of what gets evolved is a patch, not a strategy. An analysis of evolved harness trajectories found the edits were rational and well-motivated, but most of them just persisted fixes an agent could have rediscovered in a single rollout Do harness edits learn reusable strategies or memorize task fixes?. They cache what was already within reach instead of turning hard failures into successes. A harness made of such patches looks strong on the tasks it was tuned on and has little to offer on tasks it hasn't seen.

Part of the apparent gain may not come from the harness at all. If you compare an evolved harness to the bare model, you're also comparing more computation and more feedback to less. Only the gain that survives a comparison against plain test-time search, given the same feedback and inference budget, can be credited to the harness design How should we measure gains from automatic harness evolution?. Who runs the harness matters too. Models are about equally good at writing harness edits at every capability tier, but the benefit of using those edits peaks in mid-tier models. Weak models fail to invoke the harness, and strong models struggle to follow its instructions faithfully Do stronger models always evolve harnesses better?. A harness evolved around one model's habits can therefore lose value when a different model runs it.

Other notes in the collection describe the same trap in neighbouring settings. They don't test harness evolution directly, but a closed loop that learns only from what it has seen tends to narrow. Agents trained on static expert demonstrations are capped by what the curators imagined, not by what the agent could do Can agents learn beyond what their training data shows?. In RL on nearly impossible problems, rare lucky successes get rewarded heavily, so models learn shortcuts that even damage skills they already had Do overly hard RLVR samples actually harm model capabilities?. RL also squeezes the diversity of search agents' behaviour, while SFT on varied demonstrations preserves it Does reinforcement learning squeeze exploration diversity in search agents?. The general lesson is that pure self-improvement is circular, and the methods that work bring in an outside anchor such as an earlier model version, a third-party judge or tool feedback Can models reliably improve themselves without external feedback?. A harness evolved only against its own training tasks has no such anchor, so it can end up encoding those tasks.

Generalization does happen, though. One evolved agent, reached through seven accepted rewrites over 8 days, matched or beat its human-built counterpart on four held-out benchmarks, including out-of-distribution weather forecasting Does automated evolution match human-built agent performance?. Separately, transformers taught 10-digit addition generalized to 100 digits by generating solutions, keeping only the correct ones, and retraining, and they kept improving across rounds Can transformers improve exponentially by learning from their own correct solutions?. In both cases the loop kept only changes that passed a check (an accepted rewrite, a verified answer) rather than every fix that worked once. That is a plausible reason those loops transfer while typical harness evolution doesn't, though the notes don't test that directly.

The practical test is to ask of any evolved edit whether the agent would have found it anyway in one attempt, and whether it still helps against matched search on tasks the loop never saw. If it fails both, it is a memorized shortcut rather than a reusable strategy.