When an AI rewrites its own setup to improve, is it learning real lessons or just memorizing its practice problems?

This explores whether an agent's self-made edits to its scaffolding teach it general strategies or just patch the tasks it was tuned on. The corpus's direct answer is that memorization is the default. A close look at evolved harness trajectories finds that the edits are sensible and well-motivated, but most persist fixes an agent could have rediscovered within a single attempt at the task Do harness edits learn reusable strategies or memorize task fixes?. The harness is caching shortcuts that were already within reach, not turning hard failures into successes, which is why the gains stay limited. A related failure is overfitting. Recursive edits can memorize the tasks used to evolve them, and the gains shrink on tasks the harness hasn't seen Does harness self-improvement memorize tasks instead of learning broadly?.

This suggests a practical test for whether an edit is a strategy or a memorized fix: would plain extra effort have found it anyway? That is the logic behind comparing harness evolution against task-level test-time search under equal feedback and inference budgets. Only the gain beyond what matched search achieves belongs to the harness design How should we measure gains from automatic harness evolution?. The baseline is strong. Evolutionary search at inference time, with no lasting edits at all, beats best-of-N sampling and sequential revision and solves over 98% of planning tasks in one study Can evolutionary search beat sampling and revision at inference time?. A harness edit that merely replays what search would find is doing the same work with extra steps.

The corpus also shows that reusability can be engineered. ModularRSI evolves harness modules separately, using contrastive trajectories on data disjoint from the benchmark, and pools evidence across tasks before updating a component. The gains hold on unseen tasks and domains Can harness modules improve separately from benchmark data?. SkillOpt takes a different route and constrains the editing itself. A budget on how much text can change per step, a held-out validation gate, and a buffer that keeps rejected edits as negative feedback all beat letting the agent freely rewrite its instructions Does constraining edits make skill learning more stable?. Running the loop across many environments surfaced four mechanisms (action execution, context compaction, observation handling, delegated reading) that cut token traffic by 44.7–49.0% at comparable performance Can agent harnesses be automatically optimized across many environments?. Those are mechanism-level gains rather than task fixes. AIDE2's improvements likewise carried to four held-out benchmarks, including physics-based weather forecasting outside its selection distribution Do AIDE2's improvements transfer to unseen tasks?. The Darwin Gödel Machine's discoveries of better code editing and context management, validated empirically in an archive of agent variants, look like the same kind of general capability Can AI systems improve themselves through trial and error?.

A reusable edit still has to be used by the model that receives it. Models of every tier are about equally able to propose useful harness updates, but the ability to benefit from them peaks in mid-tier models. Weak models fail to invoke the harness, and strong models struggle to follow its instructions faithfully Do stronger models always evolve harnesses better?. Structure helps. Organizing a repository around a map of runtime behavior let weaker planners match stronger models at code localization Can explicit behavior maps help weaker planners compete with stronger models?. Skill libraries that compose simple skills into complex ones are a design where reuse is built in, and they avoid the forgetting that weight updates cause Can agents learn new skills without forgetting old ones?.

The collection doesn't give a clean figure for what fraction of evolved edits generalize. What it does show is that an unconstrained evolving harness drifts toward memorization, and that generalization comes from cross-task evidence, bounded edits, held-out gates and matched-budget baselines.