When AI tools automatically rewrite an agent's instructions and tools, do those upgrades work on new problems or just memorize old ones?

This explores whether the changes an automated system makes to an agent's scaffolding (prompts, tools, control flow) while the model stays frozen still help on tasks and domains they weren't evolved on. The corpus says that by default they mostly don't. A close look at evolved harness trajectories found the edits were sensible and well-motivated, but most just persisted fixes an agent could have rediscovered in a single rollout (Do harness edits learn reusable strategies or memorize task fixes?). Those edits cache shortcuts for problems already within reach. They don't turn hard failures into successes. The same pattern shows up as overfitting: recursive scaffold edits memorize the tasks used for evolution, and the gains shrink when you move out of distribution (Does harness self-improvement memorize tasks instead of learning broadly?).

What does transfer tends to be plumbing rather than task knowledge. When an auto-research loop was scaled across many environments, it converged on four mechanisms: how actions are executed, how context is compacted, how observations are handled, and how reading is delegated. Together they cut token traffic by 44.7–49.0% at comparable performance on a 51-task benchmark (Can agent harnesses be automatically optimized across many environments?). None of these depend on what a task is about, which is probably why they survive a change of domain. Related evidence points the same way. One execution runbook built around frozen weights carried over to newer models without modification (Can execution harnesses lift model performance without retuning weights?). AIDE2's gains held on four held-out benchmarks, including physics-based weather forecasting, which sat outside the distribution used for selection (Do AIDE2's improvements transfer to unseen tasks?).

Generalization also depends on how the evolution is run, not just on what gets found. ModularRSI evolves harness modules separately, uses contrastive trajectories on data disjoint from the benchmark, and aggregates evidence across tasks before touching a component. That separates mechanism-level improvements from task-specific adaptation and shows consistent gains on unseen tasks and domains (Can harness modules improve separately from benchmark data?). RRSI reaches a similar place from another direction, constraining both the proposal and selection stages to favor reusable mechanisms over benchmark-specific ones. Selecting over a population of variants under a preserve-and-extend contract also looks robust, with average 17-point gains across benchmarks (Can frozen models improve by evolving their harnesses?).

Three caveats keep the picture honest. First, a gain that looks like generalization may just be extra compute. Harness gains only count once they beat task-level test-time search given the same feedback and inference budget (How should we measure gains from automatic harness evolution?). Second, transfer across models is uneven. Every model tier can produce useful harness edits, but the ability to benefit from them follows an inverted U, peaking in mid-tier models. Weak models fail to invoke the harness, and strong models struggle to follow it faithfully (Do stronger models always evolve harnesses better?). Third, some harness value is tied to its domain by design. A safety harness strict enough for one model over-blocks another, and policies general enough to cross domains miss application-specific safety relations (Should safety harnesses be customized for each deployment?). Even a successful builder-model harness for Theory-of-Mind nearly doubled a weaker model's score largely through task-specific routing and by moving unstable reasoning into deterministic code (Can a stronger model lift a weaker one at test time without retraining?).

So the working answer is that harness edits generalize to the extent they encode how an agent handles context, actions and observations rather than how to beat a particular task. A related result points to what makes such edits portable. Organizing a harness around runtime behavior, with an explicit map from behaviors to code, let weaker planners match stronger models at code localization (Can explicit behavior maps help weaker planners compete with stronger models?).