When an AI agent tweaks its own prompts and tools after seeing some tasks, do those tweaks help on new ones?
Can harness edits trained on one batch transfer to new tasks?
This explores whether improvements an AI system makes to its own scaffolding (prompts, tools, routing code) after seeing one batch of tasks carry over to tasks it hasn't seen, or whether the edits just memorize that batch.
By default, mostly not. This explores whether edits to an agent's scaffolding learned on one batch of tasks carry over to new ones, and the corpus says the default outcome is memorization. Analysis of evolved harness trajectories finds the edits are rational and well-motivated, but most just persist fixes the agent could have rediscovered in a single rollout Do harness edits learn reusable strategies or memorize task fixes?. They cache what's already within reach rather than turning hard failures into successes. The same pattern shows up as a measurable gap: recursive self-improvement can gain a lot on the tasks it evolved against, then shrink sharply out of distribution Does harness self-improvement memorize tasks instead of learning broadly?.
The more interesting finding is that transfer can be engineered, and the fixes fall into three groups. The first is regularization: constrain how edits are proposed and selected so that reusable mechanisms win over benchmark-specific patches Does harness self-improvement memorize tasks instead of learning broadly?. The second is separation: evolve harness modules independently, using contrastive trajectories on data that is disjoint from the benchmark, and aggregate evidence across tasks before touching a component. That approach showed consistent gains on unseen tasks and domains Can harness modules improve separately from benchmark data?. The third is gating: treat a skill document as trainable state and accept an edit only if it improves performance on held-out validation data. That method matched or beat baselines across 52 benchmark cells and seven models, and it transferred across models too Can skill documents be optimized like neural network weights?.
Breadth of training matters as well. Optimizing across many environments at once surfaced four general mechanisms (action execution, context compaction, observation handling, delegated reading). They cut token traffic by 44.7–49.0% at comparable performance on a 51-task benchmark Can agent harnesses be automatically optimized across many environments?. The lesson is that what transfers tends to be mechanism-level (how to manage context, how to route) rather than task-level (what to do on this puzzle). A related result supports this: a stronger model built inference-time harnesses that nearly doubled a weaker model's Theory-of-Mind scores, mostly by moving unstable reasoning into deterministic code and task-specific routing Can a stronger model lift a weaker one at test time without retraining?. The harness there carries capability from one model to another.
The same worry appears elsewhere in the library, and the proposed remedies rhyme. AIDE2 reports gains on four held-out benchmarks, including physics-based weather forecasting outside its selection distribution, which is the kind of check that separates real transfer from overfitting Do AIDE2's improvements transfer to unseen tasks?. In fine-tuning, isolating each task's core parameters and merging the rest prevents tasks from interfering with each other, and scheduling tasks in sequence isn't enough without that structure Can isolating task-specific parameters prevent multi-task fine-tuning interference?. Harness modularity is a text-space version of the same idea.
The corpus doesn't settle how much transfer is achievable. The strongest positive results come from methods built with generalization in mind, such as held-out gating, disjoint data and cross-environment aggregation. Plain harness evolution mostly memorizes. So the practical question to ask of any self-improving harness is what it was forced to prove on data it never trained on.
Sources 8 notes
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
Recursive edits to agent scaffolding can memorize evolve-set tasks, causing in-distribution gains to shrink out-of-distribution. RRSI addresses this by constraining the proposal and selection stages to favor reusable mechanisms over benchmark-specific ones.
ModularRSI evolves harness modules independently using contrastive trajectories on benchmark-disjoint data, showing consistent gains across unseen tasks and domains. The approach isolates mechanism-level improvements from task-specific adaptation by aggregating evidence across tasks before updating components.
SkillOpt treats skill documents as trainable external state of frozen agents, using a text-space optimizer with held-out validation gating to accept only edits that improve performance. Across 52 benchmark cells and seven models, the approach matches or exceeds baselines while adding zero inference cost and enabling transfer across models.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
Show all 8 sources
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
Research shows that identifying core parameter regions per task, clustering overlapping tasks, and freezing core parameters while geometrically merging non-core parameters consistently outperforms standard multi-task fine-tuning. Temporal task scheduling alone proves insufficient without explicit structural parameter isolation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution