If you tune an AI agent's prompts and tools until it aces one test, do those gains survive elsewhere?

This explores whether tuning the scaffolding around an LLM agent (its prompts, tools, memory and execution loop) gives gains that carry over to new benchmarks, models and agent designs, or only to the tasks it was tuned on. The corpus says partly yes, and the split is useful: general mechanisms transfer, while specific fixes mostly don't.

The evidence for transfer is real. Across models, one execution runbook lifted Terminal-Bench 2.1 scores for several models with their weights frozen, then carried over to newer models unchanged, reaching 95.3% on GPT-5.6 and adding 5.4 points to DeepSeek-V4 Flash Can execution harnesses lift model performance without retuning weights?. Across environments, running the optimization loop over many environments at once found four mechanisms (action execution, context compaction, observation handling, delegated reading) that cut token traffic by 44.7–49.0% at comparable performance on a 51-task benchmark Can agent harnesses be automatically optimized across many environments?. Across data, ModularRSI evolves harness modules on benchmark-disjoint data and still shows gains on unseen tasks and domains Can harness modules improve separately from benchmark data?. In a related result, AIDE2's gains held on four held-out benchmarks, including physics-based weather forecasting, which sits outside the distribution it was selected on Do AIDE2's improvements transfer to unseen tasks?.

The counterweight is that much of what a harness learns is memorization. An analysis of evolved harnesses found the edits were sensible, but most simply stored fixes an agent could have rediscovered in a single rollout. They cache what is already within reach and rarely turn hard failures into successes Do harness edits learn reusable strategies or memorize task fixes?. Set beside the results above, this suggests generality comes from how the optimizer is set up rather than from the harness itself. ModularRSI gathers evidence across many tasks before touching a component, and the cross-environment loop optimizes over many environments simultaneously. Both setups make it hard to get credit for a fix that only helps one task. This is my reading of the notes, not something any of them tests head-to-head.

The model on the receiving end matters too. Producing useful harness edits is about equally easy for every model tier, but benefiting from them follows an inverted U that peaks in mid-tier models. Weak models often fail to invoke the harness, and strong models struggle to follow its instructions faithfully Do stronger models always evolve harnesses better?. So a harness that transfers between two frontier models may still behave differently on a much smaller or larger one. Harness structure can also stand in for capability. Reorganizing a repository around a behavior-to-code map let weaker planners match stronger models' code localization and lifted win rates by 10–19 points Can explicit behavior maps help weaker planners compete with stronger models?. That tilts the answer toward a harness working better when it hands the model well-organized structure than when it hands over more instructions. Part of why so much work lands here is that non-parametric updates (prompts, memory, tools) are cheaper and reversible compared with retraining weights Do self-improving agents really split into two distinct loops?.

How you measure generalization also changes the answer. The best-supported transfer result above is about efficiency (fewer tokens at the same score), and identical task-success rates can hide large differences in efficiency, reliability and verification cost How should we measure agent system performance beyond task success?. A study that only checked pass rates on a second benchmark could miss the gains that transfer best. The corpus has little on the other half of the question: whether one harness carries across fundamentally different agent architectures, such as single-agent versus multi-agent setups. The transfer evidence here comes almost entirely from swapping models and benchmarks under a similar agent loop.