Want to make one AI as good as another: rewrite the instructions, or build code around it? Evidence favors code.
How do prompt optimization and code harnesses compare for capability transfer?
This explores whether it's better to move capability between models by rewriting the prompt or by building code around the model (a harness: the scaffolding that runs the model, manages its context, calls tools and routes tasks). The corpus has no controlled prompt-versus-harness comparison, so the answer is assembled from adjacent evidence.
This explores whether it's better to move capability between models by rewriting the prompt or by building code around the model (a harness: the scaffolding that runs the model, manages its context, calls tools and routes tasks). The corpus has no controlled prompt-versus-harness comparison, but the evidence leans toward code for transferring capability. Prompts mostly transfer the user's expectations.
In theory, prompts can do anything. Can a single transformer become universally programmable through prompts? proves that one fixed-size transformer can compute any computable function given the right prompt. The catch is that standard training rarely produces models that actually behave this way. Meanwhile, How much does the user shape what a model generates? frames prompt refinement as steering output toward what the user already expects. Prompt iteration is best at aligning a model with what you had in mind. It is not a proven way to hand new competence to a model that lacks it.
Harnesses show the clearest transfer evidence. In Can a stronger model lift a weaker one at test time without retraining?, a stronger model built inference-time harnesses that nearly doubled a weaker model's Theory-of-Mind scores without retraining. The gain came mainly from moving unstable reasoning into deterministic code and task-specific routing, not from coaxing longer reasoning out of the model. In Can execution harnesses lift model performance without retuning weights?, the same runbook carried over to newer models unchanged, reaching 95.3% on GPT-5.6 and adding 5.4 points to DeepSeek-V4 Flash with the weights frozen. In Can explicit behavior maps help weaker planners compete with stronger models?, a behavior-to-code map let weaker planners match stronger models at locating the relevant code. So a harness carries capability partly by doing the work itself, and it does not depend on the model reading instructions well.
That transfer has limits. Which coding harness components matter most in different conditions? finds that which component pays off depends on the setup: context management matters most under tight windows, and planning helps weaker models before it starts cutting costs for stronger ones. Do stronger models always evolve harnesses better? finds that the ability to benefit from harness edits peaks at mid-tier models. Weak models fail to invoke the harness, and strong models struggle to follow its instructions faithfully. That is a reminder that harnesses still lean on a prompt-shaped interface.
Automatically optimized harnesses face the same transfer problem, a subtler one. Do harness edits learn reusable strategies or memorize task fixes? shows that evolved edits, which span prompt and tool layers, mostly cache fixes an agent could have rediscovered in one attempt. Does harness self-improvement memorize tasks instead of learning broadly? shows in-distribution gains shrinking out of distribution. Two approaches try to fix this. Can harness modules improve separately from benchmark data? evolves modules separately on data disjoint from the benchmark, and Can agent harnesses be automatically optimized across many environments? optimizes across many environments, cutting token traffic by 44.7–49.0% with comparable performance. Held-out results also exist: Do AIDE2's improvements transfer to unseen tasks? reports gains on four unseen benchmarks, one of them outside its selection distribution. The split that matters is between reusable mechanisms and cached task fixes, more than between prompt and code.
Sources 12 notes
Research proves a single finite-size transformer exists that can compute any computable function given the right prompt, achieving complexity bounds nearly matching unbounded models. However, standard training rarely produces models that learn to implement arbitrary programs this way.
Foundation Priors research shows prompt engineering as divergence minimization between synthetic output and user priors. The refinement process systematically steers generation toward what users already expect, making outputs co-productions of model and user subjectivity.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Show all 12 sources
A controlled study varying planning, action space, and context management across models and budgets found that context management becomes most valuable under tight windows, while planning shifts from helping weaker models to cutting costs for stronger ones.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
Recursive edits to agent scaffolding can memorize evolve-set tasks, causing in-distribution gains to shrink out-of-distribution. RRSI addresses this by constraining the proposal and selection stages to favor reusable mechanisms over benchmark-specific ones.
ModularRSI evolves harness modules independently using contrastive trajectories on benchmark-disjoint data, showing consistent gains across unseen tasks and domains. The approach isolates mechanism-level improvements from task-specific adaptation by aggregating evidence across tasks before updating components.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable