Does making an AI agent's tools and prompts smarter actually boost its real-world accuracy, or is that just noise?

This explores whether the code and prompts wrapped around a model, rather than the model itself, can raise accuracy on benchmarks beyond SWE-bench Verified. The corpus says yes, with an important twist. The gains show up when a harness is built or tuned for a task. They mostly do not show up when you swap one off-the-shelf coding harness for another. On SWE-bench Verified, switching among three production harnesses across five models moved pass rates no more than ordinary rerun variation, but changed token cost by up to threefold Does switching between coding harnesses change model accuracy?. So the 'harnesses don't matter' result is really about mature, general-purpose coding harnesses that have converged on similar designs.

Elsewhere, designed harnesses produce large, repeatable gains. On Terminal-Bench 2.1, StateM lifts several frozen models without touching their weights. The same runbook works unchanged on newer models, reaching 95.3% on one and adding 5.4 points to another Can execution harnesses lift model performance without retuning weights?. On Theory-of-Mind benchmarks, a stronger model wrote inference harnesses that nearly doubled a weaker model's scores. The surprising part is how: it didn't push the weaker model to 'think longer.' It moved shaky reasoning steps into deterministic code and sent different question types down different paths Can a stronger model lift a weaker one at test time without retraining?. In code localization (finding where in a codebase a change belongs), reorganizing the repository around how the program behaves at runtime let weaker planners match stronger ones while using fewer tokens Can explicit behavior maps help weaker planners compete with stronger models?.

The gains also seem to carry over to new tasks when the harness is improved in a disciplined way. ModularRSI evolves each harness component separately, on data kept apart from the benchmark. It reports consistent gains on unseen tasks and domains, which suggests some harness improvements are reusable mechanisms rather than benchmark-specific tricks Can harness modules improve separately from benchmark data?. In RSIGym's trials with six frontier models, fixing the harness reliably helped, while extra fine-tuning lowered scores in eight of ten comparisons. The top agents spent their effort diagnosing harness problems, not training harder Do harness fixes or heavier training drive frontier model gains?. This fits Weng's argument that the practical near-term route to AI improving itself runs through harness engineering, not models rewriting their own weights Does recursive self-improvement start with harness engineering?.

Two cautions keep this honest. First, a harness win is sometimes an efficiency win rather than an accuracy win. Automated harness optimization across many environments found four mechanisms that cut token traffic by 44.7–49.0% while performance stayed roughly the same Can agent harnesses be automatically optimized across many environments?. Second, many claimed gains may not survive careful measurement. On SWE-bench Verified, rerunning the same harness flips task outcomes about as often as switching harnesses. With 45 tasks, a 13-point gap gets detected only half the time, so any claimed advantage needs a rerun baseline Do harness comparisons on SWE-bench Verified measure real differences?. Evolved harnesses should also be compared against simply spending the same compute on test-time search. Only the gain beyond that comparison belongs to the harness design itself How should we measure gains from automatic harness evolution?.

The takeaway: harness gains are real and reach well beyond coding benchmarks. They are largest when the harness takes unreliable reasoning away from the model, by putting it in code, routing, or better-organized context. The flat SWE-bench result may say less about harnesses than about how alike today's coding harnesses have become.