Can you prove one piece of an AI agent's setup — like its planning step — actually caused a better score, not just a cheaper run?

This explores whether you can point to one piece of the scaffolding around a model (its planning step, how it manages context, how it filters its own reasoning) and show that piece raises a benchmark score. The short answer is yes, but less often than the headlines suggest, and the gain often shows up as lower cost rather than higher accuracy. Whole-harness gains are well documented. One execution system lifted several frozen models on Terminal-Bench 2.1 without changing their weights, and the same runbook carried over to newer models Can execution harnesses lift model performance without retuning weights?. In RSIGym trials, fixing the harness kept improving frontier models, while extra fine-tuning lowered scores in eight of ten comparisons Do harness fixes or heavier training drive frontier model gains?.

When researchers take the harness apart, the value of each component depends on the situation. A controlled study of coding harnesses found that context management matters most when the context window is tight. Planning does something different depending on the model: it improves accuracy for weaker models, but for stronger models its main effect is lowering cost Which coding harness components matter most in different conditions?. Some components do produce large, specific gains. A stronger model built inference harnesses that nearly doubled a weaker model's Theory-of-Mind scores. The gain came mainly from moving unstable reasoning steps into deterministic code and routing each task type to its own handling, not from making the model think longer Can a stronger model lift a weaker one at test time without retraining?. A narrower component, filtering reasoning traces by step-by-step confidence instead of an average over the whole trace, matched majority-voting accuracy while generating far fewer traces Does step-level confidence outperform global averaging for trace filtering?.

The twist is measurement. On SWE-bench Verified, rerunning the *same* harness flips task outcomes about as often as switching to a different one. A typical 45-task comparison catches a real 13-point gap only half the time Do harness comparisons on SWE-bench Verified measure real differences?. Across three production coding harnesses and five models, switching harnesses moved accuracy no more than rerun noise did, but it changed token cost by up to three times Does switching between coding harnesses change model accuracy?. Automated harness search shows the same pattern: four discovered mechanisms (action execution, context compaction, observation handling, delegated reading) cut token traffic by nearly half while accuracy stayed roughly flat Can agent harnesses be automatically optimized across many environments?. Even when accuracy does rise, some of the gain may just be extra compute. Evolved harnesses should be compared against plain test-time search with the same budget before the design gets credit How should we measure gains from automatic harness evolution?.

There's also a risk that a component which works on one benchmark is overfit to it. ModularRSI tackles this by evolving each harness module separately, on data kept apart from the benchmark, and updating a module only after evidence from many tasks agrees. That separates reusable improvements from benchmark-specific tuning Can harness modules improve separately from benchmark data?. Safety harnesses show the opposite problem: settings strict enough for one model block too much on another, so they have to be tuned for each deployment Should safety harnesses be customized for each deployment?. Weng argues that this kind of harness engineering, not models rewriting their own weights, is where recursive self-improvement starts in the near term Does recursive self-improvement start with harness engineering?. If so, telling real component gains apart from noise and overfitting matters more than any single benchmark number.