Switching the tool that wraps your coding AI can triple your bill without changing how often it succeeds — why?
What explains the threefold cost differences between harnesses?
This explores why one coding-agent harness can cost about three times as much as another to run the same model on the same tasks, and what that says about where harness money actually goes.
This explores why switching the harness wrapped around a coding agent can triple the bill while barely changing results. The clearest evidence comes from a comparison of three production harnesses running five models on SWE-bench Verified. Pass rates moved no more than normal rerun noise, but token cost changed by up to threefold Does switching between coding harnesses change model accuracy?. The explanation is simple arithmetic. Every step the agent takes resends a fixed overhead: the system prompt and the descriptions of every tool it can call. A harness with a heavier prompt, more tools, or a habit of taking more steps pays that overhead again on every turn, so the cost multiplies.
The surprising part is that this extra spend mostly buys nothing you can measure. Rerunning the same harness flips task outcomes about as often as swapping harnesses does. A 13-point accuracy gap is caught only half the time on typical task-set sizes, so many claimed harness advantages may be noise, while the cost gap is real and repeatable Do harness comparisons on SWE-bench Verified measure real differences?. Benchmarks that report only accuracy hide the difference that matters most for anyone paying for tokens.
Which parts of a harness earn their cost depends on the setting. A controlled study that varied planning, the set of available actions, and context management found that context management pays off most when the context window is tight. Planning helps weaker models get answers right, but for stronger models its main benefit is lower cost Which coding harness components matter most in different conditions?. Automated harness search points the same way. Optimizing harnesses across many environments produced four mechanisms (how actions run, compacting context, trimming tool output, and handing reading work to a sub-agent) that cut token traffic by 44.7–49.0% with similar performance Can agent harnesses be automatically optimized across many environments?. Much of what a harness costs is how it moves text in and out of the model, not the model's reasoning.
This also changes how harness "improvements" should be judged. If a new harness wins partly by spending more inference, the fair comparison is against plain test-time search with the same budget. Only gains beyond that come from the design itself How should we measure gains from automatic harness evolution?. Not every component is expensive: deterministic checks such as parsing validation cost orders of magnitude less than an LLM judge, so the bill is driven by what gets put in front of the model How much do deterministic guardrails actually cost to run?.
The takeaway: a cheaper harness doesn't mean a dumber agent. Within today's noise, harnesses compete mainly on efficiency, and the cost lives in repeated per-step overhead that is easy to cut once you measure it.
Sources 6 notes
Across three production harnesses and five models on SWE-bench Verified, switching harnesses shifted pass rates within normal rerun variation but changed token cost up to threefold. The cost difference came from fixed overhead in system prompts and tool schemas, multiplied by step count.
On SWE-bench Verified, rerunning the same harness flips task outcomes as often as swapping harnesses. A 13-point accuracy gap is caught only half the time on typical task set sizes, requiring a rerun baseline to verify any claimed harness advantage.
A controlled study varying planning, action space, and context management across models and budgets found that context management becomes most valuable under tight windows, while planning shifts from helping weaker models to cutting costs for stronger ones.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
Show all 6 sources
Mechanical checks like parse validation and planted cases require only arithmetic and string operations, while judges consume most budget on inference. This cost gap makes verification layers realistic without competing for computational resources.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- What Does a Harness Buy? Tokens, Mostly
- Rethinking the Evaluation of Harness Evolution for Agents
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- Harness Engineering for Self-Improvement
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Sharpening Tax in Post-Training