What Does a Harness Buy? Tokens, Mostly

A coding agent is a language model wrapped in a harness: the system prompt, the tool set, and the context management that turn a chat model into something that can work inside a repository. Much of the day-to-day engineering of these agents happens in the harness. Production harnesses ship releases daily, vendors advertise pass-rate gains from harness changes, and leaderboards mix harnesses freely. What is rarely measured is how much the harness itself moves the score when the model is held fixed. This paper measures it. We run five models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified, and rerun the same configurations to calibrate how much a score moves when nothing changes but the run. The answer to the title question is: not pass rate. On 447 tasks and the two models we ran there, Claude Code and mini-SWE-agent, the heaviest and the lightest harness, are equivalent within five points. On a 45-task hard subset and five models, swapping the harness flips as many tasks as rerunning the same harness, 13% in both cases, and the tasks a harness wins in one run are not the tasks it wins in the next. The one harness effect that clears the noise is a loss, not a gain: OpenCode trails by up to 9 points on the large pool, and on one model half of that gap sits in runs its output cap cut short. What the harness does decide is the bill. With the same model, the same tasks, and one price list, cost per task differs by up to 3× across harnesses. The gap is set at the first call, by the preamble of system prompt and tool schemas each harness sends with every step, and scaled by the number of steps; per-step growth and per-call tool output differ far less. The provider’s price for cached input scales the bill and does not reorder it. The rerun data also give the resolution a harness comparison needs: at the discordance we observe, 45 tasks catch a 13-point gap only half the time and no gap with 80% power, and 447 tasks resolve 5 points, still coarser than the gains many harness changes claim.

Introduction. Coding agents have turned into a product category, and the product is a model plus a harness. The harness, which part of the literature calls a scaffold, is everything around the model: the system prompt, the tools it can call, the way tool output is fed back, and the way the conversation is trimmed when it grows too long. It is also the part that changes most: popular harnesses ship more than two releases a day (Ben Sghaier et al., 2026), most of it prompts, tool surfaces, and context management rather than the model.

The field has come to believe that this engineering moves the score. The belief has evidence behind it. On a public terminal benchmark, the same model is listed at 52.1% under one harness and 57.8% under another, with the lower-scoring arm consuming 65× the input tokens (Vats & Golev, 2026). Six harnesses run over the same model pool span 23.8 points (Yao et al., 2026), and five harnesses on one fixed model span 27.4 points (Zheng et al., 2026). Vendors sell harnesses on this basis: one advertises a 2.4-point gain over Claude Code with its own model held fixed (Zhipu AI, 2026). Leaderboards, meanwhile, mix harnesses freely, so a ranking of models is also a ranking of whatever harness each submitter chose (Zhang et al., 2026).

Related work. Harness comparisons. Several recent studies swap whole harnesses under a fixed model and report large spreads. Six configurable harnesses over the same model pool span 23.8 points on a graded task score (Yao et al., 2026), five harnesses on one model span 27.4 points of Pass@1 (Zheng et al., 2026), and one fixed model costs 139× more tokens under its worst scaffold than under its best on a single git task (Alier Forment et al., 2026). Two findings in this line point the way we go. Three production harnesses paired on the same terminal tasks differ by 0 to 8 points with intervals that cover zero except the largest, while tokens per solved task differ 40× (Vats & Golev, 2026), and 35 consecutive releases of one harness bring no significant score gain on a fixed model while nearly doubling token use (Ben Sghaier et al., 2026). The scaffold papers on SWE-bench before them, SWE-agent, OpenHands, and the agentless pipeline, each compared their own design against the others under a fixed model (Yang et al., 2024; Wang et al., 2025; Xia et al., 2025). Every study in this line runs each cell once or twice. Rankings also move with the harness: which model is most robust to meaning-preserving rewrites of a task depends on whether it runs under mini-SWE-agent or OpenCode (Mahmud et al., 2026), and a position paper asks that comparisons disclose the harness for this reason (Zhang et al., 2026). We keep the whole-harness unit and the production artifacts, and add the two things this line lacks: reruns of the same cell, and a bill computed under one price list. On pass rate our result agrees with theirs, with the power to bound it.

What is inside a harness. A source-code taxonomy of coding agents lists twelve design dimensions and asks for exactly the control we use, a fixed model with the scaffold varied (Rombaut, 2026). The three harnesses we run sit at different points of that taxonomy. mini-SWE-agent gives the model a single bash tool and no context management, and reaches SWE-agent’s score in about a hundred lines of Python (Yang et al., 2024; Lieret et al., 2026). Claude Code and OpenCode add typed file tools, long system prompts, and automatic compaction (Anthropic, 2026; Anomaly Innovations, 2026). Component studies show that individual pieces matter under pressure: a context-management package nearly doubles a small model’s score in a 20k-token window (Lewis, 2026), and restricting the tool surface changes cost by task regime and agent while pass rates stay tied (Yang et al., 2026).

Method. Harnesses. We run three production harnesses as they ship, with the version pinned per arm: Claude Code 2.1 (releases 2.1.259 to 2.1.273 in the scored arms; 2.1.263 on GLM and Opus), mini- SWE-agent 2.4.6, and OpenCode 1.18 (1.18.30 and 1.18.31 in the scored arms; 1.18.30 on GLM) (Anthropic, 2026; Lieret et al., 2026; Anomaly Innovations, 2026). The three span the design space of Section 2: one minimal harness and two full products, one closed and one open. They differ in what they give the model. mini-SWE-agent exposes one bash tool, trims only very long observations, and has no context management beyond a hard stop at the model’s window. Claude Code and OpenCode expose typed file, search, and edit tools, prepend long system prompts, and compact the transcript automatically when it grows. None of them sends sampling parameters, so decoding is whatever the provider does by default. We hold one budget fixed across all three: a step is one model call, and every trial gets 300 of them. Wall-clock limits are a safety net, and on the locally served Qwen arms they are not equal: Claude Code, whose steps against vLLM run about twice as slow, could run for 36,000 s where OpenCode’s own soft timeout stopped it at 5,760 s (Appendix A). On the GLM model, where the API accepts a reasoning-effort setting, all three harnesses pin it to high. Everything else is left as the harness authors set it, and Table A1 in the appendix lists what that is. We also run DeepSeek’s own command-line harness, dsh in the tables, on DeepSeek’s model as a home-field check.

Models. The ladder has five models that run in all three harnesses: Qwen3.6-35B-A3B and Qwen3.8-27B, served locally with vLLM 0.27.1 in bfloat16 precision, and DeepSeek-V4-Flash (release 0731), GLM-5.3-Flash, and HY4-Preview, called through their vendor APIs (DS Flash, GLM Flash, and HY4 for short in figures and wide tables). The three API models are among the most used on OpenRouter: in its trailing-30-day ranking by tokens processed through 2026-09-20, DeepSeek- V4-Flash-0731 ranks second, HY4-Preview third, and GLM-5.3-Flash fourth (OpenRouter, 2026). Claude Opus 5 runs inside Claude Code only, through the harness’s own channel, and anchors the top of the ladder. Harbor 0.22.0 (Harbor Framework Team, 2026) orchestrates the runs, one container per trial, and runs the benchmark’s own tests to score each one.

Tasks. SWE-bench Verified (Jimenez et al., 2024; OpenAI, 2024) has 500 tasks, and 8 of them need a public web service to verify and are dropped, leaving 492. From these we take two disjoint sets. hard45 is every task in the two hardest annotated difficulty bands, 1 to 4 hours and over 4 hours, 42 and 3 tasks, taken whole so that there is no sampling to tune. pool447 is the remaining 447 tasks. Every arm on hard45 runs the same 45 tasks. On pool447 we run the two Qwen models, which we serve ourselves. The agent phase has no network beyond the model endpoint. An audit of our early web-enabled runs found that 71% of DeepSeek and Qwen trials had fetched the upstream fix from GitHub, so those runs are discarded and the condition is enforced at the network layer (Appendix D). Offline, OpenCode’s search tools fail because they download a binary at first use. We place it ahead of time and report that arm, marked rg in the tables.

Runs and scoring. The quantity we estimate is the difference in pass rate between two harnesses on the same model over the tasks of a set, for one run of each configuration. The reported score of a cell is its first scheduled run of every task, a rule fixed before the results were read. A trial is

Discussion. What a harness buys, on this evidence, is fewer ways to lose a task, and a bill, not pass rate. Every harness effect we could measure is a way to lose a task the model would otherwise solve: an output cap with no recovery, a context window with no compaction, a search tool that does not run offline. We did not find a harness winning tasks, which puts the engineering where the losses are. The one change that might move a score in our grid is a recovery step for OpenCode’s output cutoff, which Claude Code already has. Transplanted, it recovered tasks at a rate that 41 tasks cannot tell from zero, because the runaway reasoning behind the cut recurs on only about 15% of reruns of the same task. The three harnesses differ most in the parts that get the most attention, the prompt, the tool surface, the compaction policy, and none of those differences buys a score gain.

The thinking interaction is the one lead we would chase. Switching thinking off on Qwen3.6- 35B-A3B costs mini-SWE-agent 11 points and moves the other two by less than the noise. On HY4-Preview, however, the same switch inside Claude Code costs 11 points, as much as the widest harness gap on that model, and cuts output tokens by more than half. The two results suggest that dependence on explicit reasoning is specific to the model and harness pair, but our grid is too small to identify the mechanism.

Conclusion. Swapping the harness under a fixed model moves pass rate on SWE-bench Verified by as much as rerunning the same harness on the 45 hardest tasks, across five models. On the 447-task pool, the pairs that clear the noise are all OpenCode trailing, by 8 to 9 points on Qwen3.8-27B, tied to how it ends a trial. What the harness does decide is the bill, by up to 3× on the same model, through what it carries into each step and how many steps the model takes, and the cache price scales that bill without reordering it. The numbers to carry between vendors are the first call, the growth per step, and the step count. At the discordance we observe, a harness comparison on 45 tasks catches a 12.9-point gap half the time and nothing at 80% power, and on 447 tasks it resolves 5.2 points. The cheapest control in such a comparison is a second run of the same cell, the one most comparisons skip, and we recommend reporting it next to any harness gap.

Limitations. Two limits on scope. Our results come from one benchmark. The one-model check on Terminal- Bench 2.1 (Appendix L) is a transfer check, and general-purpose agents may be a different regime. And the closed anchor model runs in Claude Code only, so the three-harness ladder stops at HY4- Preview and cannot tell whether a closed frontier model shows the same gaps.