Why does one AI coding tool claim it beats another, when running the exact same test twice can flip the result just as much?
Why do vendor harness accuracy claims lack rerun verification?
This explores why claims that one coding-agent harness (the scaffolding of prompts, tools and control loops wrapped around a model) beats another often don't hold up, and what checking them properly would take. The corpus explains what goes wrong without rerun checks better than it explains why vendors skip them.
This explores why claims that one agent harness beats another so often go unchecked against simple reruns, and what that check would show. One point up front: the corpus doesn't document vendors' reasons directly. What it does show is how easily rerun noise can be mistaken for a real advantage, and how much work it takes to tell the two apart.
The main finding is that randomness is about as large as the effect being claimed. On SWE-bench Verified, rerunning the same harness flips individual task outcomes about as often as switching to a different harness. With the 45-task sets common in comparisons, a real 13-point accuracy gap is detected only about half the time, and no gap at all can be detected with the standard 80% reliability Do harness comparisons on SWE-bench Verified measure real differences?. So a single run reporting 'our harness scores 8 points higher' may be measuring luck. When researchers ran this check across three production harnesses and five models, accuracy differences stayed within rerun noise. What actually changed was cost: token spend varied by up to three times, driven by fixed overhead in system prompts and tool definitions multiplied by the number of steps Does switching between coding harnesses change model accuracy?. The real difference between harnesses may be on the bill, not the leaderboard.
A second reason claims slip through is that the right comparison is hard to set up. Work on automatic harness evolution argues that any gain has to be measured against plain test-time search given the same feedback and compute budget. Otherwise you can't tell clever design from extra attempts How should we measure gains from automatic harness evolution?. When researchers looked inside evolved harnesses, most edits turned out to store task-specific fixes the agent could have found again in a single try, rather than reusable strategies Do harness edits learn reusable strategies or memorize task fixes?. A harness can look better on a benchmark because it has partly memorized that benchmark. One countermeasure is to improve harness modules on data kept separate from the benchmark Can harness modules improve separately from benchmark data?. Large cross-environment searches report their wins as token savings at roughly equal performance, not as accuracy jumps Can agent harnesses be automatically optimized across many environments?.
The less obvious point is that a single benchmark score is a weak kind of evidence to begin with. The reward-hacking literature makes this case from another direction. Current defenses are task-specific patches that leave no portable record showing a given run stayed within the rules Do current reward-hacking defenses provide reusable evidence of safety?. Even published hack rates can rest on labeling methods the paper never discloses How were reward hacks labeled in this benchmark study?. BenchShield proposes replacing the bare number with a checkable claim based on recorded infrastructure evidence of how the task was completed Can infrastructure evidence replace terminal scores in benchmark validation?. The common thread: repeated measurement is a lever for reliability in its own right. The same idea appears in research treating verification as a separate axis you can scale through repeated evaluation Can verification accuracy scale without training models?. One caution applies: agreement across runs catches randomness but not errors a system makes the same way every time Can agreement across samples reveal when models are wrong?.
To sum up, a harness accuracy claim without a rerun baseline can't separate design from chance. On current evidence, the honest headline for most harness swaps is 'same accuracy, different cost.' Reruns rule out noise, but they are only a minimum standard; ruling out benchmark memorization or rule-bending needs separate held-out data or records of how each run was completed.
Sources 11 notes
On SWE-bench Verified, rerunning the same harness flips task outcomes as often as swapping harnesses. A 13-point accuracy gap is caught only half the time on typical task set sizes, requiring a rerun baseline to verify any claimed harness advantage.
Across three production harnesses and five models on SWE-bench Verified, switching harnesses shifted pass rates within normal rerun variation but changed token cost up to threefold. The cost difference came from fixed overhead in system prompts and tool schemas, multiplied by step count.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
ModularRSI evolves harness modules independently using contrastive trajectories on benchmark-disjoint data, showing consistent gains across unseen tasks and domains. The approach isolates mechanism-level improvements from task-specific adaptation by aggregating evidence across tasks before updating components.
Show all 11 sources
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
Existing defenses rely on task-specific patches, prompt instructions, or post-hoc detectors, but none provide reusable evidence that a concrete run remained within its evaluation boundary. Even working defenses do not give operators a portable record of integrity.
Reported hack rates (57.2–73%) and detection gaps (3.1–7.9%) rely on unstated labeling criteria. Without knowing whether hacks were identified by human review, LLM judges, checkable answers, or infrastructure records, the reliability and meaning of these measurements cannot be assessed.
BenchShield enables benchmark operators to issue claims about valid task completion grounded in recorded infrastructure evidence rather than terminal scores alone. This shifts from a single number to a verifiable claim about whether an agent followed the intended evaluation path.
Research shows verification accuracy improves independently via score granularity, repeated evaluation, and criteria decomposition—all deployable at inference without retraining. This reframes weak verifiers as under-scaled rather than fundamentally limited.
The Consistency Veto suppresses answers that vary across samples but cannot detect systematic errors the model repeats identically. It carries strong signal on some queries but inherits a fundamental blind spot: agreement looks like confidence even when both are wrong.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Rethinking the Evaluation of Harness Evolution for Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Sharpening Tax in Post-Training
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
- What Does a Harness Buy? Tokens, Mostly
- Harness Engineering for Self-Improvement