Do harness comparisons on SWE-bench Verified measure real differences?

Running the same harness and model twice on the same tasks moves the score almost as much as swapping to a different harness: "swapping the harness flips as many tasks as rerunning the same harness, 13% in both cases, and the tasks a harness wins in one run are not the tasks it wins in the next." The authors use their rerun data to calibrate what a harness comparison can actually resolve: "at the discordance we observe, 45 tasks catch a 13-point gap only half the time and no gap with 80% power, and 447 tasks resolve 5 points, still coarser than the gains many harness changes claim." That is, a comparison run on a task set the size many studies use would miss even a real double-digit-point harness effect more often than it would catch one.

The reasoning is a straightforward reproducibility audit of the comparison's own literature: "every study in this line runs each cell once or twice," so none of the prior harness-swap results the paper cites — six harnesses spanning 23.8 points, five harnesses spanning 27.4 points, a 2.4-point vendor claim — reports whether that spread would survive a second run. The paper's recommendation follows from its own numbers: "the cheapest control in such a comparison is a second run of the same cell, the one most comparisons skip, and we recommend reporting it next to any harness gap." The rerun is doing the job a confidence interval would do in a differently-designed study, but one computed from the comparison's actual noise rather than assumed.

This qualifies every single-run harness-value finding in the library rather than contradicting a specific one. Which coding harness components matter most in different conditions? reports accuracy deltas from varying harness components, and the excerpt gives no indication those cells were rerun; this note's calibration says a comparison of that kind needs a rerun baseline before its accuracy deltas can be distinguished from the kind of flip this paper found between identical reruns. Its sibling note from this same source, Does switching between coding harnesses change model accuracy?, is the direct application: the reason that paper can say pass-rate differences across harnesses sit "within noise" is exactly this rerun calibration, while the cost difference it reports is large enough (up to 3×) to clear that same noise floor.

The noise estimate is specific to this benchmark, these three harnesses, and this model ladder; the excerpt does not claim the same 13-point or 5-point resolution floor applies to other benchmarks, task-set sizes, or harness pairs, and a different discordance level would shift the numbers. What follows at the strength the evidence allows is a reading discipline, not a universal threshold: treat a reported harness-score gap as provisional until the comparison states whether it reran the same cell, and read vendor or leaderboard claims of harness-driven accuracy gains as unverified in the absence of that control.