If two AI coding assistants differ by 5 points on a benchmark, how many test tasks does it actually take to trust that gap?

This explores how many tasks a benchmark comparison needs before a real 5-point gap between two agent harnesses stands out from random variation. The corpus has a specific answer for one setting. On SWE-bench Verified, at the noise level one study measured, about 447 tasks are needed to resolve a 5-point difference. That is close to the whole 500-task benchmark. Smaller sets perform much worse: 45 tasks, a size many studies use, catch a much larger 13-point gap only half the time and can't detect any gap with the usual 80% reliability Do harness comparisons on SWE-bench Verified measure real differences?.

The number is so large because the noise is large. Run the same model with the same harness twice on the same tasks, and about 13% of tasks flip between pass and fail. That is as many as flip when you switch to a different harness, and a harness doesn't win the same tasks from one run to the next. So task count alone doesn't settle the question. What sets the required count is how much the tasks disagree between runs, and you only learn that by running the same setup twice. The authors call this second run 'the cheapest control' and recommend reporting it next to any claimed harness gap. Their 447 figure applies to this benchmark and these harnesses. A noisier or calmer setup would move it.

This calibration changes how a lot of results read. The same paper finds that switching between three production harnesses changes pass rates by no more than ordinary rerun noise. It does change token cost by up to 3x, which is large enough to clear the noise easily Does switching between coding harnesses change model accuracy?. Other studies report harness effects of a few points from single runs. One measures how much planning, action design and context management are each worth Which coding harness components matter most in different conditions?. Another reports large token savings with accuracy described as 'comparable' on a 51-task benchmark Can agent harnesses be automatically optimized across many environments?. By this calibration, accuracy differences of a few points on sets that small can't be resolved, while large cost differences can.

The same concern shows up in work on automatic harness improvement under other names. One argument is that an evolved harness has to beat plain test-time search given the same compute budget, or the gain may just be extra compute How should we measure gains from automatic harness evolution?. Another shows that self-improving harnesses can memorize the tasks they were tuned on, so gains seen on those tasks shrink on new ones Does harness self-improvement memorize tasks instead of learning broadly?. A rerun baseline, a matched-search baseline and a held-out test set are three answers to the same question: is this gain real, or would it disappear under a fair re-test?

In practice, a reported harness gain of under about 10 points on a few dozen tasks, from a single run, tells you little about accuracy. To show a 5-point gain on SWE-bench Verified, you would run nearly the whole benchmark, run it at least twice, and report how much the reruns disagree. What harness changes reliably affect is cost, which is easier to measure than accuracy.