If you rerun the exact same coding AI twice, how often does it just flip from pass to fail for no reason?

This explores how much SWE-bench outcomes shift when you simply rerun the same agent setup, and why that matters for anyone comparing setups. The short answer is: a lot more than most leaderboard comparisons assume. On SWE-bench Verified, rerunning the identical harness flips individual task outcomes about as often as swapping in a different harness does Do harness comparisons on SWE-bench Verified measure real differences?. A task that passed on Monday can fail on Tuesday with no change at all, because agent runs are long chains of sampled decisions and small early differences add up.

The practical consequence is about statistical power. With the small task sets people often use (around 45 tasks), a real 13-point accuracy gap between two harnesses gets detected only about half the time, and reaching the usual 80% confidence standard isn't possible at that size Do harness comparisons on SWE-bench Verified measure real differences?. So the right baseline for any "our harness is better" claim isn't the other harness. It's the same harness run again. If the gap you see is no bigger than what reruns produce, you haven't shown anything.

When researchers do apply that test, the result is surprising. Across three production coding harnesses and five models, switching harnesses moved pass rates no more than ordinary rerun variation, but it changed token cost by up to threefold Does switching between coding harnesses change model accuracy?. The cost gap came from fixed overhead (system prompts and tool descriptions) repeated at every step. Put differently, once rerun noise is accounted for, the main measurable difference between popular coding harnesses may be how much they cost to run, not how often they succeed.

This also applies to the growing body of work on harnesses that improve themselves automatically. If a single rerun can move scores that much, a self-improving loop can easily "discover" gains that are really lucky draws, or overfit to the particular tasks it was tuned on Does harness self-improvement memorize tasks instead of learning broadly?. That is why some researchers argue harness-evolution gains should only count if they beat an equal-budget baseline of plain retrying and searching How should we measure gains from automatic harness evolution?, and why approaches like ModularRSI test improvements on data kept separate from the benchmark Can harness modules improve separately from benchmark data?. Large reported jumps, such as harness fixes beating extra fine-tuning in RSIGym Do harness fixes or heavier training drive frontier model gains?, are worth reading with one question in mind: how big was the rerun spread?

One caveat: the collection measures this rerun noise directly only for SWE-bench Verified. It doesn't yet show whether other agent benchmarks are equally noisy, so it's safest not to assume they're any steadier.