Can harness modules improve separately from benchmark data?

The paper treats harness self-improvement as a design problem with three named obstacles, and its framework is a three-part answer. The abstract says existing approaches often evolve harnesses "directly on evaluation benchmarks or subsets drawn from them," which makes reusable improvements hard to tell from benchmark-specific adaptation. It says updates drawn from individual trajectories "entangle systematic harness deficiencies with instance-specific reasoning and solution details." And it says recurring deficiencies are hard to localize within "a monolithic harness," so whole-harness optimization yields changes that are "difficult to attribute and validate." ModularRSI is described as "benchmark-disjoint, contrastive, and modular." The pairing of each obstacle with one design element is my reading of the abstract's order, not something the excerpt states.

The mechanism, in the paper's terms: the method "contrasts successful and failed trajectories for the same task and aggregates evidence across tasks to identify recurring behavioral deficiencies," then evolves harness modules independently. The introduction motivates this by noting that task-level outcomes "provide only coarse supervision" for harness evolution. Holding the task fixed while comparing outcomes is the evident purpose of the contrast, though the excerpt does not spell out that argument. The introduction adds a data-level challenge: harness evolution needs executable long-horizon terminal tasks with reliable environments and correctness feedback, which are costly to build at scale. The conclusion claims that evolving modules independently "identifies reusable mechanism-level improvements while reducing task-specific adaptation."

Against the nearest notes, this paper states the same worry as a design constraint instead of a finding. Do harness edits learn reusable strategies or memorize task fixes? diagnoses after the fact that edits persist per-task patches. ModularRSI's "task-specific modifications that transfer poorly" is that problem, and its answer is to prevent it at the source through disjoint evolution data and cross-task aggregation. It proposes a remedy and does not rebut the diagnosis. Do AIDE2's improvements transfer to unseen tasks? applies a similar check to a different object, agent code, and like this excerpt reports transfer with no scores. The localization obstacle echoes Can explicit behavior maps help weaker planners compete with stronger models?, where mapping behavior to its distributed source is the bottleneck. The "foundation models" claim sits beside Do stronger models always evolve harnesses better?, but the excerpt does not say which models, so it cannot confirm or contradict that peak.

What the excerpt does not establish is the size of any gain. The conclusion reports "consistent gains across unseen tasks, domains, and foundation models" on TerminalBench 2.0 and SWE-Bench Verified, with no scores, baselines, task counts or model names. The authors state that they ran no ablation isolating contrastive trajectory analysis, and that the main evolution experiments use only a subset of the 2,000 curated evolution instances. The excerpt also gives no comparison that separates the modular design from benchmark-disjointness, and no matched-budget test-time-search baseline. So the three-part design is best read as the authors' hypothesis about what makes harness gains transfer, supported by "trajectory analysis and case studies," and not yet as a result showing which element does the work.