Can agent harnesses be automatically optimized across many environments?
Explores whether scaling auto-research loops across diverse harness environments can discover mechanisms that reduce token use without sacrificing task performance, and whether such discoveries generalize.
SoL-Pi treats the agent harness, not the model or the serving stack, as the object of automated search. The authors scale "auto-research loops across increasingly numerous and diverse environments for harness rollouts" and report that four mechanisms "survive selection": action execution, context compaction, observation handling, and delegated reading. On the 51-task EdgeBench evaluation the assembled stack reaches performance comparable to the Pi harness on both GPT-5.6 Sol and Opus 5, while recorded token traffic falls by 44.7–49.0% and API cost by about one third. The best individual candidates are reported to improve model performance by 5.3–12.8% and token efficiency by 9.8–18.2%.
The paper's reasoning is that long unattended coding-agent trajectories make token use "a first-order systems concern," and that existing efficiency work (faster kernels, quantization, cheaper models) leaves the harness untouched. Harness-level optimization is framed as orthogonal to those, since it needs no additional model training. The scale argument is the distinctive part: harness artifacts found in one setting tend not to generalize, and the authors take their cross-model consistency as "preliminary evidence" that exposing the harness to many tasks and updating it from the resulting trajectories mitigates this. They call the long-term direction "pretraining the harness" and state it as a hypothesis, not a result.
Against the neighbors, this is a search-discovered counterpart to hand-designed harness efficiency. Can explicit behavior maps help weaker planners compete with stronger models? also cuts planner tokens through harness structure, but by a human-specified representation, where here the mechanisms are selected by the loop. It is consistent with Does raw token spending actually predict agent performance?: if raw tokens explain little of success, then removing nearly half of them at comparable performance is what that account would allow. It also gives Where does agent reliability actually come from? a cost-side datum, since the gains sit in the harness layer. Two model families are a thin test of the claim in Do stronger models always evolve harnesses better?, which is about who writes and who uses updates, and this excerpt does not vary either role.
The excerpt does not establish how selection works, how many environments or loop iterations were run, or what each of the four mechanisms does beyond its name. It gives no ablation showing which mechanism carries the savings, and it does not define "comparable" statistically. The dollar savings are described as "estimated." Generalization is claimed only "within the evaluated setting," so transfer to other benchmarks, harnesses or weaker models is untested here. The defensible reading is that a scaled harness search found a large token saving without a visible performance loss on one benchmark, not that harnesses can now be reliably pretrained.
Inquiring lines that read this note 91
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can AI systems achieve sustainable self-improvement without external human guidance? What evaluation metrics beyond task success matter most for agents?- How can agent data flywheels improve task quality iteratively?
- What within-run behavioral dimensions reveal where long-horizon agents succeed or fail?
- Do agent behavior logs reveal hidden accuracy costs benchmarks miss?
- Can accuracy rankings be reordered by tracking agent behavior costs?
- Can runtime behavior mapping help localize harness deficiencies?
- How do prompt optimization and code harnesses compare for capability transfer?
- What makes a harness low-friction for model strategy?
- Which domains see models exceed human harness design quality?
- Why do useful harness updates often disappear during model evolution?
- How much does executor choice change a harness's actual performance?
- Do models co-adapt their harnesses to specific executor strengths?
- Can mid-tier models benefit more from harness improvements than frontier models?
- Does harness scaling represent a fundamentally different path than model scaling?
- How much does harness design contribute to reported model capability scores?
- How does harness structure affect planner token efficiency compared to model size?
- Do harness improvements eventually internalize into core model behavior over time?
- Can a harness design be optimized independently of the model inside it?
- How do evolved harness edits generalize across different benchmark domains?
- Do task-level outcomes provide sufficient supervision for harness evolution?
- Can harness edits distill reusable strategies or mostly memorize task-specific fixes?
- How much realized agent capability comes from the harness versus the model?
- Can harness edits trained on one batch transfer to new tasks?
- How do agentic systems hide harness failures from benchmarks?
- What role does effective feedback compute play in agent harness scaling?
- Does harness optimization generalize across different benchmarks and agent architectures?
- Can harness evolution gains be distinguished from test-time search improvements on matched budgets?
- Can harness edits overfit to training tasks while appearing to improve performance?
- Why do research agents optimize harness mechanisms over autonomous weight scaling?
- How much token efficiency can harness-level intervention achieve versus training approaches?
- Can harnesses that rewrite themselves through reviewed commits achieve state-of-the-art performance?
- Can better agent harnesses solve problems that model scale alone cannot?
- How do scaffolding improvements change agent capability across different task types?
- What explains the threefold cost differences between harnesses?
- Do cached input prices change which harness costs less?
- Can individual harness components improve accuracy on specific benchmarks?
- Do harness-driven accuracy gains appear on other benchmarks?
- How does cost stability in harness swaps compare to accuracy stability?
- What task set size resolves a real 5-point harness difference reliably?
- Why do vendor harness accuracy claims lack rerun verification?
- How are exploration policies refined against the simulator in practice?
- Can reinforcement learning agents optimize better on automatically scored tasks?
- What makes an agent mechanism reusable versus benchmark-specific?
- Do agents actually convert raw experience into better behavior automatically?
- Can autonomous agents detect increasingly sophisticated specification gaming as they improve?
- Does source bias affect real deployed agents or only benchmark environments?
- How much does agent scaffolding improve AI model self-replication success?
- Does constrained, efficient agent execution actually perform better in practice for bullshit work?
- Does the fixed cost of agent optimization pay off in realistic deployment scenarios?
- How do learning loops and user feedback improve agent performance over time?
- Can deterministic function calls replace agent-mediated tool selection for reliability?
- How do variance and standardization guide automation type selection?
- How does targeted fine-tuning compare to orchestration for improving agent reliability?
- Can agentic AI handle consequential travel decisions without first solving hallucination?
- Can interface design reduce an agent's adaptation burden more than model improvements?
- Does episode-level cost become the decisive factor when comparing AI agents in production?
- What distinguishes wasteful token spending from genuinely productive AI agent use?
- How much of token demand growth comes from context length versus user volume?
- What inference costs does running AI agents actually incur daily?
- How much cheaper do agents become when jointly optimizing cost and accuracy?
- How does the cost of information acquisition influence AI agent decision-making behavior?
- How do token-count variations affect cost estimates as AI workflows become more agentic?
- Why do agentic AI workflows increase token volume faster than prices fall?
- Can context management be optimized for an agent without retraining or changing the model?
- Can agents manage context through active delegation instead of progressive disclosure?
- How do organizational memory and validated plans reduce agent replanning overhead?
- Can agent reliability and environment noise level be tuned together as one compression dial?
- Can agents learn to compress verified evidence and unresolved constraints into a compact improvement state?
- How do fixed external benchmarks anchor self-improving agent systems?
- Does codifying domain rules into agent scaffolding work at library scale?
- When do agents benefit most from reusable workflow routines?
- How should agentic workflows cost per artifact instead of per token?
- Does Amdahl's law or partial substitutability better model research task automation?
- When does data collection hit diminishing returns in production AI systems?
- Are production AI agents deliberately constrained in real use?
- How is Meta's internal token usage driving AI infrastructure decisions?
- Do bounded tasks let AI close performance gaps but open tasks widen them?
- Does ARC performance on novel tasks predict real-world usefulness?
- Can we measure adaptability and environmental complexity in real-world AI deployments?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can explicit behavior maps help weaker planners compete with stronger models?
Explores whether organizing harness repositories around runtime behavior—rather than relying on model inference—can narrow the capability gap between weaker and stronger planning models, and whether this reduces computational overhead.
another harness-level route to fewer planner tokens, hand-designed rather than found by a search loop
-
Does raw token spending actually predict agent performance?
Standard measures of agent effort—tokens, tool calls, operations—may not capture what makes inference-time scaling work. This explores what actually drives performance gains when agents spend more compute.
raw token counts predict success poorly, which fits a large token cut at comparable performance
-
Where does agent reliability actually come from?
Exploring whether LLM agent performance depends on larger models or on thoughtful system design choices like memory, skills, and protocols that shift cognitive work outside the model.
supports locating gains in the harness layer, here measured as token and cost savings
-
Do stronger models always evolve harnesses better?
We explore whether base model capability predicts both the ability to write useful harness updates and the ability to benefit from them. The answer reshapes how we should allocate capability in self-evolving agent systems.
same self-evolving harness setting, but that note varies model tier and this excerpt varies only two models
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness
- Rethinking the Evaluation of Harness Evolution for Agents
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Prime Agent: A Self-Improving RLM Harness
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
Original note title
scaling auto-research loops across many harness environments yields mechanisms that cut token traffic nearly half at comparable performance