Can explicit behavior maps help weaker planners compete with stronger models?
Explores whether organizing harness repositories around runtime behavior—rather than relying on model inference—can narrow the capability gap between weaker and stronger planning models, and whether this reduces computational overhead.
If Why is finding distributed behavior code so hard?, the fix is to build the missing map explicitly rather than make an agent recover it every time. Harness Handbook is a behavior-centric representation: it links behavioral descriptions to the distributed source that implements them, constructs that mapping directly from source, and supports editing through behavior-guided progressive disclosure plus automatic resynchronization when the code changes.
The results argue that the mapping — not raw model strength — was doing the work. With the handbook, overall win rates rise 10.0 and 18.9 points on Codex and Terminus-2 while planner token use falls 12.7% and 8.6%: better plans for less compute, because the planner no longer burns tokens rediscovering where behavior lives. The sharper evidence is substitutive: with the handbook a weaker planner matches the implementation-site localization of substantially stronger models, improving every file- and symbol-level Recall/Precision/F1 comparison against two independent reference plans, and the gains persist across request types and difficulty levels.
This is a concrete instance of a recurring pattern in the harness literature — capability relocating out of the model and into the scaffolding around it. Like Can externalized bookkeeping let smaller search agents beat larger ones?, the move is to externalize a structure (here, the behavior→code map) so the model spends its budget on judgment rather than bookkeeping. It also complements What makes agent-authored code worth persisting and sharing?: a durable, resynchronized representation is exactly the kind of persistent artifact that compounds.
Inquiring lines that read this note 15
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do harness improvements transfer across model scales or memorize shortcuts?- Can harness updates benefit agents equally across all model sizes?
- How should harness scaffolding be treated as a first-class object?
- What makes harnesses more tangled than other types of agent code?
- Why do mid-tier models benefit more from memorized harness shortcuts?
- Can harness evolution be redirected toward distilling transferable procedures instead?
- Why do evolved harness edits mostly memorize rather than generalize?
- What feedback signals matter most during harness evolution search?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why is finding distributed behavior code so hard?
When developers need to modify agent harnesses, they struggle to locate all the code implementing a target behavior because behaviors are scattered across files and stages while requests describe what to do, not where to look.
the problem this representation solves
-
Can externalized bookkeeping let smaller search agents beat larger ones?
Does offloading routine record-keeping to an environment harness free RL policies to focus on semantic search decisions, and can this approach outperform larger searchers with fewer parameters?
same externalize-structure-so-the-model-does-judgment pattern
-
What makes agent-authored code worth persisting and sharing?
Agent-created artifacts like patches, tests, and skill libraries outlive single tasks, but we lack guidance on what should persist, how to maintain consistency across agents, and when persistence is worth the engineering effort.
a persistent, compounding harness artifact
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- Code as Agent Harness
- AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
Original note title
reorganizing a harness repository around runtime behavior lets a weaker planner match a stronger model's code localization