SYNTHESIS NOTE
Topics›Agent Harness›this note

Do stronger models always evolve harnesses better?

We explore whether base model capability predicts both the ability to write useful harness updates and the ability to benefit from them. The answer reshapes how we should allocate capability in self-evolving agent systems.

Synthesis note · 2026-06-03 · sourced from Agent Harness

Self-evolving agents edit an external harness — prompts, skills, memories, tools — from execution evidence, without touching model parameters. The natural assumption is that stronger base models do this better on both ends. This paper disentangles two distinct capabilities and finds neither follows that assumption.

Harness-updating — producing persistent edits that lead to gains — is flat in base capability. Models across capability tiers produce updates yielding surprisingly similar gains; even a Qwen3.5-9B evolver induces gains comparable to Claude Opus 4.6. Writing a good skill or memory is apparently not bottlenecked by raw model strength.

Harness-benefit — actually improving when handed an updated harness — is non-monotonic. Weak-tier models gain little, mid-tier models benefit most, and strong-tier models benefit less than mid-tier. Two failure modes explain the weak end: failing to activate the relevant harness artifact, and failing to follow it faithfully once activated.

The practical inversion is sharp: invest capability budget in the agent that uses the harness, not the evolver that writes it — and target agent training at harness invocation and long-horizon instruction-following rather than at generating cleverer updates. This complicates the "let a frontier model improve everything" intuition and connects to Why do better reasoning models ignore instructions?: strong models may benefit less precisely because the bottleneck is faithful instruction-following, which scaling erodes.

Inquiring lines that read this note 150

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Can harness optimization rival model scaling in effectiveness? Does evolutionary optimization at test time improve model performance effectively? How does model scale affect fine-tuning and downstream task performance? Can AI agents learn to improve performance from experience? What determines when personalization helps versus harms model behavior? Which agent-created code should persist in infrastructure versus remain task-specific? Do single-axis benchmarks adequately measure deployment-ready agent capability? How does transparency of agent reasoning affect effective oversight? How can agents learn and retain useful knowledge across diverse environments? Why does separating planning from execution enhance reasoning performance? How bounded is recursive self-improvement in current AI systems? How do different editing strategies affect output quality and agent safety? How should organizational structure guide multi-agent system design? How should context management strategies balance efficiency with agent reliability? Can individually safe agents collectively violate system-level constraints? Can post-hoc execution traces provide sufficient accountability without real-time oversight? Should agent evaluation measure execution quality beyond task success? How do multi-agent architectures affect security vulnerabilities and defense effectiveness? When does governance become the bottleneck for agentic AI? How do peer interactions drive collusion in multi-agent systems? How do frontier models develop behaviors that escape detection? How does model awareness of deployment context affect strategic behavior? Do benchmark score improvements actually reflect genuine AI capability gains? How can language models effectively manage long-context memory without quality degradation? How do standardized artifacts improve coordination in multi-agent systems? How should agents balance abstract planning with concrete environmental grounding? When do multi-agent systems outperform single agents? Can AI systems autonomously pursue unstated objectives or circumvent design constraints? What is the actual bottleneck preventing effective AI research automation? Should evaluation environments be secured as AI containment boundaries? What external mechanisms enable reliable self-improvement in language models? How does verifier integration affect the generation-verification gap? What organizational practices enable successful AI tool adoption? Does generative AI replace coordination with individual productivity? How can institutions effectively enforce controls on AI development? What fundamentally drives AI inference pricing and cost decline patterns? How does tokenizing intelligence reshape labor markets and institutional outcomes? Do standard benchmarks reveal AI capability in real-world contexts?

Related concepts in this collection 3

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 87 in 2-hop network ·medium cluster Open in graph ↗
Do stronger models always evolve harnesses bette… Where does agent reliability actually come from? Can a separate trained curator improve skill libra… How can agent self-evolution be made safe and audi…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why does instruction-following capability decrease as models scale stronger?How should we allocate model budget between evolvers and harness users?What causes weak models to fail at activating harness artifacts?Can we reverse the instruction-following deficit through targeted training?
Original note title

the capacity to produce useful harness updates is flat across model tiers but the capacity to benefit from them peaks at mid-tier