Can execution harnesses lift model performance without retuning weights?
Explores whether improving an agent's runtime system—not the model itself—can boost benchmark accuracy and transfer across different model versions without modification.
The StateM paper reports that improving the execution system around an agent, "without changing its model weights," raises benchmark accuracy across several models. On Terminal-Bench 2.1, GPT-5.5 xhigh with StateM reaches 92.1% against an 83.1% reference, above GPT-5.6 Sol Ultra at 91.9%. The same runbook "transfers to GPT-5.6 without modification," where the system records 95.3% raw accuracy across 445 public-submission trials and succeeds at least once on each of the 89 tasks. The frozen profile lifts GPT-5.6 Luna from 76.7% to 85.4%, numerically above the 84.9% Sol xhigh reference. Starting from the same runtime, runbook structure, and golden rules, under $38 of adaptation cost moves DeepSeek-V4 Flash from 82.7% to 88.1% under standard timeouts.
The paper's diagnosis is that long-horizon agents "can fail even when their underlying models can solve the constituent steps." The failures it lists are losing track of mutable state, failing to reactivate lessons from earlier executions, skipping known procedures, and stopping prematurely. StateM answers with durable states, phase-local context, checked transitions, recoverable runbooks, and versioned procedural practices that agents and users can inspect together. The discussion adds a design requirement: the runtime must be "agent-native and enforceable" and also cheap to modify. Conventional state-machine and graph runtimes push reasoning into manually constructed nodes, edges, and handlers, so changing the control policy becomes "an engineering task of its own" and failure-driven iteration gets heavy.
The paper frames harness scaling as orthogonal to model scaling, asking whether "more of a model's existing capability" can be converted into completed, reliable work. That is the thesis of Where does agent reliability actually come from?, and StateM adds a benchmark-level result and a concrete mechanism: state, procedures, and lessons kept outside the model and enforced at transitions. Its named failure of reactivating earlier lessons echoes Does raw token spending actually predict agent performance?, where feedback counts only if it is retained for later decisions. Its premature-stopping failure is the same behavior What predicts success in ultra-long-horizon agent tasks? finds in most frontier models. On transfer, it complements Do stronger models always evolve harnesses better?: gains appear for Sol, Luna, and DeepSeek-V4 Flash, but the excerpt does not order these by capability tier, so it neither confirms nor contradicts a mid-tier peak.
The excerpt is silent on several points that matter for the claim. It shows no ablation, so it cannot say which of the five components produces the gains. It reports no variance, and the 92.1% versus 91.9% comparison is a 0.2-point difference. It does not say what the 83.1% and 84.9% references consist of. It lists only two of the three "progressively stronger empirical tests." The "$15 frontier run" in the title gets no detail, and transfer is shown only within the GPT-5.6 family plus a fresh adaptation for DeepSeek. It reports accuracy and cost, not trajectory quality, so it cannot speak to How should we measure agent system performance beyond task success?. At the strength the excerpt supports, the results are consistent with control-layer failures being a real share of what a capable model loses on long tasks, and the full paper is needed to say which parts of the harness do the work.
Inquiring lines that read this note 161
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do standardized benchmarks differ from measures of real-world AI capability?- How do benchmark environments misrepresent deployment readiness?
- How fast do new benchmarks get adopted across the AI research community?
- What distortions do automated benchmarks introduce compared to real tasks?
- How much do internal benchmarks differ from independent model evaluations?
- Can optimization metrics hide actual versus apparent progress in AI systems?
- How do lab-scale benchmark tasks differ from real frontier AI research?
- How do narrow benchmark optimizations differ from genuine architectural research discoveries?
- What gap exists between AI model capability in benchmarks and real client work?
- Do bounded tasks let AI close performance gaps but open tasks widen them?
- How much does model publication lag behind actual training completion?
- Could deliberate training toward benchmarks overstate real gains in AI capability?
- How much do current chatbot benchmarks actually measure real model quality?
- How can benchmarks optimize for both accuracy and computational cost simultaneously?
- Do open source AI models actually match closed models on standard benchmarks?
- Why do AI benchmarks with verifiable rewards miss important capabilities?
- Why do AI agents outpace human benchmarks yet fail user acceptance?
- Do single-axis benchmarks systematically misrepresent what AI systems can actually do?
- What role does online RL play in scaling GUI agents?
- What makes an agent mechanism reusable versus benchmark-specific?
- Can corrected simulators replace real execution at inference time too?
- How do parametric and non-parametric updates differ in agents?
- Do agents actually convert raw experience into better behavior automatically?
- Do weight-free agent edits keep chain-of-thought observations meaningful?
- Does source bias affect real deployed agents or only benchmark environments?
- How much does agent scaffolding improve AI model self-replication success?
- Does constrained, efficient agent execution actually perform better in practice for bullshit work?
- Can distilled models maintain general agentic capability at very small scale?
- Does the fixed cost of agent optimization pay off in realistic deployment scenarios?
- How do learning loops and user feedback improve agent performance over time?
- What model capabilities must mature before agents reach production?
- Why do production AI agents typically execute fewer than ten steps?
- Does agent capability matter more than instructions in adversarial settings?
- How does targeted fine-tuning compare to orchestration for improving agent reliability?
- How does Wizard-of-Oz prototyping improve live agent refinement compared to post-hoc evaluation?
- Can interface design reduce an agent's adaptation burden more than model improvements?
- How can agent data flywheels improve task quality iteratively?
- Can gradients extracted with agent-level supervision transfer across different benchmarks?
- Should agent evaluation include trajectory quality and memory hygiene alongside task success?
- Can a single leaderboard score capture multi-dimensional differences in agent performance?
- Can accuracy rankings be reordered by tracking agent behavior costs?
- How do trajectory quality and memory hygiene affect agent evaluation?
- Which foundation model tiers most benefit from harness updates?
- How do prompt optimization and code harnesses compare for capability transfer?
- What should an external contract for model improvement actually contain?
- What makes a harness low-friction for model strategy?
- Can weaker models match stronger ones by reorganizing harness-side components?
- Which domains see models exceed human harness design quality?
- Why do useful harness updates often disappear during model evolution?
- How much does executor choice change a harness's actual performance?
- Do models co-adapt their harnesses to specific executor strengths?
- Can mid-tier models benefit more from harness improvements than frontier models?
- Does harness scaling represent a fundamentally different path than model scaling?
- How much does harness design contribute to reported model capability scores?
- How does harness structure affect planner token efficiency compared to model size?
- Can weaker models benefit equally from harness updates as stronger ones?
- How much external enforcement does each model need before utility drops?
- How does AI system design amplify model capabilities beyond the weights?
- Can scaffold-only modifications achieve lasting gains without updating the foundation model weights?
- How much does harness design separate from model intelligence affect benchmark scores?
- How much of AI improvement comes from tools versus model capability?
- How do evolved harness edits generalize across different benchmark domains?
- How much realized agent capability comes from the harness versus the model?
- How do agentic systems hide harness failures from benchmarks?
- What role does effective feedback compute play in agent harness scaling?
- Does harness optimization generalize across different benchmarks and agent architectures?
- Can harness edits overfit to training tasks while appearing to improve performance?
- Why do research agents optimize harness mechanisms over autonomous weight scaling?
- Can harnesses that rewrite themselves through reviewed commits achieve state-of-the-art performance?
- Can better agent harnesses solve problems that model scale alone cannot?
- How do scaffolding improvements change agent capability across different task types?
- Can individual harness components improve accuracy on specific benchmarks?
- Do harness-driven accuracy gains appear on other benchmarks?
- How much does scaffold choice interact with model selection in agents?
- How do fast skill injection and slow gradient updates work on different timescales?
- What makes frozen model reasoning different from weight-based parameter updates?
- What distinguishes fast non-parametric loops from slow parametric weight updates?
- How much do cross-model improvements compare to the original performance gains on the training benchmark?
- Which separable capability axes reveal when single benchmarks misrepresent deployment readiness?
- What realism standards should compliance benchmarks meet to avoid evaluation gaming?
- Can a single benchmark score capture both progress and readiness?
- Can a single agent benchmark score accurately represent deployment readiness?
- Why do single-axis benchmarks fail to measure deployment-ready agent capability?
- Can benchmark scores alone prove a model's exploitation capability?
- How do single-axis benchmarks misrepresent AI agent readiness for deployment?
- Can the benchmark-performance mismatch be estimated reliably without an oracle?
- Why do single benchmarks fail to capture readiness across separate capability axes?
- Do public benchmarks systematically overstate open model capabilities?
- Can single-axis benchmarks fully represent deployment-ready agent capability?
- How should benchmarks measure agent performance across multiple dimensions beyond accuracy?
- Why do model evaluation and downstream evaluation require different benchmark designs?
- Do current agent benchmarks confuse model development with deployment decisions?
- How do structured benchmarks differ from real-world software task performance?
- Can single benchmark scores predict whether a model is ready to deploy?
- What makes domain-specific agent evaluation different from benchmark-based testing?
- Can smaller models trained for execution handle the failure modes that stop current agents?
- Why do most AI agent solutions score near zero despite occasional breakthroughs?
- What obstacles prevent AI agents from reliably transferring data across legacy systems?
- How much instruction complexity causes agent performance to degrade?
- Can context management be optimized for an agent without retraining or changing the model?
- What makes execution and memory proxies more useful than dynamics prediction alone?
- How do modern agents separate fast non-parametric updates from slow weight learning?
- Can memory accumulation alone degrade agent safety without weight updates?
- What cache hierarchy strategies prevent model failures in long-horizon agents?
- Can compression aggressiveness be set automatically for new environments without benchmark testing?
- Do weaker agents need more aggressive compression than stronger ones to stay reliable?
- Can agent reliability and environment noise level be tuned together as one compression dial?
- Can agents learn to compress verified evidence and unresolved constraints into a compact improvement state?
- Can agents improve reliably without an external standard?
- How do fixed external benchmarks anchor self-improving agent systems?
- How do slow weight updates and fast prompt updates interact in self-improvement?
- Can large instruction sets damage a model's ability to follow behavioral rules?
- Why do isolated model benchmarks understate real-world AI security risks?
- Why do identical benchmark scores not guarantee deployment safety?
- Can outcome-only reporting hide whether agent safety comes from server-side filters or the model?
- What structural changes would make agent leaderboards trustworthy?
- What deployment conditions would prevent models from evading monitors?
- Does eval-gaming explain why models act different when tested versus deployed?
- Why did the PR lift persist longer than in comparable studies?
- How do statistical noise and revalidation bias speedrun benchmark claims?
- What structural advantages keep red teams ahead of increasingly capable models?
- Does buying third-party AI applications actually outperform custom internal builds?
- Can frontier-scale results be predicted from small-scale benchmark speedups?
- Are architectural differences between models more detectable than size differences?
- Do smaller models optimized for benchmarks perform worse than their scores suggest in real tasks?
- Why are consumer GPU open models advancing faster on LM Arena metrics?
- Where exactly does the threshold between model sizes stop mitigation from working?
- Why might API access to models produce better accuracy than consumer products?
- Do agent improvements discovered on code tasks transfer to non-coding domains as well?
- What makes shareable agent artifacts spread faster than shared platforms?
- How do frontier models handle network scanning and payload delivery together?
- Do single-axis security benchmarks miss agent failure modes in other dimensions?
- Why do model capability gains appear to regress between major releases?
- Does hardware scaling alone explain why open models stay within months of frontier capability?
- Can model release timing distort the measured slope of Elo progress?
- Why do open-weight models capture significant share if closed models score higher on benchmarks?
- Which open source models meet the threshold for viable on-premise deployment?
- How quickly do open models close the performance gap after closed model releases?
- How does rapid AI model change affect the economics of building versus buying?
- What inference costs does running AI agents actually incur daily?
- How universal is the state-of-the-art premium effect across AI benchmarks?
- How much cheaper do agents become when jointly optimizing cost and accuracy?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Where does agent reliability actually come from?
Exploring whether LLM agent performance depends on larger models or on thoughtful system design choices like memory, skills, and protocols that shift cognitive work outside the model.
states the harness thesis; this paper adds benchmark accuracy gains and a state-and-runbook mechanism
-
Does raw token spending actually predict agent performance?
Standard measures of agent effort—tokens, tool calls, operations—may not capture what makes inference-time scaling work. This explores what actually drives performance gains when agents spend more compute.
locates harness gains in retained, informative feedback; StateM's lesson reactivation and durable state fit that account
-
What predicts success in ultra-long-horizon agent tasks?
Does an agent's initial solution quality matter more than its willingness to iterate? AUTOLAB's frontier-model benchmark suggests persistence through feedback loops may be the true differentiator.
premature stopping appears in both as a failure of long-horizon agents
-
Do stronger models always evolve harnesses better?
We explore whether base model capability predicts both the ability to write useful harness updates and the ability to benefit from them. The answer reshapes how we should allocate capability in self-evolving agent systems.
same fixed-weights harness setting; StateM's cross-model gains are not ordered by tier, so the tier question stays open
-
How should we measure agent system performance beyond task success?
Current evaluation metrics collapse agent behavior into a single success score, hiding critical information about how agents operate. What dimensions—trajectory quality, memory use, context efficiency, verification cost—should benchmarks actually measure?
StateM reports accuracy and cost only, the scalar view this note says hides harness behavior
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- RSIGym: A Flexible Environment for Recursive Self-Improvement
- Sharpening Tax in Post-Training
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- What Does a Harness Buy? Tokens, Mostly
Original note title
harness scaling lifts fixed models on Terminal-Bench 2.1 without touching weights — the same runbook transfers to a newer model unchanged