SYNTHESIS NOTE
Topics›Evaluations›this note

Can execution harnesses lift model performance without retuning weights?

Explores whether improving an agent's runtime system—not the model itself—can boost benchmark accuracy and transfer across different model versions without modification.

Synthesis note · 2026-09-25 · sourced from Evaluations

The StateM paper reports that improving the execution system around an agent, "without changing its model weights," raises benchmark accuracy across several models. On Terminal-Bench 2.1, GPT-5.5 xhigh with StateM reaches 92.1% against an 83.1% reference, above GPT-5.6 Sol Ultra at 91.9%. The same runbook "transfers to GPT-5.6 without modification," where the system records 95.3% raw accuracy across 445 public-submission trials and succeeds at least once on each of the 89 tasks. The frozen profile lifts GPT-5.6 Luna from 76.7% to 85.4%, numerically above the 84.9% Sol xhigh reference. Starting from the same runtime, runbook structure, and golden rules, under $38 of adaptation cost moves DeepSeek-V4 Flash from 82.7% to 88.1% under standard timeouts.

The paper's diagnosis is that long-horizon agents "can fail even when their underlying models can solve the constituent steps." The failures it lists are losing track of mutable state, failing to reactivate lessons from earlier executions, skipping known procedures, and stopping prematurely. StateM answers with durable states, phase-local context, checked transitions, recoverable runbooks, and versioned procedural practices that agents and users can inspect together. The discussion adds a design requirement: the runtime must be "agent-native and enforceable" and also cheap to modify. Conventional state-machine and graph runtimes push reasoning into manually constructed nodes, edges, and handlers, so changing the control policy becomes "an engineering task of its own" and failure-driven iteration gets heavy.

The paper frames harness scaling as orthogonal to model scaling, asking whether "more of a model's existing capability" can be converted into completed, reliable work. That is the thesis of Where does agent reliability actually come from?, and StateM adds a benchmark-level result and a concrete mechanism: state, procedures, and lessons kept outside the model and enforced at transitions. Its named failure of reactivating earlier lessons echoes Does raw token spending actually predict agent performance?, where feedback counts only if it is retained for later decisions. Its premature-stopping failure is the same behavior What predicts success in ultra-long-horizon agent tasks? finds in most frontier models. On transfer, it complements Do stronger models always evolve harnesses better?: gains appear for Sol, Luna, and DeepSeek-V4 Flash, but the excerpt does not order these by capability tier, so it neither confirms nor contradicts a mid-tier peak.

The excerpt is silent on several points that matter for the claim. It shows no ablation, so it cannot say which of the five components produces the gains. It reports no variance, and the 92.1% versus 91.9% comparison is a 0.2-point difference. It does not say what the 83.1% and 84.9% references consist of. It lists only two of the three "progressively stronger empirical tests." The "$15 frontier run" in the title gets no detail, and transfer is shown only within the GPT-5.6 family plus a fresh adaptation for DeepSeek. It reports accuracy and cost, not trajectory quality, so it cannot speak to How should we measure agent system performance beyond task success?. At the strength the excerpt supports, the results are consistent with control-layer failures being a real share of what a capable model loses on long tasks, and the full paper is needed to say which parts of the harness do the work.

Inquiring lines that read this note 161

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How do standardized benchmarks differ from measures of real-world AI capability? Why do AI agents struggle to generalize and improve reliably? What evaluation metrics beyond task success matter most for agents? Do harness improvements deliver capability gains comparable to model scaling? Do evolved harness optimizations generalize beyond their source benchmarks? Do fine-tuning methods modify core model capabilities or just outputs? Can external memory modules outperform model weight consolidation for long-term retention? Do single-axis benchmarks adequately measure deployment-ready agent capability? Why do autonomous agents report success despite actually failing? How should agents manage memory consolidation and reuse over time? How does the generation-verification gap constrain AI self-improvement capabilities? What causes models to fail in specialized domains versus generally? Should multi-agent systems use structured protocols or natural language communication? How should GUI agents perceive and represent user interface structure? Can recurrence enable transformers to solve problems beyond fixed-depth limitations? Can prompting inject new knowledge into models or only activate existing capabilities? Should AI systems make failures transparent so users can contest and adjust? Do safety evaluations reliably measure real deployed AI safety risks? What evidence best independently verifies agent behavior for safety? Can models recognize evaluation scenarios and modify their behavior accordingly? How should evaluation expand to capture deployment realities? What organizational factors determine whether firms successfully adopt and integrate AI? Do small language models have fundamentally different scaling properties than large ones? Can sequences of locally safe agent actions violate system-level constraints? How should persistent agent-authored code be governed and shared? How does training data produce emergent misalignment in base models? Can AI systems achieve sustainable self-improvement without external human guidance? How should evaluation environments be designed as security boundaries? What unintended behaviors emerge in frontier AI models? How should agent identity and authorization work across organizational boundaries? How do inference costs, efficiency, and market competition shape AI adoption economics? Do AI shopping agents exhibit systematic bias in recommendations? How do multi-agent systems balance consensus-building with error detection in collaborative reasoning? Can AI-generated text be reliably distinguished from human writing? When do multi-agent systems outperform single-agent approaches for complex task execution? Does AI coding assistance improve productivity and code quality across experience levels? When do complex LLM tasks benefit from external verification or structured scaffolding? How do clinicians assess, trust, and effectively use medical AI in practice? How can merchant-funded agents remain loyal to consumer interests in digital commerce? How does interface design shape effectiveness in human-AI collaboration?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 102 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

harness scaling lifts fixed models on Terminal-Bench 2.1 without touching weights — the same runbook transfers to a newer model unchanged