Does upgrading an AI agent's tools and memory help every task equally, or do some jobs benefit far more than others?
How do scaffolding improvements change agent capability across different task types?
This explores whether improving the structure around an AI agent (its harness: memory, tools, skills, execution loops), rather than the model itself, helps equally across different kinds of tasks, or helps some kinds more than others.
This explores whether better scaffolding (the harness of memory, tools, skills and execution routines wrapped around a model) lifts agents evenly across task types, or helps some kinds of work more than others. The short version from the corpus: scaffolding gains are real, they often transfer, and they can be measured separately from model improvements. But how much an agent benefits depends less on the task category than on the model's tier and the kind of help the scaffold gives. A caveat first: the collection has no study that compares scaffold gains task type by task type, so part of this answer pieces the picture together from several directions.
Start with the strongest claim. In applied work, a growing share of agent reliability comes from the harness, not from model size Where does agent reliability actually come from?. On command-line tasks, optimizing only the execution system around frozen models raised Terminal-Bench scores across several models, and the same runbook carried over to newer models unchanged Can execution harnesses lift model performance without retuning weights?. Automated harness search across many environments found four general mechanisms that cut token use nearly in half without losing performance: how actions run, context compaction, observation handling and delegated reading Can agent harnesses be automatically optimized across many environments?. In ML and algorithm engineering, scaffold-driven gains even reached a held-out weather-forecasting benchmark outside the tasks used to tune them Do AIDE2's improvements transfer to unseen tasks?. So well-designed scaffolding isn't narrowly tuned to one task. Its benefits look largely separate from the model's own gains.
The less obvious finding: the model matters more than the task. Every model tier writes useful harness edits about equally well, but the ability to *benefit* from those edits peaks in mid-tier models Do stronger models always evolve harnesses better?. Weak models don't use the scaffolding. Strong models resist following it faithfully. What scaffolds actually provide also matters. In over 8,000 trials, skills mostly worked as procedural anchors that keep an agent's actions on track (65.7% of cases), and rarely as a source of missing facts (4.5%) Do skills teach procedures or inject missing facts?. That suggests scaffolding helps most on tasks where agents lose their way through long, multi-step procedures, and least on tasks where they simply lack knowledge.
There are also ceilings scaffolding can't break on its own. Tasks that need different kinds of expertise, parallel work and independent checking exceed what any single agent loop can organize, however good its harness Do single agents always hit organizational limits?. Those tasks call for multi-agent structures, with capability-based routing between agents Can semantic capability vectors replace manual agent routing?. On complex professional work, the strongest results came from training coordination and recovery as behaviors, alongside larger environments, rather than bolting them on as architecture Can agents learn to work reliably through environment and coordination scaling?. Static scaffolds learned from expert demonstrations carry their own cap: the agent can only be as capable as the curators imagined Can agents learn beyond what their training data shows?.
This is why research on self-improving agents is shifting toward the fast loop: updating prompts, memory and tools rather than weights, because scaffold changes are cheap and easy to undo Do self-improving agents really split into two distinct loops?. The next step is training the scaffold-maker itself. A separately trained skill curator moved skill libraries away from generic, wordy additions toward cross-task strategies, and it generalized across different executor models Can a separate trained curator improve skill libraries better than frozen agents?. The takeaway you might not have expected: the best scaffolds may be the ones that capture *how to approach* a kind of work, not what to do on a specific task, and their value depends on whether the model is the right size to use them.
Sources 12 notes
Applied AI research shows capability shifts from model weights to external structures like memory and skills. However, reusable skills bundle executable code and system reach, creating security costs that traditional lifecycle inspection cannot catch when attacks compose across multiple skills.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Show all 12 sources
Analysis of 8,135 trials shows procedural anchoring accounts for 65.7% of skill cases versus 4.5% for knowledge injection. Skills fail when retrieved incorrectly, invoked out of context, or followed too rigidly.
Research shows that real-world tasks requiring heterogeneous expertise, parallel execution, and independent verification exceed what any single agent loop can organize. Graph-based system abstractions are needed to distribute intelligence across specialized agents.
Versioned Capability Vectors embedded in HNSW indices couple semantic matching with policy and budget constraints, making capability discovery a first-class operation that scales sub-linearly as agent heterogeneity increases.
Apodex 1.1 demonstrates that agents trained via expanded executable environments and learned coordination behavior reach leading performance on complex professional tasks, even with smaller models. The approach achieves verifiable progress by treating state, recovery, and delivery as trainable behaviors rather than architectural requirements.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
SkillOS shows that separating a trainable curator from a frozen executor, grouped by task streams, causes skill repositories to shift from generic verbose additions toward actionable execution logic and cross-task meta-strategies. The trained curator generalizes across different executor backbones and domains.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Demystifying Agent Skills: Why They Work-Until They Don't
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Artifacts as Memory Beyond the Agent Boundary
- Towards a Science of Scaling Agent Systems
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation