How much of AI progress comes from better tools versus better weights?

Nathan Benaich's State of AI Report 2026 argues that frontier progress is not a steady climb: "Frontier capabilities remain uneven or 'spiky,' and can swing between generations." His own experience of Anthropic's models illustrates it directly — "Opus 4.6 was a good model, Opus 5 was a step back, and Opus 5.5 was good again." He pairs that with a controlled study showing how much of a model's measured capability sits outside its weights: "changing the tools, context, and feedback available to GLM-5.1 lifted success from 52.5% to 65.5% on 100 SWE-bench Verified tasks, without changing its weights." On the self-improvement question specifically, he writes that "AI is already helping build better AI," but stops short of calling it closed-loop: Karpathy's "autoresearch" has an agent "edit training code, run five-minute experiments, and keep improvements overnight on one GPU," which Benaich says "automates a useful part of research," while adding that "sustained, fully autonomous recursive improvement remains to be demonstrated."

His reasoning treats generation, scaffolding, and workflow automation as three separate axes rather than one capability curve. A model's raw weights can regress between releases even as the field's effective output keeps rising, because tools, context, and feedback loops are doing part of the work the weights get credited for. He reads Anthropic's internal figure — Claude leading "26% of measured model R&D work in August, up from under 1% in February" — the same way: as evidence that AI assists research tasks today, not as evidence of an autonomous loop that no longer needs a researcher setting tasks and supervising execution.

This matches, from the outside, what Does Claude Opus 5.5 fully automate AI research tasks? finds from a predeployment evaluation: Opus 5.5 beats Fable 5.1 on verifiable and harder-to-verify tasks alike, yet METR calls the gain "incremental ... rather than a discontinuous jump" because it is "judgement" and "taste," not benchmark score, that is missing for full automation — the same uneven, non-monotonic picture Benaich describes from the user's seat. Benaich's use of the 26% figure also sits alongside How fast is AI accelerating its own development inside labs?, which proposes that figure as part of a transparency metric Anthropic itself wants independently verified; Benaich cites the same number as a plain data point in his own narrative, without that verification caveat attached. The report's cyber-incident material, covered separately in Do safety guardrails block defenders from investigating attacks?, is the offensive-capability corollary to this same uneven-progress picture.

The excerpt does not quantify how much of overall frontier progress is attributable to scaffolding versus weights beyond the single GLM-5.1 SWE-bench case, and it does not define what evidence would count as "sustained, fully autonomous recursive improvement" — Benaich names the absence without specifying the test. His report is also a venture investor's annual synthesis of others' benchmarks and disclosures, not an independent evaluation of any model. The implication the evidence supports is narrow: more than a year into reasoning models compounding gains, the field still lacks either a demonstrated full-autonomy threshold for AI R&D or an agreed metric for tracking distance to it.