Does Anthropic's survey adequately support its automated R&D risk conclusion?

METR's review of the "Risks from automated R&D" section in Anthropic's February 2026 Risk Report finds the report's case too weak to carry its own bottom line. The report "makes the argument that catastrophic risk from Claude Opus 4.6 or a less capable Anthropic model automating R&D in any domain is very low." METR's verdict on the argument itself: "We do not think the report adequately supports its conclusion." It flags "analytical rigor" problems in the report's model-use survey — "issues including sample size, question granularity, survey framing" — and says the overall argument "misses the possibility of substantial AI R&D acceleration before its full automation." On "adequacy of information," METR finds Anthropic "summarizes the survey results in a way that miscounts one missing response to a question as a negative response."

The mechanism of the review is a split between evidence and conclusion. "If we had to solely rely on the evidence presented by Anthropic in the original Risk Report, we would likely disagree with the report's conclusion." What changes METR's mind is evidence from outside that survey: "additional evidence indicating that the model is incapable of R&D in key domains, including the results of METR evaluations and the lack of public reports of the model automating any key domain." On that separate basis, "we agree with the bottom-line conclusion of the report... but we think the evidence presented in the report is inadequate to establish this." METR also recommends Anthropic widen the internal survey — bigger sample, more granular response options, different framing — and report other, more leading indicators of AI progress.

The report METR is reviewing sits next to two self-reported Anthropic measures of the same automation trend. Is AI development already being handed to AI systems? cites task-length growth, code-authorship share and a speedup test as evidence of delegation to AI. How fast is AI accelerating its own development inside labs? goes further and names the fix directly: that snapshot, "by the source's own account, would need third-party verification... before it could show how the pace is changing." METR's review is that outside check, applied to the companion risk-assessment survey, and it reaches a sharper verdict than either note anticipates: not that the self-reported numbers are wrong, but that a survey built on them cannot establish a "very low risk" conclusion, regardless of whether that conclusion happens to be correct.

The excerpt is an executive summary: it names the problem categories — sample size, question granularity, survey framing, prior METR research on calibration difficulty, the miscounted missing response — without giving the underlying numbers, question text, or response counts, which sit in the full PDF reviews it points to but doesn't include. It also doesn't say how the miscounted response changed the report's published figures, or what a corrected survey would show. The narrower result that survives is this: a lab's self-administered survey of its own models, even when it points toward a conclusion that later holds up on independent evaluation, is not itself sufficient evidence for a catastrophic-risk judgment.