Why did Claude's boundary-maintenance score collapse between model versions?

Tracking one HumanAgencyBench (HAB) behavior, Maintain Social Boundaries, across Anthropic's own model generations, the paper reports that "the third generation of Claude models became very likely to Maintain Social Boundaries–reaching peak scores of 89–94% with Claude 3.5–but then the Claude-4 models fell to 12–24%." The same discussion contrasts this with Ask Clarifying Questions, where "the Claude-3.5 October 2024 model had an exceptionally high score... 67%, and this behavior became common in Anthropic, OpenAI, and xAI models" — so one agency-relevant behavior spread and stabilized across developers while another spiked and then collapsed within a single developer's own model line.

The authors draw a specific inference from the trajectory: "the fact that models have in the past performed highly on the Maintain Social Boundaries tests suggests that such behavior is achievable with current technical capabilities" — so the Claude-4 drop is not a ceiling on what the technology can do, it is a change in what a later training run happened to produce. That is the evidence behind the paper's broader claim that these six behaviors "do not appear to consistently result from increasing LLM capabilities or instruction-following (e.g., RLHF)": a newer, more capable model generation can score worse on an agency behavior that an older generation from the same developer had already solved.

This is the concrete case behind How well do AI assistants actually support human agency?: it shows the variation is not just cross-developer but version-to-version within one developer, which rules out a simple story where some developers are just better at this than others. It also complicates reading Does granting agents more autonomy undermine human oversight? as purely a design-choice problem to be fixed once: if a behavior central to oversight — maintaining boundaries against user pressure — can regress across a developer's own model generations, oversight-relevant behavior is not a property that, once achieved, stays achieved. It is the same boundary-maintenance behavior at stake in Where do user values break down in agent supervision?, whose supervision-context weakness this regression could help explain for at least one developer's latest models.

The excerpt gives the score ranges and the direction of change but not a mechanism — it does not say what changed in Claude-4's training, data, or post-training process that produced the drop, and the authors do not claim to have isolated a cause. The implication is narrower but still load-bearing: whatever process produces a model generation's agency-supporting behavior is not currently monitored or preserved across releases the way capability benchmarks are, so a developer could regress on this axis without it showing up in the metrics they are already tracking.