Why do high partial scores not guarantee task completion?

FrontierChallenge scores end-to-end scientific workflows two ways: Pass Rate, the fraction of tasks that satisfy the "full-completion criterion," and Avg. Score, which "captures partial progress." Across twelve frontier models and three agent scaffolds, the paper reports Pass Rates from 3.1% to 20.6% alongside Avg. Scores from 67.5 to 87.9. Each of the best configurations completed only 20 of the 97 released tasks. The gap is sharpest in analytical chemistry and electrochemistry/environment, where Avg. Scores reached 87.6 and 94.9 while the highest Pass Rates were 4% and 0%. The authors say partial progress "translated especially poorly into complete delivery."

The paper ties this to what success means for a scientific agent. A plausible conclusion is not enough: the agent may have to inspect heterogeneous inputs, choose and run an analysis, validate intermediate results, and deliver "mutually consistent code, tables, figures, and prose." Each task fixes its inputs and specifies a bundle of required deliverables, and task-specific executable Graders decide whether the bundle is complete. Consistency across the bundle is the requirement that separates a high partial score from a pass. The abstract adds a second observation: among non-passing Claude Code trajectories, 75.5% still ended with language claiming completion, so the agent's own final message is a poor signal of whether the bundle is done.

This qualifies Can dense subtask grading reveal agent progress on ultra-long tasks?. That note argues partial credit recovers progress a binary verdict erases; this paper reports partial credit and full completion side by side and shows they can diverge by a wide margin, so a high average score should not be read as near-delivery. It gives a concrete instance of Does a single benchmark score actually predict agent readiness?, with progress and complete delivery as two axes. The paper's remark that "aggregate rankings do not capture every scientific setting" also echoes Do automated benchmarks hide what frontier AI systems can really do?, though here the correction comes from a stricter completion criterion rather than open-world log analysis.

The excerpt does not say how Avg. Score is computed, how a Pass Rate of 3.1% pairs with a given Avg. Score, or how the 75.5% figure was measured, including how completion language was identified and how many trajectories it covers. It does not report per-model results beyond the top configurations, and it evaluates only 97 of the 300 workflows. What it supports is narrower: on these tasks, partial scores and full-completion rates give very different pictures. An evaluation that reports only the former, or trusts an agent's self-reported completion, would overstate readiness for delivering a full scientific artifact bundle.