If an AI only gets graded on whether it fully breaks in, do we miss how close it actually got?

This explores whether grading an AI agent only on its final outcome (here, achieving arbitrary code execution in an exploit) throws away useful signal from the smaller stepping-stones along the way. In the exploit case the corpus says yes. ExploitGym's success criterion is clear and severe, but it ignores meaningful intermediate results like gaining arbitrary read/write access to memory or escaping a sandbox. An agent that got most of the way there is scored the same as one that failed on step one Does arbitrary code execution alone capture exploit progress?. Those 'primitives' are how real exploits get built, so a pass/fail score can't separate a near-miss from a non-starter. That is the distinction you'd want in order to see how close models are getting.

The same pattern shows up in reasoning research, where the final answer hides where things went wrong. Adding intermediate verification raised task success from 32% to 87%, because most failures were process violations rather than wrong answers Where do reasoning agents actually fail during long traces?. Across 10 reasoning models, the fraction of steps spent in abandoned branches predicted correctness better than trace length did. Failed branches stay in context and bias what comes next Does failed-step fraction predict reasoning quality better?. Confidence checked step by step catches breakdowns that a global average masks, and it lets you stop a bad trace early Does step-level confidence outperform global averaging for trace filtering?. In each case the endpoint is a summary that averages away the useful detail.

The corpus also warns against scoring every intermediate step. LR²Bench scores only final answers on purpose. Grading reasoning traces credited stylistic mimicry as real reasoning and inflated a result that is really capped near 20% Should reasoning benchmarks score final answers or reasoning traces?. The two views fit together once you ask what is being graded. Exploit primitives are concrete facts you can check, such as whether the agent can read arbitrary memory or has left the sandbox. A reasoning step's polish is not a checkable fact. So the lesson is to score intermediate milestones that can be verified, not to score more steps.

Intermediate events also matter for a second reason, which is whether the score can be trusted. BenchShield models a benchmark run as a finite sequence of typed events and checks it against what the task intended, so exploit detection works from deviations in the path rather than pattern-matching Can a finite lifecycle model detect reward hacking across benchmarks?. That lets operators claim more than 'the agent scored X'. They can claim it followed the intended evaluation path Can infrastructure evidence replace terminal scores in benchmark validation?. Checks that look at one action at a time can't state constraints that depend on history Can stateless checks ever catch sequence-level constraint violations?. Anything about a path needs the path recorded. The same holds in training. Rewarding only correct outcomes lumps clean and messy successes together, and GRPO-RoC filters positive trajectories for quality while keeping diverse failures as negative signal Why do correct code trajectories teach models to tolerate errors?.