Should an AI test only check the final answer, or also how it got there, and can it do both?

This explores the tension between benchmarks you can check cleanly, such as a final answer matched against ground truth, and benchmarks whose results tell you more about what happened: how the agent got there, and whether the score means what it claims. The corpus points to a layered answer. Keep the headline score strictly checkable, then add detail only in forms that are themselves checkable.

The case for putting verifiability first is that richer-looking evaluation can flatter the model. LR²Bench scores only final answers against deterministic ground truth, and it reveals a roughly 20% ceiling that trace-based grading would have inflated by rewarding reasoning that merely looks like reasoning (Should reasoning benchmarks score final answers or reasoning traces?). Verifiable answers are also what make the recent gains possible. A 3B model matching much larger systems does so on tasks with checkable ground truth, where RL gets a clean reward signal (Can small models match frontier reasoning without massive scale?). The catch is that this only works where answers can be checked at all.

A perfectly verifiable score can still tell you very little. Qwen2.5-Math-7B can reconstruct 54.6% of MATH-500 from partial prompts yet scores 0.0% on a post-release benchmark, so the score was measuring memorization (Does RLVR success on math benchmarks reflect genuine reasoning improvement?). Genuine reasoning behavior can appear in the same model at the same time, so benchmark gains and real capability are separable (Can genuine reasoning activation coexist with contaminated benchmarks?). Other things a single number hides include whether a model believes what it says, since truthfulness and honesty are distinct (Can a model be truthful without actually being honest?). Another is whether one output is representative, since a zero-temperature answer is still one draw from a distribution (Does setting temperature to zero actually make LLM outputs reliable?).

The most promising fix adds resolution by recording evidence rather than by judging more subjectively. BenchShield lets operators claim that a run followed the intended evaluation path, backed by recorded infrastructure evidence instead of a terminal score alone (Can infrastructure evidence replace terminal scores in benchmark validation?). AgentCompass separates the benchmark, harness and environment so that trajectories can be inspected, which surfaces reward hacking that a scalar score conceals (How can we make reward-hacking visible in agent evaluation?). Asynchronous verifiers extract checkable state from a reasoning trace as it runs and step in only on violations, with near-zero latency cost on correct runs (Can verifiers monitor reasoning without slowing generation down?). The same finer-grained logic shows up in trace selection, where step-level confidence catches breakdowns that a global average masks (Does step-level confidence outperform global averaging for trace filtering?).

More checking has limits. Validator consensus guarantees agreement from the protocol alone, but semantic validity only holds statistically, so adding validators does not certify meaning (Can validator consensus guarantee both agreement and semantic correctness?). How exposed a benchmark is to reward hacking also depends on where the evaluator's errors sit and how well the system searches, so no universal ranking of vulnerable setups exists (Can distance alone rank which substrates resist reward hacking?). In practice: score the final answer for the headline, attach evidence about the path that someone else can verify, and report whatever you cannot verify as a statistical claim.