Testing a model in isolation and testing the product built on it are different jobs — so why do we grade them with the same benchmarks?
Why do model evaluation and downstream evaluation require different benchmark designs?
This explores why testing a model on its own (what can the weights do?) and testing it in a real task setting (does the deployed system get the job done?) need different benchmark designs, and what goes wrong when one is mistaken for the other.
This explores why measuring a model by itself and measuring how it performs on a real downstream task call for different benchmarks. The corpus has no single note that sets out this distinction directly. Read together, though, several notes show the same thing: a score belongs to the whole setup that produced it, and the model is only one part of that setup. Most of the disagreement between the two kinds of evaluation comes from that.
The clearest evidence comes from work on harnesses. A harness is the code, routing and execution scaffolding wrapped around a model. On Terminal-Bench, optimizing only the harness lifted several frozen models by meaningful margins, and the same runbook carried over to newer models unchanged Can execution harnesses lift model performance without retuning weights?. A stronger model nearly doubled a weaker model's Theory-of-Mind scores without retraining it, mostly by moving shaky reasoning into deterministic code Can a stronger model lift a weaker one at test time without retraining?. A small 9B model trained on whether its patches actually worked improved a frozen agent more than prompted frontier models did Does training editors on real outcomes beat prompting larger models?. So a downstream score partly measures the harness, while a model-level benchmark tries to hold the harness fixed or remove it. Each design answers a different question, and comparing their numbers directly can mislead you.
The gap also runs the other way. Capability shown on standard reasoning benchmarks doesn't reliably carry over to real tasks. Reasoning models with extended chain-of-thought showed no consistent advantage on constraint-heavy numerical problems like power-grid optimization. They produced more text but no more actual iterative computation Do reasoning models actually beat standard models on optimization?. Training effects can also reverse from one domain to another: preference tuning reduces diversity in code but increases it in creative writing Does preference tuning always reduce diversity the same way?. A benchmark built to measure a general model trait can't tell you which way that trait will move in your domain. Downstream evaluation has to be built around what that domain rewards.
You might expect interactive, trajectory-based evaluations to bridge the two. They don't. Comparability, reproducibility and turning evidence into a judgment are still problems, just spread across more dimensions Do interactive evaluations actually solve the benchmark comparison problem?. That is why provenance matters. A score with its source and setup stripped away can't support a fair comparison, because you can't tell whether two numbers measured the same thing Can benchmark scores be trusted without knowing their origin?. A related lesson from weak-model committees: performance on tasks with tests, proofs or type checks behaves differently from performance on tasks without them, because those checks are what turn a correct candidate into a correct selection When can weak models match strong model performance?. Whether a downstream task offers a check like that is a design variable that model-only benchmarks usually ignore.
The takeaway you might not have expected is that a leaderboard number is not really a property of the model. It describes a model, a harness, a domain and the available verification signals together. Model evaluation tries to hold everything except the weights constant. Downstream evaluation deliberately includes everything. Neither is wrong, but a benchmark that doesn't say which one it is, and doesn't keep the setup attached to the score, leaves you unable to judge what it measured.
Sources 8 notes
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
A 9B model trained with reinforcement learning on patch success raised a frozen agent's performance by 9.3 points across three tasks, while prompted frontier models produced unstable or lower gains. The difference stems from feedback: trained editors rerun patches to verify impact, while prompted models optimize only for plausibility.
Reasoning variants with extended CoT show no consistent advantage over standard models on constraint-bound numerical tasks like optimal power flow. Extended thinking produces more text, not more iterative computation, suggesting the bottleneck is numeric procedure rather than reasoning steps.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Show all 8 sources
Interactive evaluation relocates core problems—comparability, reproducibility, evidence-to-judgment mapping—into higher-dimensional space rather than solving them. The field needs design protocols and shared standards, not format adoption, to make trajectory scoring interpretable.
Benchmark Radar catalogs AI evaluations while keeping source identities and citations attached, enabling readers to trace scores back to their original settings. The system demonstrates that scores without provenance cannot reliably support model comparisons.
Sampling alone amplifies coverage but cannot select correct solutions. Reliable performance matching requires external soundness signals—tests, proofs, or type checks—that convert latent correct proposals into actual selections.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Sharpening Tax in Post-Training
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- Interactive Evaluation Requires a Design Science
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- PostTrainBench: Can LLM Agents Automate LLM Post-Training?
- Decomposing and Measuring Evaluation Awareness