Can an AI ace a test simply because it already saw the answers — and flop on fresh problems?
Can memorization inflate apparent capability on benchmarks with available solutions?
This explores whether a model can look more capable on a test because it has effectively already seen the answers, so the score measures recall rather than reasoning.
This explores whether a model can look more capable on a test because it has effectively already seen the answers, and the corpus says yes, with one sharp example. Qwen2.5-Math-7B can reconstruct 54.6% of MATH-500 from only a partial prompt, yet it scores 0.0% on LiveMathBench, a benchmark released after the model was built Does RLVR success on math benchmarks reflect genuine reasoning improvement?. A model that can finish a problem it was only half shown has seen that problem before. The gap between solving the famous test and solving fresh problems of the same kind is memorization made visible.
The same paper shows that this inflation can pass for training progress. On the contaminated benchmark, reinforcement-learning gains were mostly memorization. On clean benchmarks, only correct rewards improved performance, and random or inverted rewards failed or made reasoning worse. So a benchmark with available solutions can make a training method look like it teaches reasoning when it mostly surfaces answers already in the weights. The clean benchmark is the control that exposes this.
Memorization is also not an on/off switch. A shift-cipher study split chain-of-thought performance into three separate factors: how probable the output is (which alone swung accuracy from 26% to 70%), memorization that follows pre-training frequency patterns, and real but error-prone reasoning What three separate factors drive chain-of-thought performance?. Models do both at once. Another study found that memorization based on the immediately preceding tokens accounts for up to 67% of reasoning errors, and grows as problems get harder or drift from the training data Where do memorization errors arise in chain-of-thought reasoning?. A public benchmark with published solutions tilts that blend toward recall. Even the visible effort can mislead. Reasoning-trace length tracks problem difficulty only on familiar problems and decouples from it on unfamiliar ones, because it mostly reflects recall of training schemas Does longer reasoning actually mean harder problems?.
Memorization is one of several ways a score can flatter a model. Chain-of-thought prompts with logically invalid steps perform nearly as well as valid ones, so the gain comes from the form of reasoning rather than its validity Does logical validity actually drive chain-of-thought gains?. Models trained to imitate ChatGPT fool human evaluators with a confident style while closing no real capability gap Can imitating ChatGPT fool evaluators into thinking models improved?. Automated benchmarks can both overstate and understate what frontier systems can do, because they favor precisely specified, auto-gradable tasks. Open-world evaluations of messy, long-horizon work, checked by reading the logs, correct that Do automated benchmarks hide what frontier AI systems can really do?.
The defenses the corpus points to are tests the answers couldn't have leaked into: benchmarks released after the model, partial-prompt reconstruction checks, and held-out tasks from a different distribution. One example is AIDE2's gains holding on four unseen benchmarks, including physics-based weather forecasting outside its selection task Do AIDE2's improvements transfer to unseen tasks?. The hard evidence here is mostly from math, so how far the effect extends to other domains with public solutions, such as coding, is not something this material tests directly.
Sources 8 notes
Qwen2.5-Math-7B reconstructs 54.6% of MATH-500 from partial prompts but scores 0.0% on post-release LiveMathBench, revealing dataset contamination. On clean benchmarks, only correct rewards improve performance; random and inverse rewards fail or degrade reasoning ability.
A shift cipher study decomposed CoT into three independent factors: output probability alone swings accuracy from 26% to 70%, memorization matches pre-training frequency patterns, and genuine reasoning exists but accumulates error with each step. This resolves the reason-or-memorize debate by showing LLMs do both simultaneously.
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
Show all 8 sources
Imitation models fool human evaluators by mimicking ChatGPT's confident, fluent style while failing to improve factuality or generalization on novel tasks. The ceiling is set by base model capability, not fine-tuning method—better fundamentals, not shortcuts, drive real improvement.
Automated benchmarks both overstate and understate capability by privileging precisely-specified, auto-gradable tasks. Open-world evaluations of long-horizon messy tasks through qualitative log analysis—with cost explicitly reported—correct these distortions and catch emerging capabilities earlier.
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
- Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexity
- Agents' Last Exam
- Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model Prompting