SYNTHESIS NOTE
Topics›Flaws›this note

Does RLVR success on math benchmarks reflect genuine reasoning improvement?

Explores whether RLVR's apparent effectiveness with spurious rewards on contaminated benchmarks like MATH-500 represents actual reasoning gains or merely data memorization retrieval.

Synthesis note · 2026-02-23 · sourced from Flaws

The apparent success of RLVR with random, incorrect, or spurious reward signals on Qwen models may be an artifact of data contamination rather than evidence of genuine reasoning improvement.

The contamination evidence: prompting Qwen2.5-Math-7B with the first 60% of each MATH-500 question yields 54.6% exact-match reconstruction of the remaining 40% and 53.6% correct answers to these incomplete problems. On LiveMathBench — a benchmark released after Qwen2.5 — completion rate drops to 0.0%, consistent with Llama3.1-8B (3.8%/0.0% respectively). The model has memorized MATH-500.

On a fully clean benchmark (RandomCalculation — synthetic arithmetic expressions generated after Qwen's release): correct rewards deliver consistent gains surpassing the model's performance ceiling; random rewards make training highly unstable with no reliable improvement; inverse rewards rapidly erode mathematical reasoning ability.

This directly challenges Why do random rewards improve reasoning for some models but not others?. The prior interpretation — that any optimization pressure activates pretraining strategies — may confound two effects: genuine strategy activation (possible) and recall of memorized answers triggered by format-similar optimization (likely for contaminated benchmarks). On clean data, the "any reward works" finding evaporates for random and inverse signals.

The practical implication: RLVR research conclusions drawn from MATH-500 and similar benchmarks for Qwen models should be interpreted with caution. Reward engineering may matter more than the spurious-reward literature suggests — we were measuring memorization recovery, not reasoning improvement.

Inquiring lines that read this note 94

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How do benchmark scores relate to real-world task completion? What causes hallucination and how can it be addressed? What systematic failures occur in human preference-based model evaluation? How do standardized forms and measurement artifacts shape observable evidence? Do standard benchmarks reveal AI capability in real-world contexts? Should reward models emphasize process understanding or only final outcomes? How does model scale affect fine-tuning and downstream task performance? Does reinforcement learning create new reasoning capabilities or refine existing ones? How does policy entropy collapse limit reasoning RL scaling? How do pretraining data composition and task diversity shape reasoning? Does polished AI output mislead users about actual accuracy and expertise? Can formal verification substitute for human understanding of AI proofs? What are the trade-offs between memory capacity and continuous task learning? How do different reward signals affect model reasoning and calibration differently? Do benchmark score improvements actually reflect genuine AI capability gains? How does scaling test-time compute compare to increasing model size? Should agent evaluation measure execution quality beyond task success? How do different post-training structures affect learning outcomes? Does training data format shape reasoning more than domain content? What external mechanisms enable reliable self-improvement in language models? Do single-axis benchmarks adequately measure deployment-ready agent capability? How do frontier models develop behaviors that escape detection? How can parameter sparsity improve interpretability while maintaining model capability? What prevents or enables error accumulation in self-training systems? How do credentials maintain predictive value when their context changes? How do clinicians evaluate the trustworthiness of AI medical recommendations? What systematic biases do LLM judges exhibit in evaluation? What training data structures and signals produce robust learning across distributions? What is the actual bottleneck preventing effective AI research automation? How can we better validate AI research and prevent misconceptions? Can oversight systems work against strategically aware agents? Does preference-optimized training sacrifice accuracy and calibration for perceived quality? How can model evaluations remain robust against gaming and evaluator drift? Does AI assistance enhance or undermine professional skill development?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 127 in 2-hop network ·dense cluster Open in graph ↗
Does RLVR success on math benchmarks reflect gen… Why do random rewards improve reasoning for some m… Why does RLVR work with completely random rewards? Does RLVR actually expand what models can reason a… How much of LLM few-shot ability comes from traini…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do spurious reward signals improve reasoning for some pretrained models?Can RLVR expand a model's reasoning capabilities beyond its training ceiling?How widespread is task contamination in LLM evaluation benchmarks today?What distinguishes genuine reasoning activation from memorization-assisted answer recall?
Original note title

RLVR effectiveness on contaminated benchmarks is primarily data memorization — clean benchmarks eliminate spurious reward gains