SYNTHESIS NOTE
Topics›Flaws›this note

Can genuine reasoning activation coexist with contaminated benchmarks?

RLVR shows both real behavioral changes and inflated metrics. Can these contradictory findings actually describe the same phenomenon from different angles, and what does that mean for evaluating reasoning improvements?

Synthesis note · 2026-02-23 · sourced from Flaws

Two RLVR findings appeared contradictory:

Spurious rewards work: Why do random rewards improve reasoning for some models but not others?, suggesting the reward signal itself matters less than the RL training process, which activates latent pretraining capabilities. This was treated as evidence that RLVR functions as a pretraining catalyst rather than a reasoning teacher.

Benchmark contamination: Since Does RLVR success on math benchmarks reflect genuine reasoning improvement?, the metric improvement may be data memorization rather than genuine reasoning activation.

The resolution: These findings operate at different measurement levels and can coexist:

  1. Behavioral activation (genuine): RL training with any reward signal activates code reasoning formats and structured thinking patterns that exist in pretraining data but are dormant. This is visible in output format changes, thinking token usage, and exploration behavior changes — measurements not contaminated by benchmark overlap.

  2. Benchmark improvement (inflated): The metric improvement on contaminated benchmarks is partially or fully attributable to memorization. Clean benchmarks show reduced or eliminated gains for spurious rewards, while correct rewards still improve.

The practical implication: RLVR research must separate behavioral measurements (how the model's reasoning process changes) from performance measurements (how benchmark scores change). Both are informative; conflating them produces confusion about what RLVR actually does. The one-shot activation finding (single example triggers 36%→73.6% improvement) may itself need re-evaluation on clean benchmarks.

Inquiring lines that read this note 92

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What distinguishes genuine understanding from correct linguistic performance? How do benchmark scores relate to real-world task completion? How does model awareness of deployment context affect strategic behavior? What causes hallucination and how can it be addressed? Can synthetic data preserve necessary diversity to prevent model collapse? Does reinforcement learning create new reasoning capabilities or refine existing ones? How do standardized forms and measurement artifacts shape observable evidence? What determines whether visible reasoning traces improve task performance? Why do language models systematically fail at causal reasoning? Do standard benchmarks reveal AI capability in real-world contexts? Do benchmark score improvements actually reflect genuine AI capability gains? Do base models contain latent reasoning capabilities that training can activate? Why do human-AI teams underperform compared to their individual components? What actually drives chain-of-thought performance gains in reasoning tasks? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? Should reward models emphasize process understanding or only final outcomes? How does policy entropy collapse limit reasoning RL scaling? What mechanisms cause models to behave deceptively despite alignment training? Does polished AI output mislead users about actual accuracy and expertise? Can models strategically underperform to evade capability evaluation? Can verification capability keep pace with AI generation speed? Should agent evaluation measure execution quality beyond task success? Do reasoning traces accurately represent what models actually compute? How do pretraining data composition and task diversity shape reasoning? How do different post-training structures affect learning outcomes? Does training data format shape reasoning more than domain content? Does chain-of-thought reasoning faithfully reflect actual model reasoning processes? What training data structures and signals produce robust learning across distributions? Which agent-created code should persist in infrastructure versus remain task-specific? Do single-axis benchmarks adequately measure deployment-ready agent capability? Can prompts override training-embedded biases in language models? How do mechanistic features and representations drive model behavior? How can parameter sparsity improve interpretability while maintaining model capability? What is the actual bottleneck preventing effective AI research automation? How do credentials maintain predictive value when their context changes? Can oversight systems work against strategically aware agents? What fundamentally drives AI inference pricing and cost decline patterns? Can AI agents learn to improve performance from experience?

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 5

Suggested questions this note speaks to — click to search the collection, or type your own.

Do spurious rewards activate reasoning without teaching new skills?How much RLVR improvement comes from benchmark data memorization?What behavioral changes occur during reward learning training?Can clean benchmarks reveal true RLVR reasoning gains?Why do single examples trigger large reasoning improvements in models?
Original note title

RLVR behavioral activation and benchmark improvement are separable — genuine pretraining activation can coexist with contamination-inflated metrics