SYNTHESIS NOTE
Topics›RLVR›this note

Do overly hard RLVR samples actually harm model capabilities?

Explores whether training on problems beyond a model's competence band causes active regression rather than mere learning failures. Investigates whether group-relative normalization amplifies accidental successes into harmful shortcuts.

Synthesis note · 2026-05-28 · sourced from RLVR

The damage from over-hard RLVR samples is not merely "the model fails to improve." It is active regression. When almost every rollout on a problem fails, the rare success is unlikely to be a genuinely good solution — it is more often a shortcut, an answer reached by skipping necessary computation, or a lucky guess. Group-relative normalization then treats that one trajectory as the high-advantage exemplar of the group and reinforces it. The model learns the shortcut, not the reasoning.

The behavioral signature is concrete: answer repetition, skipping computation that the problem requires, and other degenerate patterns that look like reasoning collapse. More troubling, these effects do not stay local to the hard problems — they degrade the model's pre-existing capabilities, the things it could already do before training pushed it past its competence band. The internal-feature analysis corroborates this: hard problems activate reasoning-related features but those features become useful only on the rare successful trajectory, so most of the gradient on hard samples is reinforcing the wrong activations.

Why it matters: it identifies a specific corruption channel rather than a generic "training instability." The villain is the interaction between a sparse-success reward landscape and group-relative normalization, which together turn statistical noise (an accidental success) into a learning target. This sharpens the case against naively harvesting hard examples and connects RLVR difficulty to the broader pattern where verifiable-reward training rewards trajectories that pass the check without doing the work. The counterpoint a defender might raise — that some hard problems are exactly where capability frontiers expand — only holds when successful trajectories are sampled densely enough to outvote the shortcuts, which over-hard samples by definition fail to provide.

Inquiring lines that read this note 252

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How does scaling test-time compute compare to increasing model size? What causes foundation models to develop task heuristics instead of understanding? How do different post-training structures affect learning outcomes? What training data structures and signals produce robust learning across distributions? Does reinforcement learning create new reasoning capabilities or refine existing ones? What constraints limit conversational AI's effectiveness in therapeutic contexts? How do benchmark scores relate to real-world task completion? What prevents or enables error accumulation in self-training systems? Does reward hacking in training cause emergent misalignment? How can model evaluations remain robust against gaming and evaluator drift? How does model scale affect fine-tuning and downstream task performance? Does training data format shape reasoning more than domain content? Do AI safety evaluations reliably detect unsafe model behavior? How do pretraining data composition and task diversity shape reasoning? How should agents balance abstract planning with concrete environmental grounding? How faithfully does iterative DPO approximate online RL? When and why does aggregating multiple reasoning samples improve performance? How does policy entropy collapse limit reasoning RL scaling? Is model misalignment inherent to base models or induced by alignment procedures? How can knowledge graph structure improve reasoning and retrieval? How does explicit structure improve LLM reasoning on complex tasks? Can synthetic data preserve necessary diversity to prevent model collapse? Does reasoning fine-tuning trade off other critical model abilities? Does semantic diversity improve reasoning quality in reinforcement learning? Should reward models emphasize process understanding or only final outcomes? How do frontier models develop behaviors that escape detection? Can prompts override training-embedded biases in language models? How does model awareness of deployment context affect strategic behavior? When should retrieval be triggered to augment language model reasoning? What external mechanisms enable reliable self-improvement in language models? When does routing outperform scaling as a performance solution? How can parameter sparsity improve interpretability while maintaining model capability? Does learning articulatory physics explain speech communication superiority? Can AI systems autonomously pursue unstated objectives or circumvent design constraints? When do additional thinking tokens stop improving reasoning performance? Does preference-optimized training sacrifice accuracy and calibration for perceived quality? What is the actual bottleneck preventing effective AI research automation? Can harness optimization rival model scaling in effectiveness? Why does self-revision increase model confidence despite degraded accuracy? Do reasoning traces accurately represent what models actually compute? Why does separating planning from execution enhance reasoning performance? Do language models learn underlying linguistic structure or merely surface patterns? Do base models contain latent reasoning capabilities that training can activate? How can agents learn and retain useful knowledge across diverse environments? What are the trade-offs between memory capacity and continuous task learning? Why does consolidated memory sometimes hurt continual learning more than no memory? Can oversight systems work against strategically aware agents? Can scaling alone produce compositional generalization in neural networks? How do standardized forms and measurement artifacts shape observable evidence? What causes hallucination and how can it be addressed? Can we reliably detect when models engage in reward hacking? How can we better validate AI research and prevent misconceptions? Can safety evaluations accurately measure true capabilities when models obscure them? Can AI agents learn to improve performance from experience? Do benchmark score improvements actually reflect genuine AI capability gains? Do planted shortcuts in benchmarks measure real reward hacking? How do different reward signals affect model reasoning and calibration differently? How do mechanistic features and representations drive model behavior? What makes reasoning models struggle with ill-posed and unfamiliar problems? Why do human-AI teams underperform compared to their individual components? Can models strategically underperform to evade capability evaluation?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 129 in 2-hop network ·dense cluster Open in graph ↗
Do overly hard RLVR samples actually harm model … Why do medium-difficulty problems teach reasoning … Does RLVR actually improve mathematical reasoning … Why does RLVR training narrow a model's problem so… Do conversational recommender benchmarks actually … Why does RLVR work with completely random rewards? What reasoning features does each difficulty level…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 5

Suggested questions this note speaks to — click to search the collection, or type your own.

Why does medium difficulty outperform both easy and hard RLVR training samples?Can group-relative normalization be modified to resist shortcut trajectories?How do reasoning-related features behave when trained on near-impossible problems?What distinguishes genuine capability gains from coherent but invalid reasoning traces?Does importance sampling actually recover capabilities lost to hard sample training?
Original note title

overly hard rlvr samples induce degenerate behaviors and amplify shortcut trajectories degrading prior capability