SYNTHESIS NOTE
Topics›RLVR›this note

Do overly hard RLVR samples actually harm model capabilities?

Explores whether training on problems beyond a model's competence band causes active regression rather than mere learning failures. Investigates whether group-relative normalization amplifies accidental successes into harmful shortcuts.

Synthesis note · 2026-05-28 · sourced from RLVR
What does reward learning actually do to model reasoning?

The damage from over-hard RLVR samples is not merely "the model fails to improve." It is active regression. When almost every rollout on a problem fails, the rare success is unlikely to be a genuinely good solution — it is more often a shortcut, an answer reached by skipping necessary computation, or a lucky guess. Group-relative normalization then treats that one trajectory as the high-advantage exemplar of the group and reinforces it. The model learns the shortcut, not the reasoning.

The behavioral signature is concrete: answer repetition, skipping computation that the problem requires, and other degenerate patterns that look like reasoning collapse. More troubling, these effects do not stay local to the hard problems — they degrade the model's pre-existing capabilities, the things it could already do before training pushed it past its competence band. The internal-feature analysis corroborates this: hard problems activate reasoning-related features but those features become useful only on the rare successful trajectory, so most of the gradient on hard samples is reinforcing the wrong activations.

Why it matters: it identifies a specific corruption channel rather than a generic "training instability." The villain is the interaction between a sparse-success reward landscape and group-relative normalization, which together turn statistical noise (an accidental success) into a learning target. This sharpens the case against naively harvesting hard examples and connects RLVR difficulty to the broader pattern where verifiable-reward training rewards trajectories that pass the check without doing the work. The counterpoint a defender might raise — that some hard problems are exactly where capability frontiers expand — only holds when successful trajectories are sampled densely enough to outvote the shortcuts, which over-hard samples by definition fail to provide.

Inquiring lines that read this note 195

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

When does architectural design matter more than raw model capacity? How can identical external performance mask different internal representations? How can AI systems learn from failures without cascading errors? Does fine-tuning modify underlying model capabilities or only behavioral outputs? What pretraining choices and baseline capability constrain reinforcement learning gains? Why do LLM chatbots fail as independent therapeutic agents? Why do benchmark improvements fail to reflect actual reasoning quality? What constrains reinforcement learning's ability to expand model reasoning? Can alternative training methods improve on supervised fine-tuning for language models? Why does supervised fine-tuning improve accuracy while degrading reasoning quality? How does memorization interact with learning and generalization? What determines success in training models on multiple tasks? How do self-generated feedback mechanisms enable effective model learning? What are the consequences of models training on synthetic data? Can language model RL training avoid reward hacking and misalignment? Can ensemble evaluation methods reduce bias more than single judges? Does domain specialization cause models to lose capabilities elsewhere? How does example difficulty affect learning efficiency in language models? Why does training format shape reasoning strategy more than domain content? What makes weaker teacher models effective for stronger student training? How do policy learning algorithm choices affect multi-objective optimization stability? Why do self-improving systems struggle without clear external performance metrics? Why do continual learning scenarios trigger catastrophic forgetting and interference? How does test-time aggregation affect reasoning correctness and reliability? How do adversarial and manipulative prompts attack reasoning models? How does policy entropy collapse constrain reasoning-focused reinforcement learning? How do knowledge injection methods compare across cost and effectiveness? How do neural networks separate factual knowledge from reasoning abilities? Can model confidence signals reliably improve reasoning quality and calibration? Does externalizing cognitive work and state improve agent reliability? Can self-supervised signals enable process supervision without human annotation? Why does reinforcement learning suppress output diversity compared to supervised fine-tuning? How do evaluation mechanisms prevent error accumulation in autonomous research systems? How do training priors constrain what context information can override? When should retrieval-augmented systems decide to fetch new information? Does reinforcement learning teach reasoning or just when to reason? How should we design LLM systems to maintain alignment and control? When does optimizing for quality undermine the value of diversity? Can model routing outperform monolithic scaling as an efficiency strategy? Can next-token prediction alone produce genuine language understanding? When do additional thinking tokens stop improving reasoning performance? Does RLHF training sacrifice accuracy and grounding for user agreement? Do harness improvements transfer across model scales or memorize shortcuts? Why does self-revision increase model confidence while degrading accuracy? Why do correct reasoning traces tend to be shorter than incorrect ones? Why does finetuning cause catastrophic forgetting of model capabilities? Do base models contain latent reasoning that training can unlock? How can LLM user simulators model realistic goal-driven conversation? Can AI systems balance emotional competence with factual reliability? How can process reward models supervise complex reasoning traces? Why does consolidated memory sometimes degrade agent performance? How does sequence length affect sparsity tolerance in models? What actually drives chain-of-thought reasoning improvements in language models? Can language model hallucination be prevented or only managed?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 127 in 2-hop network ·dense cluster Open in graph ↗
Do overly hard RLVR samples actually harm model … Why do medium-difficulty problems teach reasoning … Does RLVR actually improve mathematical reasoning … Why does RLVR training narrow a model's problem so… Do conversational recommender benchmarks actually … Why does RLVR work with completely random rewards? What reasoning features does each difficulty level…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 5

Suggested questions this note speaks to — click to search the collection, or type your own.

Why does medium difficulty outperform both easy and hard RLVR training samples?Can group-relative normalization be modified to resist shortcut trajectories?How do reasoning-related features behave when trained on near-impossible problems?What distinguishes genuine capability gains from coherent but invalid reasoning traces?Does importance sampling actually recover capabilities lost to hard sample training?
Original note title

overly hard rlvr samples induce degenerate behaviors and amplify shortcut trajectories degrading prior capability