Why does medium difficulty outperform both easy and hard RLVR training samples?
This explores the 'inverted-U' pattern in RLVR (reinforcement learning from verifiable rewards) — why problems of moderate difficulty teach reasoning better than problems that are too easy or too hard.
This explores why medium-difficulty problems hit a sweet spot in RLVR training, and the corpus is unusually consistent on it: learning value follows an inverted-U curve. The core mechanism is the *advantage signal* — RLVR learns from the contrast between a model's successes and failures on the same problem. Easy samples produce almost all successes, so there's no variance and nothing to learn from. Hard samples produce almost all failures. Medium-difficulty problems are the only band where the model succeeds often enough and fails often enough that the gradient carries information Why do medium-difficulty problems teach reasoning better than hard ones?.
The surprising part is that hard samples aren't just neutral — they're actively harmful. When a model almost never solves a problem, the rare accidental success gets treated by group-relative normalization as a hugely high-advantage trajectory, so the model over-learns whatever produced that lucky win: answer repetition, skipping computation, degenerate shortcuts. Worse, those shortcuts then bleed into capabilities the model already had Do overly hard RLVR samples actually harm model capabilities?. So the 'too hard' side of the curve isn't a missed opportunity — it's a contamination risk.
What makes this more than a tuning tip is that the productive band *moves*. A sample's difficulty isn't a fixed property of the problem — it's the interaction between the problem and the model's current ability. As the model improves, problems that were medium become easy, and the informative band drifts within a few training steps, which makes any static difficulty filter obsolete almost immediately How does model ability change what samples teach?. This is the same lesson that shows up far outside math: training empathetic agents, researchers found that moderately demanding, well-aligned environments beat maximally challenging ones, because maximum difficulty pushes the model outside the space it can actually explore Do harder training environments always produce better empathetic AI agents?.
There's a deeper through-line here about exploration. RLVR tends to exploit what already works rather than explore, which can quietly narrow a model's problem-solving range — a failure mode called capability boundary collapse Why does RLVR training narrow a model's problem solving ability?. Hard samples accelerate that collapse by rewarding shortcuts; medium samples keep the model in the region where genuine exploration still pays off. One fix the corpus points to is sequencing: run imitation-style supervised RL first to build reasonable rollouts, *then* apply RLVR — because the imitation phase manufactures the partial successes that make outcome rewards informative in the first place Does sequencing imitation then exploration training improve reasoning?.
If you want to follow the thread further, the corpus also complicates what 'works' even means here: some RLVR gains turn out to be memorization on contaminated benchmarks rather than real reasoning Does RLVR success on math benchmarks reflect genuine reasoning improvement?, and reasoning *activation* can be separated from benchmark improvement entirely Can genuine reasoning activation coexist with contaminated benchmarks?. Most strikingly, even random or spurious rewards can boost some models by surfacing latent behavior from pretraining — which suggests the difficulty curve isn't only about the data, but about what optimization pressure a given model is primed to respond to Why do random rewards improve reasoning for some models but not others?.
Sources 9 notes
RLVR learning follows an inverted-U curve across difficulty: medium problems yield strongest gains because they balance success frequency with informative failures, while easy samples lack variance and hard samples amplify shortcuts.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
A sample's learning value depends on the interaction between its difficulty and the model's current ability, not difficulty alone. The productive band of medium-difficulty problems drifts during training, making static difficulty estimates obsolete within steps.
RLVER research shows moderately demanding, well-aligned training environments produce better empathetic agents than maximally challenging configurations. Overly difficult setups push models outside their explorable space, causing instability rather than growth.
RLVR narrows models' problem-solving scope by prioritizing exploitation over exploration, a phenomenon called capability boundary collapse. Multiple importance sampling with exploration-based advantage functions can counteract this by integrating external data and explicitly rewarding discovery of underexplored but valuable reasoning paths.
Running Supervised RL first to establish reasoning foundations, then RLVR to refine against verifiable rewards, substantially outperforms both methods in isolation. The imitation phase makes outcome rewards informative by creating reasonable rollouts the RL phase can then sharpen.
Qwen2.5-Math-7B reconstructs 54.6% of MATH-500 from partial prompts but scores 0.0% on post-release LiveMathBench, revealing dataset contamination. On clean benchmarks, only correct rewards improve performance; random and inverse rewards fail or degrade reasoning ability.
RLVR activates genuine reasoning patterns through RL training while benchmark improvements may reflect data memorization on contaminated datasets. These operate at different measurement levels and can coexist without contradiction.
Qwen2.5-Math gains 16-25% MATH-500 improvement from random or incorrect rewards by activating latent code-reasoning behavior from pretraining, while Llama and OLMo show no gains. Pretraining format determines what optimization pressure can surface.