Why does group-relative normalization make uniform episode rewards work across rollouts?
This explores the GRPO trick where every step in an episode gets the same total reward, yet learning still works — because normalizing rewards against the rest of the group, not against an absolute scale, is what turns a flat signal into a learning signal.
This explores why you can stamp one uniform episode-level reward onto every step of a rollout and still get useful credit assignment — the answer is that the signal doesn't live in the reward's absolute value, it lives in how that value compares to the other rollouts you sampled. Group-relative normalization subtracts the group's mean and divides by its spread, so a trajectory that scored a 'good' outcome only counts as good *relative to its siblings*. A reward of 1.0 means nothing on its own; a reward of 1.0 when the rest of the batch averaged 0.3 means 'this sequence of actions did something right.' That's why uniform per-step rewards work: the comparison across rollouts, not the magnitude, is doing the teaching. Can full episode rewards per step enable better credit assignment? is the cleanest illustration — assigning the cumulative episode reward to every step looks like it should smear credit everywhere, but group-relative normalization across rollouts surfaces exactly which action sequences succeeded, and a 3B model trained this way beat 72B baselines.
The deeper move is that you're using the population of rollouts as a self-supervised baseline. There's no learned value function or separate critic deciding what 'average' looks like — the group *is* the baseline. This is why the quality of your sampled group matters so much. Can shared-prefix trees reduce redundancy in agent rollouts? shows that branching trajectories from shared prefixes produces more *distinct* rollouts per token budget, which directly sharpens the advantage estimates — more spread in the group means cleaner relative ranking. The flip side: if every rollout in a group scores identically, normalization produces zero signal, and that group teaches nothing.
That degenerate case is itself useful, and the corpus shows people weaponizing it. cross-rollout-variance-functions-simultaneously-as-reward-signal-and-query-filter reuses within-group variance as both the token-weighting signal and a filter to *throw out* queries where all rollouts look the same — a low-variance group is a query the model already solved or can't distinguish, so it's dead weight. How should multiple reward objectives be weighted during training? takes the same statistic and weights multiple objectives by their within-group variance, automatically amplifying whichever objective is currently producing separable signal and suppressing noise. The within-group spread keeps showing up because it's the same quantity that makes normalization work in the first place.
The most interesting twist is that you can manufacture the relative structure rather than just sampling for it. Can tree structure alone convert outcome rewards into process supervision? uses branching: when two sibling subtrees diverge at a decision point and one ends up winning, comparing them converts a single outcome reward into a step-level preference — process supervision with no process annotation, just because the tree gives you matched comparisons. And Can rubrics and dense rewards work together without hacking? shows the group can be gated *before* normalization: use rubrics to accept or reject whole rollout groups rather than folding rubric scores into the reward, which preserves the categorical signal and prevents the reward hacking you'd get from over-shaping. Across all of these, the lesson is the same — in GRPO the reward is just bookkeeping, and the real machinery is the comparison structure you build across the group.
Sources 6 notes
MS-GRPO assigns cumulative episode reward to each step, and group-relative normalization across rollouts surfaces which action sequences succeed. A 3B model post-trained this way outperforms 72B baselines by 50%, showing the training method matters more than scale for multi-step tasks.
Tree-structured rollouts that branch from shared prefixes produce more distinct trajectories within a fixed token budget than independent chain sampling. This improves advantage estimation statistics and enables longer-horizon tasks within the same compute constraint.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.
DVAO weights objectives by their within-group variance, automatically up-weighting high-signal objectives and suppressing noise without hyperparameter tuning. This keeps advantage magnitudes bounded and replaces fixed scalarization constants with data-driven weighting.
Tree-GRPO uses branching structure to transform trajectory-level outcome rewards into step-level preference signals through sibling subtree comparison, eliminating the need for separate process reward models or step-level annotation while scaling with computational budget.
DRO shows that using rubrics to accept or reject rollout groups—rather than converting rubric scores into dense rewards—prevents reward hacking. This separation preserves the categorical strength of rubrics while letting token-level rewards optimize within valid answers.