What does process supervision reveal about step-level reasoning versus outcome rewards?
This explores what we learn by scoring each reasoning step (process supervision) instead of only judging the final answer (outcome rewards) — and the corpus reframes the question from 'which is better' to 'how do you get dense step-level signal without paying for it.'
This explores what process supervision reveals about reasoning when you grade each step rather than just the final answer — and the corpus's most striking move is to treat that distinction not as a binary but as a signal-density problem. The core tension is simple: outcome rewards are sparse and silent about *why* a chain failed, while process supervision is dense and diagnostic but historically required expensive human step-by-step annotation. Most of these papers are different attempts to get the diagnostic richness of process supervision without that annotation cost.
The clearest lesson is that step-level structure is often already latent in the work itself, waiting to be extracted from outcome signals. Tree-search rollouts convert a single trajectory-level reward into step-wise preferences by comparing sibling branches — the tree topology itself tells you which step diverged toward success or failure Can tree structure alone convert outcome rewards into process supervision?. Reverse curriculum learning slides the starting point of a problem backward from near-completion, so failures surface step-by-step using nothing but outcome feedback Can curriculum learning approximate expensive process supervision?. And more broadly, several methods exploit different structural features — tree shape, expert-aligned actions, tool-call positions — to manufacture dense step rewards from sparse outcomes Can trajectory structure replace hand-annotated process rewards?. The recurring finding: you rarely need a separately trained process reward model; the trajectory's own shape carries step-level information.
What process supervision *reveals*, then, is that outcome rewards are informationally impoverished, not just sparse. Numerical rewards tell a model it failed but not where or how — and models stuck on plateaus can suddenly produce correct solutions when given chain-of-thought critiques instead of a scalar score, because the language feedback carries the missing 'why' Can natural language feedback overcome numerical reward plateaus?. This is why judges that *reason about* each step outperform classifiers that merely label steps good or bad: the generative act of explaining a step is itself a richer supervisory signal, and it needs far less training data Can judges that reason about reasoning outperform classifier rewards?. The step is the unit where error actually accumulates — a decomposition of chain-of-thought shows genuine reasoning exists but compounds error with each additional step What three separate factors drive chain-of-thought performance?, which is exactly the failure mode outcome-only rewards are blind to.
There's a deeper, almost deflationary thread here too. Outcome-based RLVR may not be teaching new reasoning at all — it mostly sharpens sampling efficiency within capabilities the model already had from pretraining, to the point that spurious rewards work nearly as well as correct ones What does reward learning actually do to model reasoning?. That reframes why step-level signal matters: if outcome rewards can only *activate* existing strategies, then dense step rewards earn their keep precisely when a model needs to learn something it can't already do — which is why step-wise expert-similarity rewards let small models crack hard problems where every outcome-only rollout fails Can step-wise expert rewards help small models learn hard reasoning?.
The synthesis the corpus points toward is sequence, not rivalry. The strongest results come from imitation-then-exploration curricula: step-wise supervised RL first builds reasonable rollouts, then outcome-based RLVR sharpens them — each method alone underperforms the pairing, because the step-level phase makes the later outcome rewards informative in the first place Does sequencing imitation then exploration training improve reasoning?. The thing you didn't know you wanted to know: process supervision's real contribution may be less about being a better grader than about *making outcome rewards legible* — it populates the search space with good-enough reasoning so that the sparse final signal finally has something worth sharpening.
Sources 9 notes
Tree-GRPO uses branching structure to transform trajectory-level outcome rewards into step-level preference signals through sibling subtree comparison, eliminating the need for separate process reward models or step-level annotation while scaling with computational budget.
R3 progressively slides the reasoning start state backward from near-completion, creating a curriculum that reveals step-level failure modes using only outcome feedback. This achieves process supervision granularity without expensive human step annotations.
Tree-GRPO, Supervised RL, and ToolPO each convert sparse outcome rewards into dense step signals by exploiting different structural features—tree topology, expert-aligned actions, and tool-call positions—eliminating the need for annotated process reward models.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
StepWiser demonstrates that training judges to produce reasoning chains about policy reasoning—rather than classify steps—yields better judgment accuracy and data efficiency. Independent confirmation from GenPRM and ThinkPRM shows generative PRMs outperform discriminative ones with orders of magnitude less training data.
A shift cipher study decomposed CoT into three independent factors: output probability alone swings accuracy from 26% to 70%, memorization matches pre-training frequency patterns, and genuine reasoning exists but accumulates error with each step. This resolves the reason-or-memorize debate by showing LLMs do both simultaneously.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Supervised Reinforcement Learning rewards models by measuring alignment with expert actions at each step, providing dense learning signals even when all rollouts fail. This approach bridges the gap between rigid token-by-token imitation (SFT) and sparse outcome-only rewards (RLVR), and works best as a curriculum foundation before outcome-based refinement.
Running Supervised RL first to establish reasoning foundations, then RLVR to refine against verifiable rewards, substantially outperforms both methods in isolation. The imitation phase makes outcome rewards informative by creating reasonable rollouts the RL phase can then sharpen.