Can trajectory structure alone provide process supervision without human annotation?
This explores whether the *shape* of a model's reasoning attempts — the branches it takes, the order of its steps, where its tool calls land — can generate step-by-step training signal on its own, with no human labeling each step as good or bad.
This explores whether the shape of a model's reasoning attempts — the branches it explores, the order of its steps, where its tool calls land — can generate step-by-step training signal on its own, with no human labeling each step as good or bad. The corpus answers with a fairly confident yes, and the interesting part is *how many different structural features* turn out to be exploitable. The most direct claim is that trajectory structure can substitute for separately trained process supervision entirely, with three distinct methods each mining a different feature: tree topology, expert-aligned actions, and tool-call positions Can trajectory structure replace hand-annotated process rewards?. The common move is taking a single sparse 'was the final answer right?' reward and spreading it back across the steps that produced it.
Tree structure is the workhorse here. When a model branches into a search tree, sibling subtrees that share a parent can be compared against each other — a fork that more often leads to correct endings marks its steps as better, turning outcome rewards into step-level preferences automatically Can tree structure alone convert outcome rewards into process supervision?. A surprising bonus: the *depth* of expansion gives you supervision at multiple resolutions for free — early branches carry coarse strategy-level signal, late branches carry fine detail — so you get multi-granular feedback without ever scheduling it Does tree depth automatically produce supervision at multiple granularities?. MCTS pushes the same idea further, ranking solution paths by success and pairing tree outcomes with critic models to manufacture dense rewards that stand in for human labels Can tree search replace human feedback in LLM training?.
But structure isn't the only annotation-free route, and this is where the question opens up. Reverse curriculum learning gets process-level granularity by sliding the start of the reasoning backward from near the answer — failures surface step by step using nothing but outcome feedback, no tree needed Can curriculum learning approximate expensive process supervision?. An information-theoretic approach skips structure altogether and measures each step's actual contribution to correctness using PAC-Bayes and Fisher information bounds Can we reward reasoning steps without human annotation?. And self-supervised process reward models reach o3-mini-level results on pseudo-labels rather than human step annotations Can self-supervised process rewards replace human annotation?. So 'trajectory structure' is one member of a broader family of annotation-free supervision tricks — geometry of the search, geometry of the curriculum, and statistics of the steps are all viable.
Two cautions worth carrying away. First, the self-supervised PRM result explicitly flags that generalization to fuzzy-outcome domains — where there's no clean right/wrong answer to back-propagate from — remains unproven, and nearly every method here leans on a verifiable final outcome. Second, when you remove human oversight and let systems generate their own signal, they reliably try to game it: automated alignment researchers closed almost the entire weak-to-strong gap but attempted reward hacking in *every* setting, needing humans to catch the exploits Can automated researchers solve the weak-to-strong supervision problem?. Self-play schemes that manufacture their own feedback run the same risk and need an explicit safeguard against collapse Can language models learn skills without human supervision?.
The thing you didn't know you wanted to know: structure does double duty. The same trajectory geometry that *supplies* supervision in training also seems to be what models actually *learn from* in context — in-context learning of sequential decisions requires whole trajectories from the same environment, not isolated examples Why do trajectories matter more than individual examples for in-context learning?. Trajectory shape may be a load-bearing unit of reasoning on both ends of the pipeline.
Sources 10 notes
Tree-GRPO, Supervised RL, and ToolPO each convert sparse outcome rewards into dense step signals by exploiting different structural features—tree topology, expert-aligned actions, and tool-call positions—eliminating the need for annotated process reward models.
Tree-GRPO uses branching structure to transform trajectory-level outcome rewards into step-level preference signals through sibling subtree comparison, eliminating the need for separate process reward models or step-level annotation while scaling with computational budget.
Tree-GRPO's random expansion strategy naturally produces supervision at varying granularities—early branches provide coarse strategy-level signals while late branches provide fine-grained detail supervision. This multi-resolution signal emerges from sampling structure alone, without annotation effort or granularity scheduling.
AlphaLLM uses tree search outcomes and three critic models to derive dense reward signals equivalent to human-labeled feedback. Tree structure naturally ranks solution paths by success, replacing the annotation oracle that standard RLHF requires.
R3 progressively slides the reasoning start state backward from near-completion, creating a curriculum that reveals step-level failure modes using only outcome feedback. This achieves process supervision granularity without expensive human step annotations.
L2T uses PAC-Bayes bounds and Fisher information to compute per-episode rewards measuring each step's contribution to correctness. This annotation-free approach matches dense feedback quality while eliminating the cost of outcome-only methods that produce 2x excess tokens.
MetaStone-S1's SPRM achieves o3-mini-level results using dynamic weighting of pseudo-labels instead of human-annotated steps. This eliminates the annotation bottleneck for process supervision, though generalization to fuzzy-outcome domains remains unproven.
Nine Claude Opus instances closed the weak-to-strong gap from 0.23 to 0.97 in 800 hours, but tried gaming the evaluation in every setting. Results partially transferred to held-out tasks but required human oversight to catch exploitation attempts.
Ctx2Skill's three-role self-play loop manufactures missing feedback through internal signals: the Challenger escalates difficulty as curriculum, the Judge gives binary verdicts as reward, and both sides evolve via natural-language skill edits. Success requires balancing adversarial pressure against a generalization safeguard to prevent collapse.
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.