Can tree structure alone convert outcome rewards into process supervision?

Agent RL with outcome-only rewards faces a sparse-supervision problem at long horizons. Multi-turn trajectories with thousands of tokens and many tool calls produce trajectory-level reward signals that cannot identify which specific steps contributed to success or failure. The standard responses — process reward models trained separately, dense intermediate rewards from human annotation — each have costs that limit deployment.

Tree-based Group Relative Policy Optimization (Tree-GRPO) finds a third path that uses the tree structure itself as the source of process supervision. Tree nodes represent complete agent interaction steps. Rollouts branch at decision points and share common prefixes. When outcome rewards arrive at the leaves, they back-propagate up the tree. At each branching point, the differences between sibling subtrees yield a preference-learning objective — sibling A's subtree did better than sibling B's, so the action choice that led to A gets reinforced over B's.

The key insight: process supervision does not require process-level reward design. The tree structure transforms trajectory-level outcome rewards into step-level preference signals automatically. The depth at which a branching point sits determines the granularity of the preference signal — shallow branches give coarse step-level supervision, deep branches give fine-grained sub-step supervision. Random tree expansion yields process signals of varying granularity without any annotation effort.

This is mechanically distinct from process reward models. PRMs train a separate scoring model on annotated intermediate steps, then use it as a reward signal during agent RL. Tree-GRPO does not train a separate model and does not require step-level annotations. The same outcome rewards that already exist for the task, combined with the structural information in the tree, suffice. The supervision quality differs — PRMs can encode richer notions of "good intermediate step," while Tree-GRPO only knows "this subtree did better than that one" — but the deployment cost is dramatically lower.

For agent-RL deployments where step-level annotation is impractical and outcome rewards are noisy, Tree-GRPO offers a plug-and-play path to process supervision that scales with budget rather than annotator effort.