How does entropy collapse in reinforcement learning differ from entropy maintenance in graph reasoning?
This explores a contrast between two ways 'entropy' shows up in the corpus: as a failure mode in reinforcement learning (policies collapsing toward a single narrow strategy) versus as something deliberately preserved when a model navigates a structure like a knowledge graph under uncertainty.
This explores how 'entropy' plays opposite roles in two settings — as a thing that *collapses* and ruins RL training, and as a thing you'd want to *keep alive* when a model reasons its way through a graph. Worth saying up front: the corpus has a deep, well-developed account of the first and only an oblique account of the second, so the cleaner framing is that entropy collapse is a documented disease, and 'maintenance' is the family of cures and the conditions under which exploration must be protected.
On the collapse side, the corpus is unusually crisp. Policy entropy collapse is described as *the* primary bottleneck in scaling RL for reasoning: there's an empirical law where performance saturates exactly as policy entropy drops toward zero, because the model stops exploring and converges on whatever narrow strategy maximizes reward Does policy entropy collapse limit reasoning performance in RL?. The same mechanism is shown to recur in search agents — RL squeezes behavioral diversity the same way it does in reasoning, while supervised fine-tuning on diverse demonstrations keeps exploration broad Does reinforcement learning squeeze exploration diversity in search agents?. And it's not one problem but two: training-time entropy collapse and test-time variance inflation are *dual* failures of exploration-exploitation balance that need structurally separate fixes Why do reasoning models fail differently at training versus inference?.
What makes the 'maintenance' angle interesting is that entropy is not uniformly bad to lose — the corpus shows it should be preserved *selectively*. Only about 20% of tokens are high-entropy 'forking points' where the real reasoning decisions happen, and RLVR essentially works by adjusting those; train on just the forking tokens and you match full updates Do high-entropy tokens drive reasoning model improvements?. Even better, entropy isn't monolithic across a trajectory: RL training moves through two phases where *execution* entropy stabilizes (you want that part to converge) while *planning* entropy actually rises as strategic exploration becomes the bottleneck Does RL training follow a predictable two-phase learning sequence?. So healthy training looks like collapse in the routine parts and maintained entropy at the decision points.
This is exactly where graph reasoning enters. When a model navigates a knowledge graph, the corpus replaces exhaustive whole-graph reading with a *learned traversal policy* — Graph-O1 uses MCTS plus RL to step through the graph selectively, explicitly trading certainty about the full structure for decision-making under uncertainty Can learned traversal policies beat exhaustive graph reading?. Each branching node in a traversal is a forking point in the same sense as the high-entropy tokens above: collapse your exploration too early and you commit to a bad path; the tree search exists precisely to *maintain* a spread of live options. So the difference you're asking about is less a contradiction than a question of *where*: collapse is what you fight at the decision frontier (planning tokens, graph branch points), and convergence is what you want everywhere else (execution, settled sub-paths).
The reader-takeaway worth carrying away: the cures for collapse and the demands of graph reasoning are the same idea seen twice. Whether it's natural-language critiques injecting fresh signal to break a reward plateau Can natural language feedback overcome numerical reward plateaus?, or a tree search keeping multiple graph paths open, both are mechanisms for keeping the model's exploratory capacity alive at exactly the points where premature certainty would be fatal — and letting it collapse safely where the answer is already routine.
Sources 7 notes
Empirical law R = -a·exp(H) + b shows performance saturates when policy entropy approaches zero. Interventions like Clip-Cov, KL-Cov, and GPPO preserve exploratory capacity by managing entropy reduction during training.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Both failures stem from failed exploration-exploitation balance but occur at different timescales requiring structurally distinct interventions. Training-time fixes (entropy bonuses, critique diversity) cannot prevent inference-time variance inflation, and vice versa; both loops must be managed independently.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Graph-O1 replaces whole-graph ingestion with step-by-step agentic navigation using Monte Carlo Tree Search and reinforcement learning. This approach fits within LLM context windows while learning domain-specific traversal policies, though it trades certainty about the full graph for decision-making under uncertainty.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.