How do critique models prevent policy entropy collapse during reasoning training?
This explores how adding a critique step to the reasoning-RL loop keeps a model from collapsing into a single overconfident solution path — and why that turns out to matter more than the accuracy bumps it's usually credited for.
This explores how critique models guard against *policy entropy collapse* — the failure where a reasoning model, trained with reinforcement learning, narrows so hard onto one way of solving problems that it stops exploring and its performance hits a ceiling. The corpus frames this collapse as the central bottleneck, not a side effect: there's an empirical law, R = -a·exp(H) + b, showing that performance saturates exactly as policy entropy drops toward zero Does policy entropy collapse limit reasoning performance in RL?. So anything that preserves exploratory capacity isn't a nicety — it's what raises the ceiling.
The most direct answer is that critique models keep the solution distribution wide. Step-level critique inside the training loop counteracts 'tail narrowing' — the gradual loss of rare-but-valid solution branches across self-training iterations — and the note is emphatic that this diversity-preservation is *more fundamental* than the test-time accuracy gains critique is usually advertised for Do critique models improve diversity during training itself?. That reframes critique from a quality filter into an entropy-management tool.
Why critique specifically, rather than just turning up an entropy bonus? Because numerical rewards are information-starved. A scalar reward tells the model *that* it failed but not *why* or *how to improve*, so models stuck on a plateau stay stuck. Natural-language, chain-of-thought critiques carry exactly the missing signal — Critique-GRPO shows models breaking through numerical-reward plateaus once they're told what went wrong in words Can natural language feedback overcome numerical reward plateaus?. Language feedback injects new directions to explore instead of just sharpening the gradient on directions already taken.
Here's the part worth knowing that the question doesn't ask: critique only addresses *half* the exploration problem. Training-time entropy collapse and test-time variance inflation are treated as dual, separately-fixable failures — critique diversity and entropy bonuses fix the training loop, but they cannot prevent the model from going erratic at inference, and vice versa Why do reasoning models fail differently at training versus inference?. So a critique model is a training-time intervention with a hard boundary. There's also a where-to-spend-it nuance: only about 20% of tokens are high-entropy 'forking points' that actually drive learning, which suggests critique pressure is best aimed at those decision tokens rather than the whole trace Do high-entropy tokens drive reasoning model improvements?.
Stepping back, critique fits a broader pattern in the corpus: it's one of several mechanisms — alongside confidence-as-reward Can model confidence work as a reward signal for reasoning? and minimal RL steering — that *elicit* reasoning the base model already latently contains rather than installing it from scratch Do base models already contain hidden reasoning ability?. Read that way, preventing entropy collapse isn't about teaching the model more; it's about not prematurely silencing the alternative reasoning paths the model could already have taken.
Sources 7 notes
Empirical law R = -a·exp(H) + b shows performance saturates when policy entropy approaches zero. Interventions like Clip-Cov, KL-Cov, and GPPO preserve exploratory capacity by managing entropy reduction during training.
Step-level critique in the training loop counteracts tail narrowing and maintains solution diversity across self-training iterations. This training-time benefit—preventing premature convergence—is more fundamental than test-time accuracy gains.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Both failures stem from failed exploration-exploitation balance but occur at different timescales requiring structurally distinct interventions. Training-time fixes (entropy bonuses, critique diversity) cannot prevent inference-time variance inflation, and vice versa; both loops must be managed independently.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.