Why do language models collapse into generic templates?
Explores whether low reward variance during RL training causes policies to abandon input-specific reasoning in favor of boilerplate responses, and whether filtering for high-variance prompts can prevent this failure.
Naming template collapse is diagnostic; RAGEN-2 also gives a mechanism for why it happens, framed as a signal-to-noise ratio problem in RL. The task-learning signal comes from reward variance across samples of the same prompt. When that within-input reward variance is low — the prompt is too easy, too hard, or otherwise uninformative — the task gradient it produces is weak. With the task signal weak, the regularization terms in the objective (entropy bonuses, KL penalties, and similar) dominate the update, and their effect is to smooth the policy toward generic outputs. The result is that the policy is pushed toward input-agnostic templates: the collapse is not a random drift but the predictable outcome of regularization out-muscling a starved task gradient.
This turns a mysterious failure into a controllable one. Because reward variance is the culprit and it is cheap to compute, the paper proposes SNR-Aware Filtering — selecting high-signal prompts (high reward variance) before each parameter update — which improves performance across tasks, scales, and modalities and slots into existing pipelines. The mechanism generalizes a familiar theme: Does policy entropy collapse limit reasoning performance in RL? describes regularization/optimization forces flattening a policy, and here the same regularization-dominates dynamic produces collapse along a different axis (cross-input distinguishability rather than token entropy). It also motivates curriculum-by-variance: the useful training signal lives at the prompts where reward is neither guaranteed nor impossible, echoing why Why does RL succeed more on some tasks than others? — signal quality, not signal presence, is what trains reasoning.
Inquiring lines that read this note 81
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How faithfully does iterative DPO approximate online RL?- Can trust region constraints prevent the sample inefficiency problems of RLHF?
- Do policy gradient methods carry less training signal than other RL algorithms?
- What causes on-policy distillation to become unstable at scale despite dense rewards?
- What distinguishes high-signal prompts from low-signal ones in RL training?
- How do weights, selection, and prompts create different geometric landscapes of accessible behaviors?
- Why does outcome-only reinforcement learning need more than double the tokens to train agents?
- How should training distribution distance be defined when the policy evolves?
- Why does alternating RL training stabilize learning better than simultaneous updates?
- How does policy entropy collapse constrain zero RL scaling for reasoning?
- How does regularization dominance explain policy entropy collapse in reasoning?
- Does stable entropy in policy training actually guarantee stable reasoning behavior?
- What causes policy entropy collapse in scaling language model reasoning?
- Does post-training collapse policy entropy more than base model sampling?
- Why does prolonged RL with entropy control beat base models at all pass@k levels?
- What makes student-teacher distributional mismatch derail on-policy distillation?
- How do learning dynamics on one example shift predictions on other responses?
- How does contrastive feedback improve compression guideline quality versus weaker optimization?
- Why does exploration diversity behave differently under reinforcement learning versus supervised fine-tuning?
- Does specification gaming emerge universally or depend on task structure?
- Can RL training on small verifiable tasks transfer to real-world AI research?
- Does RL scaling deliver much worse returns than inference scaling?
- Does RL primarily teach output format rather than new knowledge?
- Does a tight learning-rate bound prevent skills from escaping poor starting points?
- Can RL-trained policies outperform text-space optimizers for evolving skill repositories?
- Why do generic skill descriptions evolve into execution-oriented ones?
- What distinguishes bounded classification models from open-ended agentic systems?
- Do shared prompts and infrastructure keep model biases correlated?
- Why does grouping feedback by shared correction pattern improve prompt generalization?
- Is lower context-following a failure or appropriate model behavior?
- How does prompt-based compression compare to reinforcement learning approaches in cost and effectiveness?
- What makes a synthetic belief robust versus generative for downstream learning?
- Can scaling up contradictory training data overcome unpredictable override effects?
- Is model collapse a property of data replacement or synthetic data itself?
- Do identical task structures mean repeated instances or new synthetic samples with same design?
- How does task decomposability differ from task variance as a design criterion?
- What alignment training removes from an agent's available speech acts?
- Does representational distance account hold for on-policy reinforcement learning training?
- How do hidden partitions in evaluators compare across training and selection substrates?
- How does reference-anchoring compare to training judges with reinforcement learning?
- Can adaptive rubric generation defend against policy exploitation of criteria?
- Can diversity across multiple verifiers cancel out individual biases in retraining?
- Why is the fast non-parametric loop vulnerable to overfitting differently than model weights?
- Why is sparse per-student data a bottleneck for building adaptive tutoring systems?
- How does tournament selection without gradients compare to gradient-based hyperparameter tuning?
- How does learnability at the observer's current state prevent novelty from breaking model reasoning?
- Can applicability conditions and veto rules make self-training stable across substrates?
- What causes code quality to degrade across multiple rounds of recursive self-training?
- Does higher temperature sampling help bootstrapping loops generate more training data?
- How do complexity, diversity, and real-world fidelity interact in agent training?
- Can distillation help AIs scale their learned objectives across many copies?
- Do larger language models naturally avoid repetitive loops in agent loops?
- Do human judges and language models agree on what counts as AI slop?
- Can lexical repetition and syntactic structure predict whether text feels templated?
- What emergent internal mechanisms in large language models carry unintended safety risks?
- How do language models infer a benchmark's purpose without seeing its examples?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does policy entropy collapse limit reasoning performance in RL?
As reinforcement learning models become more confident in their policy choices, entropy drops and performance plateaus. Can we identify and counteract this bottleneck to sustain scaling?
extends: same regularization-dominates dynamic, but collapse along cross-input distinguishability rather than token entropy
-
Why does RL succeed more on some tasks than others?
Reinforcement learning shows wildly different improvement rates across conversational tasks—from near-total capability unlock to modest gains. What determines whether RL will transform performance or produce incremental progress?
relates: both hinge on reward-signal quality, not mere presence, as the driver of learning
-
Can entropy metrics detect when reasoning becomes formulaic?
Entropy measures diversity within single inputs, but cannot reveal whether a model's reasoning actually varies across different inputs. This matters because high entropy can mask template collapse, where reasoning looks varied but is fundamentally the same.
grounds: this note supplies the causal mechanism for the failure that note diagnoses
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- RAGEN-2: Reasoning Collapse in Agentic RL
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- Efficient Reinforcement Learning via Large Language Model-based Search
- Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
Original note title
low reward variance is what drives reasoning toward input-agnostic templates — weak task gradients let regularization erase cross-input differences