SYNTHESIS NOTE
Topics›Reasoning Critiques›this note

Does RL training collapse format diversity in pretrained models?

Exploring whether RL fine-tuning systematically selects one output format from pretraining while suppressing others, and how this selection mechanism drives performance gains.

Synthesis note · 2026-02-22 · sourced from Reasoning Critiques
How should we spend compute at inference time? How do you build domain expertise into general AI models?

A study with full pretraining transparency (models pretrained from scratch on known open datasets) reveals a striking structural pattern: RL fine-tuning does not simply improve reasoning — it systematically selects for and amplifies a single format from the pretraining mixture while collapsing all others.

The mechanism: early in RL training (within the first epoch), the model shifts toward generating outputs in the format of one specific distribution — code-like formats for smaller models, natural language formats for larger models. This transition coincides with the largest accuracy gain, suggesting the selection of a dominant format is what drives improvement, not a gradual enhancement across all formats.

Key findings:

This is distinct from Does policy entropy collapse limit reasoning performance in RL? in an important way. Entropy collapse describes diversity reduction within an output distribution. The echo chamber finding describes distribution selection: RL picks one distribution and amplifies it at the expense of all others. It is a format-level convergence, not just a diversity-level collapse.

The implication for practitioners: RL fine-tuning results depend on what the pretraining data mixture looks like, but this dependence is largely hidden when starting from existing pretrained models whose training data is proprietary. The performance gains attributed to RL algorithms may partially reflect which pretraining distribution was selected, not algorithmic superiority.

Inquiring lines that read this note 272

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What are the consequences of models training on synthetic data? Can AI-generated outputs constitute genuine knowledge or valid claims? How can AI alignment serve diverse human preferences at scale? How do prompt structure and constraints affect model instruction reliability? How can identical external performance mask different internal representations? How can AI systems learn from failures without cascading errors? What structural factors drive popularity bias in recommendation systems? How should we design LLM systems to maintain alignment and control? Can prompting inject entirely new knowledge into language models? How do self-generated feedback mechanisms enable effective model learning? When does optimizing for quality undermine the value of diversity? Does fine-tuning modify underlying model capabilities or only behavioral outputs? How does memorization interact with learning and generalization? What pretraining choices and baseline capability constrain reinforcement learning gains? How can conversational AI maintain consistent personas across conversations? Can alternative training methods improve on supervised fine-tuning for language models? Why does supervised fine-tuning improve accuracy while degrading reasoning quality? Does alignment training create blind spots in detecting genuine safety threats? What determines success in training models on multiple tasks? Can language model RL training avoid reward hacking and misalignment? What constrains reinforcement learning's ability to expand model reasoning? What articulatory information do speech signals carry that text cannot? How does AI adoption affect human skill development and labor equality? How do training priors constrain what context information can override? Why do continual learning scenarios trigger catastrophic forgetting and interference? Why does reinforcement learning suppress output diversity compared to supervised fine-tuning? Do base models contain latent reasoning that training can unlock? Why does training format shape reasoning strategy more than domain content? How do policy learning algorithm choices affect multi-objective optimization stability? Which computational strategies best support reasoning in language models? Does RLHF training sacrifice accuracy and grounding for user agreement? How do training data properties shape reasoning capability development? How does test-time aggregation affect reasoning correctness and reliability? Do autonomous architecture discoveries follow predictable scaling laws? Can model confidence signals reliably improve reasoning quality and calibration? Why do benchmark improvements fail to reflect actual reasoning quality? How do we evaluate AI systems when user perception misleads actual performance? Why does finetuning cause catastrophic forgetting of model capabilities? Do corrupted reasoning traces serve as effective supervision signals? Can next-token prediction alone produce genuine language understanding? How should agents balance memory condensation to optimize context efficiency? How does policy entropy collapse constrain reasoning-focused reinforcement learning? What prevents language models from reliably adopting diverse personas? When does architectural design matter more than raw model capacity? How do evaluation mechanisms prevent error accumulation in autonomous research systems? Does reinforcement learning teach reasoning or just when to reason? What actually drives chain-of-thought reasoning improvements in language models? What makes weaker teacher models effective for stronger student training? Do language model representations contain causally steerable task-specific features? How should dialogue recommender systems manage conversation history and state? Do harness improvements transfer across model scales or memorize shortcuts? How do knowledge injection methods compare across cost and effectiveness? How can AI agents autonomously learn and transfer skills across tasks? What limits mechanistic interpretability's ability to characterize models? Can model routing outperform monolithic scaling as an efficiency strategy? Can AI systems balance emotional competence with factual reliability? Is model self-awareness based on genuine introspection or pattern matching? Why does consolidated memory sometimes degrade agent performance? Does decoupling planning from execution improve multi-step reasoning accuracy? How does example difficulty affect learning efficiency in language models? Why do self-improving systems struggle without clear external performance metrics? Does self-reflection enable models to reliably correct their errors? Can language model hallucination be prevented or only managed?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 154 in 2-hop network ·dense cluster Open in graph ↗
Does RL training collapse format diversity in pr… Does policy entropy collapse limit reasoning perfo… Why do reasoning models fail differently at traini… Can simple rewards alone teach complex domain reas… Does RL improve domain reasoning by adding knowled… Does reinforcement learning squeeze exploration di… Why does RLVR training narrow a model's problem so…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do smaller models favor code formats while larger models prefer natural language?How does KL penalty strength affect the degree of format collapse during RL?Can RL format selection explain performance gains attributed to algorithmic improvements?Is distribution selection during RL the same compression mechanism as entropy collapse?
Original note title

rl post-training converges on a single dominant pretraining distribution format, suppressing all others