SYNTHESIS NOTE
Topics›Reasoning Critiques›this note

Does RL training collapse format diversity in pretrained models?

Exploring whether RL fine-tuning systematically selects one output format from pretraining while suppressing others, and how this selection mechanism drives performance gains.

Synthesis note · 2026-02-22 · sourced from Reasoning Critiques

A study with full pretraining transparency (models pretrained from scratch on known open datasets) reveals a striking structural pattern: RL fine-tuning does not simply improve reasoning — it systematically selects for and amplifies a single format from the pretraining mixture while collapsing all others.

The mechanism: early in RL training (within the first epoch), the model shifts toward generating outputs in the format of one specific distribution — code-like formats for smaller models, natural language formats for larger models. This transition coincides with the largest accuracy gain, suggesting the selection of a dominant format is what drives improvement, not a gradual enhancement across all formats.

Key findings:

This is distinct from Does policy entropy collapse limit reasoning performance in RL? in an important way. Entropy collapse describes diversity reduction within an output distribution. The echo chamber finding describes distribution selection: RL picks one distribution and amplifies it at the expense of all others. It is a format-level convergence, not just a diversity-level collapse.

The implication for practitioners: RL fine-tuning results depend on what the pretraining data mixture looks like, but this dependence is largely hidden when starting from existing pretrained models whose training data is proprietary. The performance gains attributed to RL algorithms may partially reflect which pretraining distribution was selected, not algorithmic superiority.

Inquiring lines that read this note 353

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Can synthetic data preserve necessary diversity to prevent model collapse? Why do multiple language models independently generate similar outputs? Does preference-optimized training sacrifice accuracy and calibration for perceived quality? How does verifier integration affect the generation-verification gap? What causes foundation models to develop task heuristics instead of understanding? How do different post-training structures affect learning outcomes? How can recommendation systems balance individual and population-level goals? How does explicit structure improve LLM reasoning on complex tasks? Does prompting inject new knowledge into models or activate existing knowledge? Can language models develop genuine communicative grounding with humans? Does semantic diversity improve reasoning quality in reinforcement learning? Can scaling alone produce compositional generalization in neural networks? What training data structures and signals produce robust learning across distributions? How do benchmark scores relate to real-world task completion? What prevents or enables error accumulation in self-training systems? Does reinforcement learning create new reasoning capabilities or refine existing ones? Is model misalignment inherent to base models or induced by alignment procedures? Does reward hacking in training cause emergent misalignment? How does policy entropy collapse limit reasoning RL scaling? Does learning articulatory physics explain speech communication superiority? How does AI adoption reshape employment and wage outcomes? Can prompts override training-embedded biases in language models? How do pretraining data composition and task diversity shape reasoning? Does training data format shape reasoning more than domain content? Does evolutionary optimization at test time improve model performance effectively? How faithfully does iterative DPO approximate online RL? How does model scale affect fine-tuning and downstream task performance? When and why does aggregating multiple reasoning samples improve performance? Why does separating planning from execution enhance reasoning performance? Does reasoning fine-tuning trade off other critical model abilities? How does model awareness of deployment context affect strategic behavior? What are the trade-offs between memory capacity and continuous task learning? Do language models learn underlying linguistic structure or merely surface patterns? Do reasoning traces accurately represent what models actually compute? How can parameter sparsity improve interpretability while maintaining model capability? How can persona-based systems overcome their inherent limitations in modeling diversity? How does silent agreement emerge and undermine deliberation in multi-agent systems? Can persona prompting reliably change how language models behave? How do frontier models develop behaviors that escape detection? What actually drives chain-of-thought performance gains in reasoning tasks? How do different reward signals affect model reasoning and calibration differently? What mechanisms cause models to behave deceptively despite alignment training? Should reward models emphasize process understanding or only final outcomes? How can agents learn and retain useful knowledge across diverse environments? What conversational structures and patterns determine dialogue effectiveness and coherence? What determines how neural networks organize their internal knowledge representations? Can harness optimization rival model scaling in effectiveness? Which agent-created code should persist in infrastructure versus remain task-specific? When does routing outperform scaling as a performance solution? Do base models contain latent reasoning capabilities that training can activate? What constraints limit conversational AI's effectiveness in therapeutic contexts? How can we distinguish genuine model self-knowledge from behavioral performance? How do mechanistic features and representations drive model behavior? Why does consolidated memory sometimes hurt continual learning more than no memory? How do standardized forms and measurement artifacts shape observable evidence? Can AI agents learn to improve performance from experience? Do standard benchmarks reveal AI capability in real-world contexts? Does generative AI replace coordination with individual productivity? Can we reliably detect when models engage in reward hacking? Does architectural recurrence enable computational capabilities that fixed-depth models cannot achieve? How can we better validate AI research and prevent misconceptions? What causes hallucination and how can it be addressed? Can safety evaluations accurately measure true capabilities when models obscure them? Can validator agreement alone ensure semantic correctness of reasoning? Do language models genuinely understand or primarily match statistical patterns? Does AI assistance enhance or undermine professional skill development? How does scaling test-time compute compare to increasing model size? How do different editing strategies affect output quality and agent safety? Can we reliably distinguish AI-generated from human-written text? How can model evaluations remain robust against gaming and evaluator drift? Why do human-AI teams underperform compared to their individual components? When do multi-agent systems outperform single agents?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 156 in 2-hop network ·dense cluster Open in graph ↗
Does RL training collapse format diversity in pr… Does policy entropy collapse limit reasoning perfo… Why do reasoning models fail differently at traini… Can simple rewards alone teach complex domain reas… Does RL improve domain reasoning by adding knowled… Does reinforcement learning squeeze exploration di… Why does RLVR training narrow a model's problem so…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do smaller models favor code formats while larger models prefer natural language?How does KL penalty strength affect the degree of format collapse during RL?Can RL format selection explain performance gains attributed to algorithmic improvements?Is distribution selection during RL the same compression mechanism as entropy collapse?
Original note title

rl post-training converges on a single dominant pretraining distribution format, suppressing all others