SYNTHESIS NOTE
Does training order reshape how models handle different task types?
Explores whether the sequence of multi-task RL training systematically affects model capabilities across structured and creative domains, and whether this ordering effect can be predicted and optimized.
The standard framing of Does policy entropy collapse limit reasoning performance in RL? treats entropy collapse as a uniform phenomenon — RL training decreases entropy. Omni-Thinker (2025) reveals this is domain-dependent: structured domains (math, coding) decrease output entropy, while open-ended domains (creative writing, dialogue) increase it.
This is not a minor observation — it makes training order a mechanistic variable, not just a scheduling convenience. If you train creative writing first and structured reasoning second, the structured training will collapse the entropy that creative training expanded, potentially degrading creative capability. If you train structured reasoning first and creative writing second, the creative training preserves and expands the model's expressive range. The ordering effect is predictable from backward transfer (BWT) measurements.
Omni-Thinker uses BWT-guided scheduling: order tasks so that later tasks experience minimal negative backward transfer from earlier tasks. The approach uses hybrid rewards — verifiable (rule-based) for deterministic domains + preference-based (LLM-as-Judge) for subjective domains — enabling unified training across domain types within a single policy. The "short-form" QA tasks condition on distractors to reduce reward hacking from random guessing.
The gains are substantial: 6.2% over joint multi-task training, 12.4% over model merging. The accuracy of final multi-task models is well-predicted by forgettability rankings, even under simplifying assumptions — suggesting BWT-guided scheduling has principled theoretical grounding.
This extends Does gradually tightening token budgets beat fixed budget training? from temporal budgets to task ordering: the dimension that matters for multi-task RL is not just how much compute per task, but which tasks come first. And it enriches the entropy collapse understanding: entropy collapse is not a bug to fix everywhere — in structured domains, it reflects desirable precision. The problem is when structured-domain entropy collapse propagates to damage open-ended capabilities.
Inquiring lines that read this note 111
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can scaling alone produce compositional generalization in neural networks? What causes foundation models to develop task heuristics instead of understanding?- Does the heuristic dominance ratio vary predictably across model architectures?
- Do task-specific heuristics improve gradually or appear suddenly at scale?
- What makes some tasks bounded enough for reliable RL?
- Does task ordering affect multi-task reinforcement learning outcomes?
- Can in-context learning replicate the timing effects that RL teaches models?
- How do residual connections and layer norm stabilize training in deep RL?
- Can smaller models achieve domain expertise through focused RL training?
- Does RL refine existing knowledge or discover entirely new capabilities?
- Can negative reinforcement alone match full RL performance on domain tasks?
- How does RL compress reasoning path diversity during training?
- What makes software engineering environments better suited for RL than other interactive domains?
- What distinguishes RL that creates new capabilities from RL that merely teaches timing?
- What makes pretraining composition more important than reward engineering?
- How do RL training and base models differ in creating MI peaks?
- Does format-based pretraining determine how models respond to reinforcement learning?
- How do self-evolving curricula help RL break beyond base model capability boundaries?
- Why does RL behavior differ between standard reasoning tasks and complex planning domains?
- What scaling properties emerge from RL training dynamics beyond verification?
- Why do single-turn RL methods fail to generalize to multi-turn tasks?
- Why do overtrained domains show different RL training outcomes than novel tasks?
- What training duration is actually needed for RL to expand capabilities?
- How does prolonged RL training differ from standard RLVR approaches?
- Why does supervised fine-tuning on diverse demonstrations expand exploration diversity compared to RL?
- Does the pretrained model prior limit RL search capability more than the optimization algorithm itself?
- Why does outcome-based RL specifically lose diversity during training?
- Does specification gaming emerge universally or depend on task structure?
- Does RL teach models new reasoning or just better timing?
- Can RL training on small verifiable tasks transfer to real-world AI research?
- How do training objectives shape what a world model actually learns?
- Do different function-calling subtasks have different entropy profiles during training?
- Can backward transfer measurements reliably predict optimal multi-task training order?
- Which recipe choices determine the asymptotic ceiling in RL training?
- Can continuous spectrum training outperform sequential SFT-then-RL stages?
- Can trajectory quality filtering improve model training in noisy environments?
- How does trajectory burstiness compare to other structural properties that shape emergent capabilities?
- How does a challenger's escalating difficulty function as curriculum?
- How does absolute-advantage weighting concentrate training on boundary cases?
- How should multi-objective post-training balance competing behavioral goals?
- How do weight visualizations reveal temporal structure in cyclic training?
- How does model scale affect anticipatory behavior in structured training?
- What distinguishes surface mechanisms from the training regimes that produce them?
- Can feedback loop frequency harm performance on finite task sets?
- Why does alternating RL training stabilize learning better than simultaneous updates?
- What causes models to develop domain capability cliffs after specialization?
- What task structures benefit most from geometric parameter merging?
- Do different domains require different types of model investment?
- What capabilities actually require massive scale versus specialized training regimes?
- What performance trade-offs emerge when composing multiple independently trained model capabilities?
- Can depth scaling and breadth scaling unlock independent capability axes?
- Can RL format selection explain performance gains attributed to algorithmic improvements?
- How do finetuning and pretraining improvements differ in their effects on model capabilities?
- Are cheap testbeds and skewed task distributions linked by design necessity?
- How much of the combinatorial task space must training data cover?
- Why does training data format matter more than domain content?
- Why does training data format matter more than its domain content?
- How does training data distribution create asymmetric competence across relation types?
- Why does training order matter across different domain types?
- Why does curriculum order matter when information theory says data order is irrelevant?
- Can training order and structure shape what networks retain and learn?
- Can training data organization by capability outperform source or task-based mixing?
- Does constraining AI access during early task phases preserve skill formation?
- Does AI training preserve learning that transfers to independent subsequent tasks?
- How do task difficulty and skill type interact in model performance?
- Does specialized training in one domain create capability cliffs elsewhere?
- Can training on diverse related tasks be more efficient than task-specific training?
- How do task frequency and complexity interact with model capacity during training?
- Does task diversity in pretraining data transfer reasoning better than larger models?
- What specific tasks should evaluate whether models understand pedagogical sequencing?
- Does knowledge structure matter more than knowledge volume for model training?
- Can curriculum graphs as training data improve model understanding of prerequisite chains?
- Can diversity-aware RL objectives prevent format convergence?
- Does critique training improve exploration diversity during model training or only test time?
- Can explicitly optimizing for semantic diversity during RL training improve both quality and variation?
- Does training on granular tasks beat training on the full function calling problem?
- How should skill libraries coordinate with gradient-based weight optimization?
- How does training order affect knowledge acquisition in language models?
- How do chunk-based step segmentation and trajectory structure modeling differ?
- What structural differences emerge between early generic skills and later meta-strategy skills?
- How does stage-wise training scheduling resolve conflicts between constraint-following and creative tasks?
- Why does decomposition ability transfer across domains but solving ability does not?
- Can architectural changes reorder when uncertainty and empowerment signals influence decisions?
- How does task decomposability differ from task variance as a design criterion?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Does training order reshape how models handle di…
Does policy entropy collapse limit reasoning perfo…
Does gradually tightening token budgets beat fixed…
Why do reasoning models fail differently at traini…
Does RL training collapse format diversity in pret…
Can isolating task-specific parameters prevent mul…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Does policy entropy collapse limit reasoning performance in RL?
As reinforcement learning models become more confident in their policy choices, entropy drops and performance plateaus. Can we identify and counteract this bottleneck to sustain scaling?
entropy dynamics are domain-dependent, not uniformly negative; structured tasks decrease entropy while creative tasks increase it
-
Does gradually tightening token budgets beat fixed budget training?
Can models learn reasoning more efficiently by starting with generous token allowances and progressively constraining them, rather than training with fixed budgets from the start? This matters because it addresses how to teach models to think effectively while remaining concise.
BWT-guided scheduling extends curriculum insight from temporal budgets to task ordering
-
Why do reasoning models fail differently at training versus inference?
Reasoning models exhibit two distinct failure modes—entropy collapse during training and variance inflation during inference—that appear unrelated but may share underlying causes. Understanding these dual problems could reveal whether separate or unified solutions are needed.
Omni-Thinker adds that entropy direction depends on task type, further complicating the dual problem
-
Does RL training collapse format diversity in pretrained models?
Exploring whether RL fine-tuning systematically selects one output format from pretraining while suppressing others, and how this selection mechanism drives performance gains.
multi-task training with BWT scheduling may partially address format convergence by exposing the model to diverse task types
-
Can isolating task-specific parameters prevent multi-task fine-tuning interference?
Explores whether identifying and protecting task-specific parameter regions can prevent the performance degradation that occurs when fine-tuning models on multiple tasks simultaneously. This matters because it could enable safe multi-task adaptation without sacrificing individual task performance.
complementary multi-task approach: CPI-FT addresses interference through spatial parameter isolation while Omni-Thinker uses temporal task ordering; CPI-FT shows temporal scheduling alone is insufficient, suggesting combining both spatial isolation and BWT-guided ordering could further improve multi-task training
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling
- From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
- Sharpening Tax in Post-Training
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- Reinforcement Learning with Rubric Anchors
- Eliciting Reasoning in Language Models with Cognitive Tools
- RAGEN-2: Reasoning Collapse in Agentic RL
- Post-training makes large language models less human-like
Original note title
multi-task rl reveals complementary entropy dynamics — structured domains systematically decrease output entropy while creative domains increase it making training order a mechanistic variable