How does stage-wise training scheduling resolve conflicts between constraint-following and creative tasks?
This explores how the *order* you train tasks in—structured, rule-following work versus open-ended creative work—can keep one kind of skill from quietly destroying the other.
This explores how the order you train tasks in can stop constraint-following from eating creativity (and vice versa). The corpus has a sharp, mechanical answer: the two task types pull a model's output entropy in opposite directions, so the schedule isn't a tuning convenience—it's the thing that decides which capability survives. The clearest case is Omni-Thinker, where structured domains (math, format-following) *lower* a model's output entropy while creative domains *raise* it; training them jointly lets the entropy-collapsing tasks flatten the open-ended ones. Training the structured tasks first, guided by how much each stage degrades earlier skills, recovers about 6% over throwing everything in at once Does training order reshape how models handle different task types?. The conflict is real and physical, not stylistic—and sequence is the lever.
Why does order matter so much? Because RL training is itself staged, even when you don't design it to be. One line of work finds RL moves through two phases on its own—first nailing execution correctness, then shifting the bottleneck to strategic planning, with planning entropy rising as execution entropy settles Does RL training follow a predictable two-phase learning sequence?. If the model is naturally consolidating procedure before it explores, then front-loading the constraint-heavy, low-entropy tasks works *with* that grain instead of against it. Stage-wise scheduling is partly just respecting a sequence the optimizer was already going to follow.
The same imitation-then-exploration logic shows up in curriculum design directly: running supervised reasoning first to build a foundation, then verifiable-reward RL to sharpen it, beats either method alone—because the early phase produces reasonable attempts that make the later reward signal *informative* Does sequencing imitation then exploration training improve reasoning?. Read alongside Omni-Thinker, a pattern emerges: stages aren't just ordered for convenience, they're ordered so each phase prepares the ground the next one needs, and so the destructive phase can't run before the fragile capability is established.
There's a darker reason sequencing matters, and it's the thing you didn't know you wanted to know: RL doesn't gently blend behaviors—it *collapses* them. Controlled experiments show RL amplifies a single dominant output format from pretraining within the first epoch and actively suppresses the alternatives, with the winner determined by model scale rather than which format is actually best Does RL training collapse format diversity in pretrained models?. Creative diversity is exactly the kind of thing a careless schedule extinguishes early and irreversibly. So stage-wise scheduling is really damage control against a known failure mode: protect the high-entropy, many-valid-answers capability by not letting the convergent, one-right-answer pressure run first.
A useful counterpoint sits in the corpus too: maybe the deeper fix isn't scheduling within one model but *separation*. Splitting a decomposer from a solver, or wrapping LLM calls inside explicit algorithms that show each step only what it needs, prevents planning and execution from interfering at all Does separating planning from execution improve reasoning accuracy? Can algorithms control LLM reasoning better than LLMs alone?. And it's worth knowing what instruction tuning actually transfers is the output *format* distribution more than task understanding Does instruction tuning teach task understanding or output format?—which reframes the whole constraint-vs-creative conflict as a fight over which output distribution a model gets locked into. Scheduling resolves the conflict by controlling that lock-in: establish the strict distribution first, then widen it, rather than letting the two collide and the narrow one win by default.
Sources 7 notes
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Running Supervised RL first to establish reasoning foundations, then RLVR to refine against verifiable rewards, substantially outperforms both methods in isolation. The imitation phase makes outcome rewards informative by creating reasonable rollouts the RL phase can then sharpen.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Modular architectures with separate decomposer and solver models outperform monolithic LLMs, with decomposition ability transferring across domains while solving ability does not. The separation prevents planning-execution interference and produces more generalizable skills.
LLM Programs embed LLMs within explicit algorithms that manage control flow and state, presenting only step-specific context to each LLM call. This information hiding addresses capability and context window limits while treating complex reasoning as modular, debuggable sub-tasks.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.