SYNTHESIS NOTE
Does RL training follow a predictable two-phase learning sequence?
This explores whether reinforcement learning exhibits consistent phases where basic execution skills must consolidate before strategic reasoning emerges. Understanding this sequence could reveal bottlenecks in scaling reasoning capabilities.
Across eight text-only and vision-language models, RL training reveals a consistently two-phase dynamic. In the first phase, the learning bottleneck is procedural correctness — a single calculation error invalidates an entire solution, creating powerful gradient signal that compels mastery of low-level execution tokens (arithmetic, variable substitution, formula application). In the second phase, the bottleneck shifts to strategic planning — exploring and mastering high-level planning tokens (deduction like "we can use the fact that," branching like "let's try a different approach," backtracing like "but the problem mentions that").
The phases are not mutually exclusive. Procedural refinement continues throughout training. But the primary driver of marginal performance gains shifts to strategic planning. This is why the "aha moment" phenomenon appears when it does — it represents the discovery and internalization of high-level reasoning strategies, which only becomes the active learning frontier after procedural skills are consolidated.
The entropy dynamics tell the same story. Planning tokens show increasing strategic diversification over training — the model explores new ways to combine established skills. Execution tokens show stable conditional entropy — once arithmetic is mastered, there's little incentive to find diverse ways to perform it. The performance improvement comes from discovering new combinations of established skills, which is the core function of planning.
This insight exposes a core inefficiency in algorithms like GRPO that apply optimization pressure uniformly across all tokens. If the learning frontier is in planning tokens but gradient signal is diluted across execution tokens, optimization is wasteful. HICRA addresses this by concentrating optimization on planning tokens, achieving significant performance gains.
The connection to existing insights is illuminating. Since Which sentences actually steer a reasoning trace?, HICRA's planning tokens are likely the same phenomenon identified from a mechanistic perspective. The two-phase dynamic also explains why Do reasoning cycles in hidden states reveal aha moments? — the graph structure reflects the transition from procedural execution (local structure) to strategic planning (global topology).
Inquiring lines that read this note 120
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can AI agents autonomously learn and transfer skills across tasks? Can self-supervised signals enable process supervision without human annotation? How do training priors constrain what context information can override? Why do LLM chatbots fail as independent therapeutic agents? What constrains reinforcement learning's ability to expand model reasoning?- What behavioral changes occur during reward learning training?
- How do reward signals in RLVR interact with pretraining biases?
- What distinguishes high-signal prompts from low-signal ones in RL training?
- How does entropy collapse in reinforcement learning differ from entropy maintenance in graph reasoning?
- Does policy entropy collapse represent the main bottleneck in reasoning-focused RL scaling?
- Does policy entropy collapse limit how many iterations of reasoning training work?
- How does policy entropy during training affect search discipline during inference?
- Why does policy entropy collapse predict sigmoid saturation points?
- What happens to model reasoning when policy entropy collapses during RL?
- How does representational convergence differ from policy entropy collapse in iterative training?
- How do high-entropy tokens concentrate reinforcement learning's effect?
- How does on-policy entropy recognition differ from training-time entropy collapse?
- Why does policy entropy collapse when scaling RL for reasoning?
- Can entropy regularization or critique models prevent search strategy collapse during RL training?
- What causes policy entropy collapse in reasoning-focused reinforcement learning?
- How does policy entropy collapse constrain zero RL scaling for reasoning?
- Does stable entropy in policy training actually guarantee stable reasoning behavior?
- Why must procedural skills consolidate before strategic reasoning can develop?
- What makes bilevel metacognition architectural rather than emergent in current systems?
- What structural differences emerge between early generic skills and later meta-strategy skills?
- Why does decoupling planning from execution improve over sequential interleaving?
- How does stage-wise training scheduling resolve conflicts between constraint-following and creative tasks?
- How does early commitment in reasoning differ from early exploitation in planning?
- What cascading bottlenecks appear when skill routing is decomposed into stages?
- How do training objectives shape what a world model actually learns?
- How do developmental curriculums emerge from learning progress signals?
- How does temporal anchoring maintain the learning signal in self-rewarding loops?
- What makes Effective Rank Acceleration a stable training signal for dual-channel incentives?
- What separates bootstrapping gains from sustained self-improvement gains?
- How does trajectory burstiness compare to other structural properties that shape emergent capabilities?
- Why does imitation learning alone plateau without outcome-based refinement?
- What training interventions could close the perception-action gap?
- Does the productive difficulty band ever stabilize during training?
- What makes exploration a verifiable and measurable training objective?
- What distinguishes surface mechanisms from the training regimes that produce them?
- How does scaffolding unstable mechanics improve reinforcement learning for search?
- Does task ordering affect multi-task reinforcement learning outcomes?
- How do complete multi-turn trajectories differ from isolated task examples?
- How should multi-objective post-training balance competing behavioral goals?
- How should guidance levels adapt as the model's capability boundary shifts?
- What capabilities actually require massive scale versus specialized training regimes?
- How does model scale affect anticipatory behavior in structured training?
- What breaks when you apply reinforcement learning after supervised fine-tuning?
- Why does early experience provide better warm-starts for downstream reinforcement learning?
- Can in-context learning replicate the timing effects that RL teaches models?
- Can meta-reinforcement learning explain why this bias pattern emerges rationally?
- How do residual connections and layer norm stabilize training in deep RL?
- Which recipe choices determine the asymptotic ceiling in RL training?
- How does behavior cloning reduce complexity before RL training in rerankers?
- Why do models follow a two-phase pattern of procedural then strategic learning?
- Does format-based pretraining determine how models respond to reinforcement learning?
- Can out-of-distribution tests expose memorization in reinforcement learning fine-tuned models?
- How does post-training shift models from passive prediction to on-policy action?
- What scaling properties emerge from RL training dynamics beyond verification?
- Why do single-turn RL methods fail to generalize to multi-turn tasks?
- What training duration is actually needed for RL to expand capabilities?
- What capacity threshold determines whether RL teaches activation versus shortcut learning?
- Does RL training redirect self-doubt into productive gap analysis?
- How does pretraining determine what RL can later teach a model?
- How does imitation pretraining followed by RL exploration compare to either method alone?
- Can energy minimization replace reasoning-specific reinforcement learning for system 2 thinking?
- Can RL teach when to use reasoning versus when to respond directly?
- Does reinforcement learning learn optimal per-turn reasoning discipline?
- How does reinforcement learning differ from chain-of-thought distillation?
- Does RL refine existing knowledge or discover entirely new capabilities?
- How does RL compress reasoning path diversity during training?
- What limits RL's ability to scale for reasoning at training time?
- Can RL training teach models when to activate reasoning versus when to skip it?
- Why do high entropy tokens carry most of the learning signal in RL?
- What distinguishes RL that creates new capabilities from RL that merely teaches timing?
- Can models learn both what and how to study through reinforcement learning?
- Does RL training actually restore the critical thinking that reasoning models lose?
- How does reinforcement learning on outcomes reinforce template-matching rather than computation?
- Why does prolonged RL discover strategies absent from any base model sample?
- Why does RL behavior differ between standard reasoning tasks and complex planning domains?
- Does reinforcement learning teach models how to reason or when to reason?
- Does RL primarily teach when to use reasoning or how to reason?
- What does RL post-training actually teach reasoning systems?
- When does reinforcement learning actually produce true reasoning gains in models?
- Does targeting the edge of competence during RL pretraining unlock true reasoning gains?
- Can reinforcement learning add new capabilities or only remove inaccurate knowledge?
- Do emergent abilities result from genuine new capabilities or implicit in-context learning?
- How does policy initialization with sub-policies enable emergent thinking?
- Does RL training activate latent meta-learning capacity or create it from scratch?
- How does dual-rate learning separate episodic and procedural memory in neural networks?
- Does grokking in modular arithmetic follow the same three-phase learning trajectory?
- How do complementary learning systems explain the need for fast and slow consolidation?
- Can training order and structure shape what networks retain and learn?
- How do learning dynamics on one example shift predictions on other responses?
- Do depth thresholds correspond to transitions between procedural and strategic learning?
- Do thought anchors correspond mechanistically to planning tokens in RL?
- How do evaluative versus directive signals differ in next-state training?
- Can architectural changes like decoupling intent understanding help overcome next-turn reward limitations?
- Why do next-turn reward objectives fail to encourage multi-turn goal progress?
- How does next-turn reward optimization contribute to agent passivity?
- Can early experience replace external rewards as a learning signal?
- Can reinforcement learning fix the reasoning gaps that supervised fine-tuning misses?
- What makes supervised fine-tuning worsen RL exploration later?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Does RL training follow a predictable two-phase …
Which sentences actually steer a reasoning trace?
Do reasoning cycles in hidden states reveal aha mo…
Does policy entropy collapse limit reasoning perfo…
Does RL teach reasoning or just when to use it?
What happens inside models when they suddenly gene…
Can language modeling close the knowing-doing gap …
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Which sentences actually steer a reasoning trace?
Can we identify which sentences in a reasoning trace have outsized influence on the final answer? Three independent methods converge on a surprising answer about planning and backtracking.
converges: planning tokens in HICRA likely correspond to thought anchors
-
Do reasoning cycles in hidden states reveal aha moments?
What if the internal loops in model reasoning—visible in hidden-state topology—correspond to the reconsidering moments that happen during reasoning? This note explores whether graph cyclicity captures a mechanistic signature of insight.
extends: the two-phase dynamic explains how graph topology evolves during training
-
Does policy entropy collapse limit reasoning performance in RL?
As reinforcement learning models become more confident in their policy choices, entropy drops and performance plateaus. Can we identify and counteract this bottleneck to sustain scaling?
reframes: entropy collapse may be acceptable for execution tokens but catastrophic for planning tokens
-
Does RL teach reasoning or just when to use it?
Does reinforcement learning in thinking models actually create new reasoning abilities, or does it simply teach existing capabilities when to activate? This matters for understanding where reasoning truly emerges.
deepens: the "when" is specifically about planning tokens; execution tokens are "how"
-
What happens inside models when they suddenly generalize?
Grokking appears as an abrupt shift from memorization to generalization. But is the underlying process truly discontinuous, or does mechanistic analysis reveal continuous phases we can measure and predict?
analogous phased development: grokking's memorization-then-circuit-formation parallels the procedural-then-strategic progression; both show that generalization requires passing through a consolidation phase before higher-order structure emerges
-
Can language modeling close the knowing-doing gap in AI?
Current LLMs reason well but act poorly in interactive tasks, while RL agents act well but cannot explain themselves. Can reformulating decision-making as language modeling with environmental feedback bridge this fundamental split?
TiG operates on the same procedural-vs-strategic axis HICRA identifies, but at the architectural level: language-as-policy refined by RL preserves declarative reasoning while building procedural competence — HICRA's two-phase dynamic predicts the order TiG observes during training
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Emergent Hierarchical Reasoning In LLMs Through Reinforcement Learning
- Reinforcement Learning with Rubric Anchors
- RAGEN-2: Reasoning Collapse in Agentic RL
- The Art of Scaling Reinforcement Learning Compute for LLMs
- From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
- Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess
- Teaching Large Language Models to Reason with Reinforcement Learning
Original note title
rl training exhibits a two-phase dynamic where procedural consolidation precedes strategic planning exploration