When a new AI is built from an older one, how much is truly new — and how much was already hiding inside?
How do labs actually train next-generation models from previous ones?
This explores how a new model gets built on top of what an earlier one left behind (the base it starts from, the outputs it learns from, and the loops that connect them), not any one lab's private recipe.
This explores how a new model gets built on top of what an earlier one left behind (the base it starts from, the outputs it learns from, and the loops that connect them), not any one lab's private recipe. The corpus doesn't document a full lab pipeline. It does hold a lot of evidence about what each step does, and some of it is surprising.
The first surprise is how little of the 'new' capability is new. Base models already carry reasoning ability in latent form, and RL post-training mostly teaches the model when to use it. Hybrid models recover 91% of the performance gains just by routing which tokens get the reasoning treatment Does RL post-training create reasoning or just deploy it?. That isn't the whole story. On standard reasoning tasks RL mostly activates what's already there, but on multi-step planning it produces strategies the base model couldn't reach even with heavy sampling Does reinforcement learning create new reasoning abilities or activate existing ones?. So a next-generation model is sometimes an earlier one with its abilities uncovered and sometimes one that is genuinely extended, depending on the task.
The second surprise is what gets handed down when reasoning traces become training data. It's the shape, not the facts. Models tolerate 50% corrupted numbers in reasoning demonstrations with only a 3.2% accuracy loss, but shuffling the order of steps costs 13.3% What do models actually learn from chain-of-thought training?. Training on the messy version (failed attempts, backtracking, recovery) also teaches sturdier reasoning than showing only clean shortcut solutions Can models learn better by training on messy exploration paths?. The handoff can still go wrong. Nearly impossible problems in RL training teach shortcuts like repeating answers and skipping computation, and those shortcuts contaminate abilities the model already had Do overly hard RLVR samples actually harm model capabilities?. Fine-tuned models also often keep leaning on memorized templates and drop sharply on small variations of a problem Do fine-tuned language models actually learn optimization procedures?.
The bigger risk is the loop itself. If you train on a previous generation's output and repeat, rare events and unusual patterns vanish first, and the loss is irreversible. Each generation compounds it, which makes real human data more valuable, not less Does training on AI-generated content permanently degrade model quality?. A related pressure shows up within a single model's lifetime. Methods that keep the model close to its base preserve its ability to learn the next task, while ones that drift far stall when the domain changes Does staying close to the base model preserve learning ability?. Every adaptation method also has a sweet spot and hidden costs, such as weaker reasoning faithfulness or less format flexibility How do domain training techniques actually reshape model behavior?.
Post-training also changes what the model is, not just what it knows. Post-trained models behave as if they recognize their outputs as their own future inputs, with 3-4x lower output entropy on-policy Do models recognize their own outputs as actions shaping future inputs?. Turning a language model into an agent takes more than retraining, because the data curation, memory and tool infrastructure, and safety evaluation around it decide whether its actions are grounded Can you turn an LLM into an agent by just fine-tuning?. One alternative to making the next model bigger is to loop the same layers repeatedly, which beats larger feedforward networks on reasoning tasks Can models learn by looping instead of growing larger?. What the corpus lacks is a direct account of how labs filter and mix a predecessor's outputs into the next training run. These notes explain the mechanics of the pieces, not the assembly line.
Sources 12 notes
Evidence shows base models already contain reasoning capability in latent form; RL training optimizes deployment timing rather than capability creation. Hybrid models recover 91% of performance gains by routing tokens only, and activation vectors for reasoning strategies pre-exist before any RL.
For standard reasoning tasks, RL activates latent abilities already present in base models. For complex planning requiring multi-step coordination, RL generates genuinely novel strategies inaccessible to base models even with extensive sampling.
Controlled ablations show models tolerate 50% corrupted numbers (3.2% accuracy loss) but fail under step shuffling (13.3% loss). What distills across reasoning demonstrations is logical architecture—how steps sequence and connect—not factual accuracy.
Research shows that training on messy trajectories—failed attempts, self-correction, and backtracking—teaches more robust reasoning than training only on shortcut solutions. This approach models o1-style deep reasoning as search internalization rather than solution memorization.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Show all 12 sources
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
Models trained on mixtures of real and AI-generated data progressively lose rare events and unusual patterns across VAEs, GMMs, and LLMs. Each generation compounds the loss, making genuine human data increasingly valuable.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
Research shows every adaptation method—from parameter-efficient tuning to knowledge graph curricula—has optimal conditions tied to specific domains. The key finding: visible benefits like performance gains often come with hidden degradation in reasoning faithfulness, capability transfer, and format flexibility.
Post-trained language models exhibit a measurable shift where they recognize their outputs become their own future inputs, closing an action-perception loop absent in pretraining. Evidence includes 3-4x lower output entropy on-policy and behavioral signatures of trajectory recognition.
Converting LLMs to action-capable systems requires four distinct stages: curating action-environment-user datasets, training for action grounding, integrating agent infrastructure with memory and tools, and rigorous safety evaluation. The surrounding system and harness determine whether actions are grounded or hallucinated.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- Eliciting Reasoning in Language Models with Cognitive Tools
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?