Can reasoning be learned during pretraining rather than after?
Does building iterative computation into the pretraining phase itself allow language models to develop reasoning before post-hoc fine-tuning? And if so, does latent reasoning align better with outputs than explicit chain-of-thought?
Modern LLMs learn to "think" mainly through explicit text generation (CoT), which defers reasoning to post-training and under-leverages pretraining data. Ouro takes the opposite path: a family of pretrained Looped Language Models (LoopLM) that build reasoning into the pretraining phase through iterative computation in latent space, an entropy-regularized objective for learned depth allocation, and scaling to 7.7T tokens. The headline efficiency is striking — 1.4B and 2.6B Ouro models match up to 12B standard transformers (a 2–3× efficiency gain).
Two findings make this more than a parameter-efficiency trick. First, controlled experiments show the advantage stems not from increased knowledge capacity but from superior knowledge manipulation — the same facts, used better. Second, LoopLM's intermediate predictors are strongly aligned with the final predictor, so its latent reasoning traces are more faithful to the final answer than explicit CoT — a safety-relevant property, since articulated reasoning that diverges from the answer is exactly the failure mode CoT-monitoring fears.
This is the pretraining-native member of the recurrence cluster. Where How do looped language models actually improve reasoning in depth? explains the mechanism and Can tiny recursive networks outperform massive language models? shows it at tiny scale post-hoc, Ouro shows the same looping pays off when baked into pretraining — and reframes the faithfulness debate, since the latent trace is structurally tied to the output.
Inquiring lines that read this note 10
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does reinforcement learning teach reasoning or just when to reason? How do training priors constrain what context information can override? How do training data properties shape reasoning capability development? Do base models contain latent reasoning that training can unlock? Why do continual learning scenarios trigger catastrophic forgetting and interference? What capability tradeoffs emerge when scaling model reasoning abilities? Do language models learn genuine linguistic structure or just surface patterns? How do neural networks separate factual knowledge from reasoning abilities?Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How do looped language models actually improve reasoning in depth?
Mechanistic analysis investigates whether looping transformer layers creates genuinely new computation or reuses existing inferential stages. Understanding this distinction clarifies why recurrent depth can match standard scaling.
the mechanism; Ouro applies it during pretraining at scale
-
Can tiny recursive networks outperform massive language models?
Can a small network that recursively refines its reasoning on a latent state match or beat billion-parameter LLMs on hard reasoning puzzles? This challenges assumptions about scale and hierarchy in AI reasoning.
recursion at tiny scale; Ouro at pretraining scale
-
Why is predicting latents more sample-efficient than tokens?
Explores whether learning from a network's own abstract representations requires far fewer training samples than learning from raw tokens, and what mechanism drives this efficiency gap.
both argue manipulating latents beats token-level work
-
Can models reason without generating visible thinking steps?
Do machine reasoning systems actually require verbalized chains of thought, or can they solve complex problems through hidden computation? This challenges how we measure and understand reasoning.
LoopLM's faithful latent traces are evidence that verbalized CoT is not the only reasoning substrate
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Hierarchical Reasoning Model
- Reasoning to Learn from Latent Thoughts
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Scaling Latent Reasoning via Looped Language Models
- A Mechanistic Analysis of Looped Reasoning Language Models
- Training Large Language Models to Reason in a Continuous Latent Space
- Implicit Chain of Thought Reasoning via Knowledge Distillation
Original note title
looped language models build reasoning into pretraining via iterative latent computation — efficiency comes from knowledge manipulation not capacity