Why does handing an AI a short playbook of what worked beat handing it a full replay of past attempts?
Why does distilling reasoning strategies outperform raw trajectory memory?
This explores why storing compact, reusable strategies distilled from past reasoning can beat replaying whole past attempts (full step-by-step trajectories) when an AI faces a new problem.
This explores why storing compact, reusable strategies distilled from past reasoning can beat replaying whole past attempts when an AI faces a new problem. The corpus has no head-to-head test of strategy memory against trajectory memory, so what follows is assembled from adjacent findings. Those findings agree on one point: a raw trajectory carries the strategy plus everything specific to the problem it came from.
The closest direct evidence is on the abstraction side. RLAD trains one model to write short abstractions of how to approach a problem and another to solve using them. At large compute budgets, spending effort on diverse abstractions beats sampling more parallel solutions, and the abstractions force breadth-first exploration that avoids the depth-only failure of underthinking (Can abstractions guide exploration better than depth alone?). A strategy tells the model which kinds of approach to try. A raw trajectory shows one path that already happened to work, which can anchor the model to that path.
Raw trajectories are also tightly tied to the situation they came from. In-context learning for sequential decision-making works only when the trajectories in context come from the same environment level, and isolated examples don't do the job (Why do trajectories matter more than individual examples for in-context learning?). So replay pays off when the new problem closely matches the stored one. Chain-of-thought shows the same weakness: it degrades predictably under shifts in task, length, and format, and models produce fluent reasoning that imitates the form without valid logic (Does chain-of-thought reasoning actually generalize beyond training data?). A strategy is a compressed statement of what transferred across situations. A raw trace is the form of reasoning at its most distribution-specific.
Raw traces are also mostly bulk. One steering vector cut chain-of-thought length by 67% with accuracy maintained, which suggests much of a long trace is not doing the work (Can we steer reasoning toward brevity without retraining?). Accuracy also follows an inverted U with length, so more tokens eventually hurt (Why does chain of thought accuracy eventually decline with length?). Stuffing whole trajectories into memory puts a lot of that bulk in front of the model.
Two results complicate the story, and they suggest where the advantage sits. Journey learning finds that training on messy trajectories, with failures, backtracking, and recovery, teaches more robust reasoning than training on clean shortcuts (Can models learn better by training on messy exploration paths?). That is training, though. It changes the model's weights over many examples, so the model can extract the pattern itself. Retrieval-time memory gives the model one trace to reuse as-is, so the extraction has to happen beforehand. Memory-amortized inference frames intelligence as structured reuse of prior inference paths rather than recomputation (Can cognition work by reusing memory instead of recomputing?). Read alongside the other notes, the payoff comes from how well the stored material is structured for reuse. Distilling strategies does that structuring up front, which is a plausible reason it wins. This is an inference from these notes and not a measured result.
Sources 7 notes
RLAD jointly trains abstraction and solution generators, showing that allocating test-time compute to diverse abstractions outperforms parallel solution sampling at large budgets. Abstractions create structured breadth-first exploration that prevents the underthinking failure mode of depth-only reasoning chains.
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.
DataAlchemy experiments show CoT fails systematically under distributional shifts in task, length, and format. Models produce fluent but logically inconsistent reasoning — imitating reasoning form without valid underlying logic.
Activation-Steered Compression extracts a single vector from 50 paired examples to reduce chain-of-thought length by 67% while maintaining accuracy and achieving 2.73x speedup. The method is training-free and generalizes across model sizes and domains.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Show all 7 sources
Research shows that training on messy trajectories—failed attempts, self-correction, and backtracking—teaches more robust reasoning than training only on shortcut solutions. This approach models o1-style deep reasoning as search internalization rather than solution memorization.
Memory-Amortized Inference proposes intelligence arises from structured reuse of prior inference paths over topological memory, inverting RL's reward-forward logic into cause-backward reconstruction. This duality explains energy efficiency and suggests memory trajectories form the substrate of adaptive thought.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Break the Chain: Large Language Models Can be Shortcut Reasoners
- RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- Teaching Large Language Models to Reason with Reinforcement Learning
- Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Generalization to New Sequential Decision Making Tasks with In-Context Learning