Why do trajectories matter more than individual examples for in-context learning?
Can language models learn new sequential decision-making tasks from context alone, and if so, what data properties make this possible? This explores why isolated state-action pairs fail where full trajectories succeed.
In-context learning for supervised tasks works by providing a few input-output examples. Naively applying this to sequential decision making (providing a few state-action pairs) fails to enable ICL of new tasks. The key finding: the context must contain full or partial trajectories from the same environment level as the query — not just isolated examples. This property is called trajectory burstiness.
Why the difference matters: In supervised learning, examples can be from different instances — the model learns the function mapping. In sequential decision making, the model must generalize from the same level/environment to handle the wide range of states it may encounter at deployment. A sparse set of state-action pairs doesn't cover the state space; full trajectories do.
Trajectory burstiness is the probability that a given input sequence contains at least two trajectories from the same level. When this property is present in pre-training data, the model acquires the capacity to learn new tasks from demonstrations at inference time without weight updates.
Additional factors that increase ICL performance:
- Larger model and dataset size
- More task diversity in pre-training
- Environment stochasticity (forces generalization over trajectory variation)
- Higher trajectory burstiness in pre-training data
Generalization scope demonstrated: Train/test tasks differ greatly — different states, actions, dynamics, and reward functions. The model generalizes from, e.g., platform games to maze navigation from a handful of expert demonstrations. This is substantially harder than prior work that generalizes across reward function variants of the same environment.
The implication for dataset construction: sequential decision-making ICL requires a data distribution property (trajectory burstiness) that standard language modeling data does not naturally contain. This is a data structural requirement, not just a scale requirement.
This connects to Does training data format shape reasoning strategy more than domain? — here the structural property is at the trajectory level rather than the reasoning step level, but the principle is the same: data structure determines capability.
Inquiring lines that read this note 59
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do language models truly understand their encoded knowledge or merely retrieve patterns? What training data structures and signals produce robust learning across distributions?- How does in-context learning trigger phase transitions in model behavior?
- Why does the order of training examples matter for what models learn?
- What makes a good in-context learning example for a given task?
- Why is in-context learning brittle to the order of examples presented?
- How does demonstration coverage in context examples determine operation generalization?
- Does task superposition explain how models learn from multiple in-context trajectories?
- Can a single model trained on two tasks predict untrained decision tasks?
- What makes some contexts learnable as rules versus requiring model retraining?
- Do text-space skills transfer learning across different frontier models?
- How do training objectives shape what a world model actually learns?
- What makes session-aware multi-turn tracking necessary for asynchronous training?
- Can trajectory quality filtering improve model training in noisy environments?
- How do complete multi-turn trajectories differ from isolated task examples?
- How does temporal anchoring maintain learning signals when preference gaps collapse?
- Can parallel trajectories reveal better decision branches without human labeling?
- Why do context-sensitive languages transfer better than regular or context-free languages?
- Can in-context learning substitute for domain-specific training altogether?
- How does training order affect knowledge acquisition in language models?
- Can language acquisition analogies mislead us about how models actually learn?
- Can in-context learning replicate the timing effects that RL teaches models?
- How does trajectory filtering handle noise when language models use code execution tools?
- Can in-context reinforcement learning match human sample efficiency on real problems?
- What role does sequence model in-context learning play in multi-agent cooperation?
- Can episodic memory alone enable learning without parameter updates?
- Does environment stochasticity force models to generalize better across trajectory variations?
- Why does delegation training help models that work alone?
- Why do trajectory-based skills fail to transfer across different environments and use cases?
- How does explicit exploratory prompting compare to fine-tuned reinforcement learning for in-context adaptation?
- Do few-shot examples improve in-context learning or add noise?
- How do retrieved memories differ from decision-context passages for prediction?
- What computational cost does trajectory-bursty inference impose on per-query context requirements?
- Should user context live in tokens or in learned model representations?
- Can in-context learning's advantage erode once interaction histories exceed the context window?
- What role does a model's representational structure play in learning?
- Does next-state prediction alone build mechanistic world models or just sophisticated interpolation?
- How do models understand rich context better than they can generate it?
- Why do foundation models fail at hidden state prediction despite sequence accuracy?
- Can graph topology represent successful trajectory clusters more effectively than skill libraries?
- What makes trajectory quality matter more than one-shot task success?
- What makes next-state signals from agent trajectories a reliable learning source?
- Can out-of-band observers bound unbounded action sequences efficiently?
- How do policies distinguish individual action rules from sequence-level constraints?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does training data format shape reasoning strategy more than domain?
What explains why models trained on multiple-choice data reason differently than those trained on free-form text? The research isolates format and domain effects to measure which one matters more.
trajectory burstiness is another case where data structure determines emergent capability
-
What do models actually learn from chain-of-thought training?
When models train on reasoning demonstrations, do they memorize content details or absorb reasoning structure? Testing with corrupted data reveals which aspects of CoT samples actually drive learning.
structural properties of training data drive learning; applies at both the reasoning trace and trajectory levels
-
Can we allocate inference compute based on prompt difficulty?
Does adjusting how much compute each prompt receives—rather than using a fixed budget—improve model performance? Could smarter allocation let smaller models compete with larger ones?
the context-length requirements for trajectory-bursty inference raise per-query compute costs
-
Can LLMs handle multiple tasks at once during inference?
Do language models maintain multiple distinct in-context learning tasks simultaneously in their internal representations, and if so, what prevents them from actually generating outputs for more than one task?
task superposition may be the representational mechanism enabling trajectory-bursty ICL: the model maintains multiple task interpretations from in-context trajectories simultaneously before committing to a single policy at generation time
-
Can transformers learn to solve new problems within episodes?
Explores whether transformer models can develop meta-learning abilities through RL training, enabling them to adapt to unseen environments by learning from within-episode experience alone, without updating weights.
ICRL is the RL-trained capability that trajectory burstiness enables: same-level trajectories create the meta-learning pressure during training that ICRL exploits at inference time for adaptation to unseen environments
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Generalization to New Sequential Decision Making Tasks with In-Context Learning
- Supervised Pretraining Can Learn In-Context Reinforcement Learning
- Teaching Large Language Models to Reason with Reinforcement Learning
- Schema-learning and rebinding as mechanisms of in-context learning and emergence
- STaR: Bootstrapping Reasoning With Reasoning
- Training a Generally Curious Agent
- In-Context Principle Learning from Mistakes
- BDH-CQ: In-Context Learning with Recurrent Latent Reasoning
Original note title
trajectory burstiness — same-level trajectories in context — is required for in-context learning of sequential decision-making across new tasks