When an AI agent acts and sees what happens next, why is that outcome a trustworthy way for it to learn?
What makes next-state signals from agent trajectories a reliable learning source?
This explores why the consequence of an agent's own action (the tool output, error, user reply or screen change that follows it) works as a dependable training signal, and what the corpus says keeps it from being noise.
This explores why the consequence of an agent's own action works as a dependable training signal. The corpus's main answer is that the signal is grounded. Every action is followed by something the world actually did: a user reply, a tool output, an error, a GUI change. That consequence can train the policy directly, with no separate dataset, and it looks the same across conversations, terminal tasks, software engineering and tool use Can agent deployment itself generate training signals automatically?. Compare expert demonstrations, which cap an agent at what the curators imagined and never show it its own failures Can agents learn beyond what their training data shows?. A next-state signal comes from the agent's real situation, not from someone's guess about it.
The signal also needs no external reward, and it comes from the agent's own behavior. Across eight environments, agents that used the future states produced by their own actions as supervision matched expert-dependent baselines with half the data. They also made better warm-starts for later RL training Can agents learn from their own actions without external rewards?. Post-trained models seem built to read the signal this way. They measurably recognize that their outputs become their own future inputs, with 3-4x lower output entropy on-policy. So the model treats the consequence as the result of what it did, not as text to continue Do models recognize their own outputs as actions shaping future inputs?.
The third reason is abundance and structure. Qwen-AgentWorld trained a language model on next-state prediction across 10M+ trajectories. It outperformed real-environment training on three benchmarks and transferred across seven domains, so there is enough signal for a model to learn how an environment behaves Can language models learn to simulate agent environments?. Trajectories also carry structure that turns sparse outcomes into step-level feedback. Tree topology, expert-aligned actions and tool-call positions each stand in for hand-annotated process rewards Can trajectory structure replace hand-annotated process rewards?. A deployed routing harness already records this material as a side effect, so trajectories and outcomes can be turned into labeled examples Can a routing harness generate its own training data automatically?. One signal is rarely enough alone. In-context learning of sequential decisions needs full or partial trajectories from the same environment level, not isolated examples Why do trajectories matter more than individual examples for in-context learning?. A next state is informative because of the run of actions around it.
What the corpus stresses most is that reliability comes from the loop, not from any single signal. On 36 ultra-long optimization tasks, repeated benchmark-edit-incorporate cycles predicted success better than a model's first attempt did What predicts success in ultra-long-horizon agent tasks?. VOYAGER uses environmental feedback to refine stored skills Can agents learn new skills without forgetting old ones?. AgentFly reached 87.88% on GAIA validation by using outcomes to update episodic memory alone, with no weight changes Can agents learn continuously from experience without updating weights?.
The corpus is thin on what happens when a signal misleads, for example an ambiguous user reply or an error that isn't the agent's fault. The closest thing to a filter is asymmetry. SkillRL keeps successful episodes as concrete demonstrations and distills failures into abstract lessons, which avoids the degradation of treating all consequences alike Should successful and failed episodes be processed differently?.
Sources 12 notes
Every agent action produces a next-state signal (user reply, tool output, error, GUI change) that can train the policy directly. This universal signal source eliminates the need for separate training datasets across conversations, terminal tasks, SWE, and tool use.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
Research across eight environments shows that agents can use future states from their own actions as supervision without external rewards, matching expert-dependent baselines with half the data and providing superior warm-starts for subsequent RL training.
Post-trained language models exhibit a measurable shift where they recognize their outputs become their own future inputs, closing an action-perception loop absent in pretraining. Evidence includes 3-4x lower output entropy on-policy and behavioral signatures of trajectory recognition.
Qwen-AgentWorld demonstrates that native language world models trained via next-state prediction on 10M+ trajectories outperform real-environment training on three benchmarks and transfer across seven domains, positioning next-state prediction as a foundation objective for agents.
Show all 12 sources
Tree-GRPO, Supervised RL, and ToolPO each convert sparse outcome rewards into dense step signals by exploiting different structural features—tree topology, expert-aligned actions, and tool-call positions—eliminating the need for annotated process reward models.
A deployed routing system records execution trajectories, capability demand estimates, and outcome data that can be converted into labeled training examples for fine-tuning and distillation, turning the harness into both a serving component and a difficulty labeler.
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.
Across 17 frontier models on 36 expert-curated optimization tasks, repeated benchmark-edit-incorporate cycles within a wall-clock budget proved the dominant success predictor. Most models terminated early or burned budget unproductively; Claude Opus 4.6 stood out as persistent.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Agent Learning via Early Experience
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- OpenClaw-RL: Train Any Agent Simply by Talking
- Demystifying Agent Skills: Why They Work-Until They Don't
- Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
- Useful Memories Become Faulty When Continuously Updated by LLMs