Can an AI get smarter just by keeping a diary of its past attempts, with no retraining at all?
Can agents learn from their own experience without fine-tuning through episodic memory?
This explores whether an agent can get better by writing down what happened in past attempts and consulting those records later, leaving the underlying model's weights untouched.
This explores whether an agent can improve by recording its past attempts and consulting them later, without any fine-tuning of the model. The corpus says yes, and the evidence is strong. AgentFly treats learning as something done to memory rather than to weights. It keeps case, subtask and tool memories, and it reached 87.88% on the GAIA validation set with the LLM's parameters never modified (Can agents learn continuously from experience without updating weights?). Reflexion shows the same idea at its simplest. After an attempt fails, the agent writes a short verbal diagnosis of what went wrong and reads it on the next try (Can agents learn from failure without updating their weights?). It works best when the feedback is a clear success or failure, because that leaves the agent no room to rationalize a bad outcome. Keeping the reflections uncompressed also keeps them usable.
The interesting part is that what you store matters more than the fact that you store something. SkillRL treats successes and failures differently. Wins are kept as concrete demonstrations, and losses are boiled down into abstract lessons. That beat uniform treatment while using much less context (Should successful and failed episodes be processed differently?). DeepAgent lets the agent fold its own history into separate episodic, working and tool memories. This cuts token cost and gives it a natural moment to stop and rethink its strategy, and the autonomy plus the structure avoid the degradation of sloppier compression (Can agents compress their own memory without losing critical details?). VOYAGER stores learned skills as runnable code in a searchable library and builds harder skills out of simpler ones. That avoids the catastrophic forgetting that weight updates risk, since old skills sit untouched on the shelf (Can agents learn new skills without forgetting old ones?).
Designing the memory turns out to be a real engineering problem, and it can even be passed between agents. Dumping a large agent's raw experience on a small one gives only marginal gains. Reorganizing the same trajectories into workflow, subtask and function memory, again with no retraining, lifted 4B to 8B models by up to 27.2 percentage points, sometimes close to the teacher (Can small agents learn from larger agents' memories?). Retrieval design matters too. Letting the agent reason while it walks a memory graph, instead of retrieving and then reasoning, gained up to 23% on reasoning tasks (Can agents reconstruct memory on demand instead of retrieving it?). Keeping episodic events separate from distilled semantic knowledge is what lets M3-Agent pick up user preferences just from watching (Can agents learn preferences by watching rather than asking?).
This matters because the alternatives have real ceilings. Agents trained only on expert demonstrations are capped by what the dataset's curators imagined, since they never meet their own failures (Can agents learn beyond what their training data shows?). Memory lets an agent collect those failures after deployment. Its weight-updating cousin is worth knowing about. In early experience training, the agent treats the consequences of its own actions as supervision, matching expert-dependent baselines with half the data. It still changes weights, though, and its main payoff is a better starting point for later RL (Can agents learn from their own actions without external rewards?). Even agents with no memory module can end up using their surroundings as one. Mathematical analysis shows RL agents leave and read environmental traces that cut the history they need to carry (Do RL agents accidentally use environments as memory?). Memory-based learning may be closer to something agents already do than to an add-on.
Sources 11 notes
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
DeepAgent's autonomous memory folding consolidates interaction history into episodic, working, and tool memory schemas. This reduces token overhead while letting agents pause to reconsider strategies—the autonomy and structure together avoid degradation that plagues poorly designed consolidation.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
Show all 11 sources
Naive memory transfer from large to small agents yields only marginal gains, but restructuring the teacher's trajectories into workflow, subtask, and function memory—without retraining—closes the gap significantly. Students of 4B to 8B parameters achieved up to 27.2 percentage point gains and sometimes approached teacher-level performance.
MRAgent achieves up to 23% gains on reasoning tasks by reconstructing memory through active graph traversal that prunes paths based on accumulated evidence, while reducing token and runtime cost compared to fixed-retrieval pipelines.
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
Research across eight environments shows that agents can use future states from their own actions as supervision without external rewards, matching expert-dependent baselines with half the data and providing superior warm-starts for subsequent RL training.
Mathematical proof shows that environmental artifacts reduce information needed to represent history in RL agents. Path-following agents naturally develop memory-like behavior through standard reward optimization, satisfying situated cognition criteria without explicit memory objectives.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
- Artifacts as Memory Beyond the Agent Boundary
- Rethinking Memory as Continuously Evolving Connectivity