AI agents can get smarter just by using them — through feedback, memory, and self-correction, no retraining required.
How do learning loops and user feedback improve agent performance over time?
This explores how AI agents get better with use: by learning from their own successes and failures, from environment signals, and from people steering them, rather than staying fixed after training.
This explores how AI agents get better with use: by learning from their own successes and failures, from environment signals, and from people steering them, rather than staying fixed after training. The corpus points to something surprising here. Most real improvement over time doesn't come from retraining the model. One survey splits self-improving agents into two loops: a slow one that updates the model's weights, and a fast one that updates everything around the model, like its prompts, memory and tools. Recent progress is concentrated in the fast loop, because those changes are cheap and can be undone Do self-improving agents really split into two distinct loops?.
The fast loop shows up in several forms. Reflexion has an agent write a short note to itself after each attempt about what went wrong, then reread that note on the next try. This works best when the feedback is a plain pass or fail, because a clear verdict leaves the agent no room to talk itself into thinking a bad attempt was fine Can agents learn from failure without updating their weights?. AgentFly goes further and treats memory itself as the learning mechanism. It stores past cases, subtasks and tool uses, and reached strong benchmark results without changing a single weight Can agents learn continuously from experience without updating weights?. VOYAGER saves working skills as reusable code and builds harder skills out of simpler ones. That sidesteps the 'catastrophic forgetting' that happens when retraining overwrites old abilities Can agents learn new skills without forgetting old ones?.
One detail stands out: successes and failures should be stored differently. SkillRL keeps successful runs as concrete examples to copy and boils failures down into general lessons. This beats treating every episode the same way, and it uses less context too Should successful and failed episodes be processed differently?. It's roughly how experienced people learn: you remember exactly how you did the thing that worked, and you remember the gist of why the other thing didn't.
Why is learning from feedback so important? An agent trained only on expert demonstrations can never do better than what the people who built the dataset thought of, because it never fails and never learns from failing Can agents learn beyond what their training data shows?. There's a related twist. Post-training can make models better at easy tasks while wiping out rare solutions to hard ones. Base models given enough attempts eventually cover more ground Do base models find more solutions than post-trained ones?. On long tasks, the best predictor of success was simply persistence: whether the agent kept running the cycle of test, edit and incorporate feedback, rather than how good its first attempt was. Most models quit too early What predicts success in ultra-long-horizon agent tasks?.
The loop can also improve the scaffolding itself, meaning the 'harness' of memory, skills and interaction rules that wraps the model Where does agent reliability actually come from?. Automated research loops that tuned harnesses across many environments cut token use by almost half without losing performance Can agent harnesses be automatically optimized across many environments?. Harness tuning on a frozen model raised scores across several models, and the same recipe carried over to newer models Can execution harnesses lift model performance without retuning weights?. Human feedback is the least developed part of the corpus. The clearest example is Cocoa, where users revise the agent's plan step by step while it runs and get more control without extra effort Can users steer AI agents better by revising plans mid-execution?. That is steering in the moment, though, not learning that builds up over time. How user feedback turns into lasting agent improvement is still a gap in the collection.
Sources 12 notes
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
Show all 12 sources
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
Across 14 model pairs and three agentic benchmarks, base models equipped with only relaxed system prompts eventually surpass post-trained counterparts in pass@K coverage as rollout budget grows. Post-training bimodalizes task outcomes, sharpening performance on easy cases while eliminating rare-but-reachable solutions.
Across 17 frontier models on 36 expert-curated optimization tasks, repeated benchmark-edit-incorporate cycles within a wall-clock budget proved the dominant success predictor. Most models terminated early or burned budget unproductively; Claude Opus 4.6 stood out as persistent.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Cocoa's system, which lets users assign plan steps to themselves or an agent and revise plans mid-execution, afforded greater steerability than chat-baseline interaction in a lab study (n=16) while maintaining ease of use, with field users valuing explicit work delegation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Useful Memories Become Faulty When Continuously Updated by LLMs
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Sharpening Tax in Post-Training
- Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory