Making AI agents train on harder, more varied, more realistic tasks sounds great — but each fix seems to create a new problem.
How do complexity, diversity, and real-world fidelity interact in agent training?
This explores how three dials of agent training (how hard the tasks are, how varied the training experience is, and how closely training matches the real world) push against or support each other.
This explores how three dials of agent training (task complexity, diversity, and real-world fidelity) push against or support each other. No single note in the collection varies all three at once, but read together the notes show that each dial fixes one problem and tends to cause another.
Start with fidelity versus diversity. Expert demonstrations look like the most faithful training you can get, since they show how real practitioners work. But agents trained only on static demonstrations never interact with an environment, so they can't learn from their own failures, and their competence is capped by Can agents learn beyond what their training data shows?. Letting the agent act and learn from outcomes removes that cap, but RL then narrows what the agent tries. Does reinforcement learning squeeze exploration diversity in search agents? finds that search agents lose behavioral diversity the same way reasoning models do, while SFT on diverse demonstrations keeps exploration broad. A likely mechanism is in Why do language models collapse into generic templates?. When a prompt's outcomes barely differ across attempts, the learning signal is weak and the policy drifts toward generic, input-agnostic templates. Choosing prompts where outcomes actually vary restores performance. That is a way of tuning task difficulty to keep diversity alive.
Fidelity also doesn't have to mean the real world itself. Can language models learn to simulate agent environments? reports that a language model trained to predict what happens next, on over 10 million trajectories, beat training in real environments on three benchmarks and transferred across seven domains. Similarly, Can agents learn to work reliably through environment and coordination scaling? gets strong results on complex professional tasks, even with smaller models, by expanding executable environments and treating state, recovery, and delivery as trainable behaviors. Here the real world is the bottleneck on variety. A simulator or a large family of environments can supply far more scenarios than any curator could list. The notes don't say how faithful a simulation has to be before the benefits stop transferring, and that is the open question in this corner.
Complexity, meanwhile, seems to be less about hard single steps than about long loops of trying, failing, and revising. What predicts success in ultra-long-horizon agent tasks? found across 17 frontier models that the best predictor of success on very long tasks was persistence in benchmark-edit-retry cycles, not how good the first attempt was. Two notes suggest that loop can be learned without touching model weights. Can agents learn from failure without updating their weights? shows that clear success-or-failure feedback lets agents write useful self-diagnoses, and Can agents learn continuously from experience without updating weights? reaches 87.88% on GAIA validation purely through memory operations. Where does agent reliability actually come from? generalizes the point: reliability comes from moving memory, skills, and protocols into the surrounding system. And Can small language models handle most agent tasks? suggests most agent work is repetitive and well-defined. So training for complexity may mean training the loop and the coordination, not making every step harder.
Diversity has its own catch. Can multiple agents stay diverse during training together? keeps diversity alive by giving generation and critic agents different role-specific data, avoiding the overfitting collapse of single-agent finetuning. But Does cognitive diversity alone improve multi-agent ideation quality? warns that diversity without real expertise underperforms even one competent agent. So variety only helps when it sits on top of grounded competence.
The picture that emerges: realism without variety caps the agent at what curators imagined, and hard tasks without variety collapse into templates. Variety without expertise adds noise, and simulated scale helps only as far as it stays useful. The dials have to be tuned together, and the collection has no study that does that yet.
Sources 12 notes
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Qwen-AgentWorld demonstrates that native language world models trained via next-state prediction on 10M+ trajectories outperform real-environment training on three benchmarks and transfer across seven domains, positioning next-state prediction as a foundation objective for agents.
Apodex 1.1 demonstrates that agents trained via expanded executable environments and learned coordination behavior reach leading performance on complex professional tasks, even with smaller models. The approach achieves verifiable progress by treating state, recovery, and delivery as trainable behaviors rather than architectural requirements.
Show all 12 sources
Across 17 frontier models on 36 expert-curated optimization tasks, repeated benchmark-edit-incorporate cycles within a wall-clock budget proved the dominant success predictor. Most models terminated early or burned budget unproductively; Claude Opus 4.6 stood out as persistent.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
SLMs handle the repetitive, well-defined language tasks that constitute most agent work at 10–30× lower cost than LLMs, making heterogeneous architectures (SLMs by default, LLMs selective) the economically rational design pattern.
Training generation and critic agents on distinct role-dependent data prevents the overfitting collapse that limits single-agent finetuning to one productive iteration. Removing critics or summarization degrades performance, confirming both components are critical.
Multi-agent teams substantially outperform solo ideation, but only when members possess genuine senior knowledge. Diverse teams without expertise underperform even a single competent agent, because cognitive stimulation without expertise triggers process losses instead of insight.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Towards a Science of Scaling Agent Systems
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Agent Learning via Early Experience
- Artifacts as Memory Beyond the Agent Boundary
- SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Survey on Evaluation of LLM-based Agents