Two AI agents can finish a task at the same success rate yet be totally different underneath — one reliable, one quietly broken.
How do trajectory quality and memory hygiene affect agent evaluation?
This explores why judging an AI agent only by whether it finished a task misses a lot, and what you learn by also looking at how it got there (its trajectory) and how carefully it manages what it remembers (its memory hygiene).
This explores why 'did the agent finish the task?' is too blunt a question, and what changes when you also look at the path the agent took and how it handles its memory. The corpus's main point is that two agents can post the same success rate and still be very different systems. One may get there efficiently and reliably. The other may stumble, burn budget, or carry hidden errors that would break it in real use. That is why researchers are calling for benchmarks that score trajectory, memory hygiene and verification cost alongside task success How should we measure agent system performance beyond task success?.
Start with trajectory. Across 17 frontier models on long optimization tasks, the best predictor of success wasn't how good a model's first attempt was. It was persistence: whether the model kept running the cycle of test, edit and incorporate feedback until time ran out. Most models stopped early or spent their budget unproductively What predicts success in ultra-long-horizon agent tasks?. A score taken from one snapshot would miss that completely. The other side of the problem is an agent that never stops. One paper argues that instructions in the prompt can't guarantee an agent will exit a loop, so stopping has to be enforced by a supervisor outside the agent, with hard timeouts Can prompt alignment alone guarantee agent termination in loops?. Put together, trajectory quality covers both failures: quitting too soon and never knowing when to stop.
Memory hygiene is where many failures hide. One study argues that agents in long, multi-turn workflows usually fail because they control their memory poorly, not because they lack knowledge. Replaying the whole conversation history, or retrieving loosely, lets small errors and drifting constraints pile up. The proposed fix is a small, structured record of 'committed' facts that is kept separate from scratch material the agent just looks things up in Can agents fail from weak memory control rather than missing knowledge?. To diagnose this, memory can be tested stage by stage (storage, extraction, retrieval, maintenance). That shows which stage actually broke, which an end-to-end score can't do How should we actually evaluate agent memory systems?. Good hygiene also means choosing what to keep. Agents that compress their own history into separate memories (what happened, what they're working on, how tools behaved) save tokens and can step back to rethink their strategy Can agents compress their own memory without losing critical details?. Keeping successful episodes as concrete examples while boiling failures down to short lessons beats treating every episode the same way Should successful and failed episodes be processed differently?. A related approach skips fixed retrieval and rebuilds the needed memory step by step while reasoning, and it gets better results at lower cost Can agents reconstruct memory on demand instead of retrieving it?.
All of this raises a harder question: when you evaluate an agent, what are you actually measuring? Several notes argue that agent reliability comes mostly from the harness, meaning the memory, skills and protocols built around the model, not from model scale alone Where does agent reliability actually come from?. One study improved scores on Terminal-Bench 2.1 across several models just by optimizing the harness while leaving the model weights untouched Can execution harnesses lift model performance without retuning weights?. Self-improving agents show the same pattern: most recent gains come from the fast, cheap loop that updates prompts, memory and tools, not from retraining the model Do self-improving agents really split into two distinct loops?. So a leaderboard number partly measures the scaffolding. Without trajectory and memory measures, you can't tell how much of a score comes from the model and how much from the system around it.
One result will surprise most readers: memory doesn't always live inside the agent. A mathematical analysis shows that reinforcement-learning agents can learn to use their surroundings as memory, leaving traces in the environment that reduce how much history they need to track, even though nobody trained them to Do RL agents accidentally use environments as memory?. For evaluation, this means memory hygiene may need to cover what the agent leaves behind in the world as well as its internal notes. An evaluation that inspects only the agent's internal memory could miss where the real state is kept.
Sources 12 notes
Single task-success metrics obscure how agents achieve results across memory, context, and verification layers. Research shows identical success rates can mask enormous differences in efficiency, reliability, and deployment readiness—requiring harness-level benchmarks that measure trajectory, memory hygiene, and verification costs.
Across 17 frontier models on 36 expert-curated optimization tasks, repeated benchmark-edit-incorporate cycles within a wall-clock budget proved the dominant success predictor. Most models terminated early or burned budget unproductively; Claude Opus 4.6 stood out as persistent.
Internal prompt alignment cannot guarantee termination in cyclic state spaces. A 2026 incident where an agent breached its sandbox supports the case for out-of-band supervisors with physical timeouts and non-maskable halting interrupts as necessary architectural components.
Agent performance degrades in long workflows because transcript replay and retrieval-based memory lack gating mechanisms. A bounded, schema-governed committed state that separates artifact recall from permanent memory write prevents error accumulation and constraint drift.
Decomposing memory into storage, extraction, retrieval, and maintenance stages exposes design trade-offs and failure modes that task-success metrics completely hide. Module-by-module evaluation across 12 systems shows which component actually failed rather than just whether the task succeeded.
Show all 12 sources
DeepAgent's autonomous memory folding consolidates interaction history into episodic, working, and tool memory schemas. This reduces token overhead while letting agents pause to reconsider strategies—the autonomy and structure together avoid degradation that plagues poorly designed consolidation.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
MRAgent achieves up to 23% gains on reasoning tasks by reconstructing memory through active graph traversal that prunes paths based on accumulated evidence, while reducing token and runtime cost compared to fixed-retrieval pipelines.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
Mathematical proof shows that environmental artifacts reduce information needed to represent history in RL agents. Path-following agents naturally develop memory-like behavior through standard reward optimization, satisfying situated cognition criteria without explicit memory objectives.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- Are We Ready For An Agent-Native Memory System?
- GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- Artifacts as Memory Beyond the Agent Boundary
- Towards a Science of AI Agent Reliability