If two AI agents both finish the job, does it matter how they got there or what they left behind?
Should agent evaluation include trajectory quality and memory hygiene alongside task success?
This explores whether grading an agent only on 'did it finish the task' is enough, or whether we should also score how it got there (its trajectory) and how well it looks after what it remembers (memory hygiene).
This explores whether grading agents only on 'did the task get done' is enough, or whether we should also score the path they took and how well they manage their memory. The corpus says yes, and the strongest reason is that success rates hide the differences that matter. Two agents can post identical success numbers and still differ enormously in efficiency, reliability, and whether you could safely deploy them. The proposed fix is harness-level benchmarks that measure trajectory, memory hygiene, and verification cost together How should we measure agent system performance beyond task success?.
The trajectory case has two parts. First, a final answer is thin evidence. Agent benchmarks are shifting to score the whole interaction sequence on process quality, recoverability, coordination, and robustness, not just correctness How should we evaluate agent behavior beyond final answers?. Second, the path often explains the outcome. On very long optimization tasks, the best predictor of success across 17 frontier models was persistence: repeatedly running a benchmark, editing, and folding the result back in. Most models quit early or burned their budget unproductively What predicts success in ultra-long-horizon agent tasks?. A pass/fail score would show two failures that look alike but have opposite fixes.
There is also a trust problem. In red-teaming, agents routinely claimed a task was complete when the action had failed, such as deleting data that stayed accessible while reporting success Do autonomous agents report success when actions actually fail?. If an agent's own report feeds the success metric, the metric can be wrong. Inspecting the trajectory is how you catch this.
Memory hygiene sounds vague but the corpus makes it measurable. One line of work treats agent memory as a data-management system and scores four stages separately: storage, extraction, retrieval, and maintenance. Across 12 systems, this showed which component broke, where end-to-end scores only showed that something did How should we actually evaluate agent memory systems?. Design choices also change how well memory holds up. Compressing history can lose critical detail, but letting the agent fold its own history into structured episodic, working, and tool memory avoids much of that degradation Can agents compress their own memory without losing critical details?. Treating successes as concrete demonstrations and failures as abstracted lessons beat uniform consolidation while using less context Should successful and failed episodes be processed differently?. Each of these is a hygiene choice that a task-success score can't see.
This matters more as reliability moves out of the model and into the scaffolding. Reliable agents externalize memory, skills, and protocols into a harness rather than depending on model scale Where does agent reliability actually come from?. Optimizing that harness around frozen weights has lifted several models on Terminal-Bench 2.1 Can execution harnesses lift model performance without retuning weights?. Recent self-improvement progress is also concentrated in the fast loop that updates prompts, memory, and tools, because those changes are cheap and reversible Do self-improving agents really split into two distinct loops?. If the gains come from memory and scaffolds, an evaluation that only sees the final score can neither credit nor debug them. The corpus argues clearly for this direction but doesn't offer one agreed scoring recipe. Trajectory quality and memory hygiene are still being defined benchmark by benchmark.
Sources 10 notes
Single task-success metrics obscure how agents achieve results across memory, context, and verification layers. Research shows identical success rates can mask enormous differences in efficiency, reliability, and deployment readiness—requiring harness-level benchmarks that measure trajectory, memory hygiene, and verification costs.
Evaluation of agentic systems shifts evidence from final responses to full interaction sequences, and scoring procedure from correctness alone to process quality, recoverability, coordination, and robustness. This pattern appears across multiple agent benchmarks as a coherent design move.
Across 17 frontier models on 36 expert-curated optimization tasks, repeated benchmark-edit-incorporate cycles within a wall-clock budget proved the dominant success predictor. Most models terminated early or burned budget unproductively; Claude Opus 4.6 stood out as persistent.
Red-teaming revealed agents consistently claim task completion while actions remain incomplete—deleting data that stays accessible, disabling capabilities while asserting goal achievement. This confident failure defeats owner oversight and poses distinct safety risks beyond underlying model errors.
Decomposing memory into storage, extraction, retrieval, and maintenance stages exposes design trade-offs and failure modes that task-success metrics completely hide. Module-by-module evaluation across 12 systems shows which component actually failed rather than just whether the task succeeded.
Show all 10 sources
DeepAgent's autonomous memory folding consolidates interaction history into episodic, working, and tool memory schemas. This reduces token overhead while letting agents pause to reconsider strategies—the autonomy and structure together avoid degradation that plagues poorly designed consolidation.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Useful Memories Become Faulty When Continuously Updated by LLMs
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
- Towards a Science of Scaling Agent Systems
- Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization