When an AI agent patches its own setup after every failure, is it learning anything — or just memorizing fixes?

This is about lasting code artifacts versus per-task patches to an agent's harness (the prompts, tools and scaffolding around a model). The corpus has no head-to-head test, but it points one way: patches mostly save the agent work it could have done anyway, while durable artifacts change what the agent can do. When researchers analyzed evolved harnesses, the edits were sensible, but most just recorded fixes an agent could have rediscovered in a single rollout. They cache what is already within reach instead of turning hard failures into successes, so gains stay small (Do harness edits learn reusable strategies or memorize task fixes?).

The durable alternatives all put the value in structure that carries over between tasks. Reorganizing a harness repository around a map from runtime behavior to code raised win rates by 10–19 points. It also let a weaker planner match stronger models at finding the right code to change (Can explicit behavior maps help weaker planners compete with stronger models?). Separating a trained curator from a frozen executor pushed skill libraries away from long, generic additions. They moved toward actionable execution logic and cross-task meta-strategies, and the curator worked across different executor models (Can a separate trained curator improve skill libraries better than frozen agents?). The Darwin Gödel Machine kept an archive of agent variants and found general capabilities like better code editing and context management, roughly doubling its SWE-bench and Polyglot scores (Can AI systems improve themselves through trial and error?). The test that separates the two kinds of artifact is transfer. AIDE2's gains held on four held-out benchmarks, including physics-based weather forecasting, which sits outside the distribution it was selected on (Do AIDE2's improvements transfer to unseen tasks?). A task-specific patch can't pass that test by definition.

Durable artifacts also pay off through where they live and how often they're reused. In a 115-day persistent-agent case study, 82.9% of tokens were cache reads. When context persists and gets reused, the meaningful cost unit becomes the completed artifact rather than the token (Do persistent agents really cost less per token?). Something reused across many tasks spreads its cost over all of them. A safeguard encoded in the memory layer the agent actually consults, such as governance rules, worked better than a policy sitting outside the agent's working environment (Can governance rules embedded in runtime memory actually protect autonomous agents?). A durable artifact only helps if it sits where the agent looks.

Two caveats keep this from being a clean win. First, "durable" is a matter of degree. BenchShield sells itself as more reusable than task-specific defenses, but it checks each run against task bindings, and the paper doesn't say who writes them or what they cost. If those are per-task artifacts, the contrast with patches is one of degree rather than kind (How reusable is BenchShield if task bindings require per-task work?). Second, the consumer matters as much as the artifact. Models of every size are about equally good at writing harness updates, but the ability to benefit from them peaks in mid-tier models. Weak models fail to invoke the harness, and strong ones struggle to follow its instructions faithfully (Do stronger models always evolve harnesses better?). A well-built artifact can still go unused, so its value depends on a model that will use it.