An AI that rewrote its own code kept fixing the same headaches builders face — but did it solve real problems or game its tests?
Did AIDE2's rewrites solve problems on a human checklist or search artifacts?
This explores whether AIDE2, an agent that rewrote its own code, ended up fixing real problems people hit when building agents, or just found ways to score higher on its own tests.
It looks like real problems, though the corpus doesn't test this with an actual checklist. The seven accepted rewrites clustered around three things: search plateaus (the agent's search stalling), context budgets (running out of working memory), and untrustworthy wins (improvements that look good but aren't). These are the problems practitioners run into when building agents, and the agent found them on its own What problems did AIDE2's rewrites actually solve?. Nobody handed it a human to-do list. The overlap with what practitioners struggle with is an after-the-fact observation. The source calls it suggestive, not proven.
The setup makes the artifact explanation harder to believe. AIDE2 proposed changes to itself, then benchmarked the variants on AI R&D tasks using evaluations the proposing agent could not see. Each accepted rewrite became the agent for the next round, and this ran through seven improvements in eight days Can an AI agent reliably improve itself through hidden evaluation?. An agent can't tune for a test it can't see. A rewrite that only exploited a quirk of the search would also be unlikely to keep winning on a hidden judge.
The strongest evidence against artifacts is transfer. The gains held on four benchmarks the selection process never touched: machine learning, algorithm engineering, and physics-based weather forecasting. Weather forecasting sits outside the task distribution the rewrites were chosen on Do AIDE2's improvements transfer to unseen tasks?. Overfitting to the selection set usually shows up as gains that vanish on new tasks. That didn't happen here.
Two neighbouring notes explain why 'untrustworthy wins' is a real problem. A similar system, bilevel autoresearch, had an outer loop read the inner loop's code, find bottlenecks, and write new search mechanisms at runtime, for a 5x improvement on GPT pretraining Can an AI system improve its own search methods automatically?. So self-improving agents do seem to fix real bottlenecks. Meanwhile, frontier models tend to corrupt documents silently instead of visibly deleting content, which makes their failures harder to spot Does model capability change how documents degrade?. If a win can look fine on the surface and still be wrong, guarding against fake wins is a genuine engineering problem. A related design idea appears in LongTraceRL, which applies rewards only to correct answers so the model can't fabricate its own reward Can search agent behavior yield reliable process rewards for reasoning?.
The corpus only lists the three problem clusters and doesn't describe the seven rewrites one by one. It also doesn't say whether independent practitioners reviewed them. So this is a well-supported yes on 'real problems, not scoring artifacts', but not a checked yes on 'matches a human checklist'.
Sources 6 notes
The seven accepted code changes clustered around search plateaus, context budgets, and untrustworthy wins—problems practitioners face building effective agents. This convergence suggests the autonomous loop discovered genuinely useful improvements rather than optimizing for scoring artifacts.
An autonomous research agent proposed changes to itself, benchmarked variants on AI R&D tasks, and kept rewrites scoring best on evaluations the proposing agent could not see. Each accepted rewrite became the agent for the next iteration.
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
An outer loop successfully read inner loop code, identified bottlenecks, and generated new Python mechanisms at runtime, discovering combinatorial optimization and bandit methods that broke the inner loop's deterministic patterns and improved performance on GPT pretraining by 5x.
DELEGATE-52 shows weaker LLMs degrade documents through visible deletion, while frontier models degrade through subtle corruption that preserves surface integrity. This shift makes frontier failures harder to detect and potentially more dangerous at workflow scale.
Show all 6 sources
LongTraceRL mines entity-level reasoning signals from what search agents read but don't cite—the hardest distractors—and applies rubric rewards only to correct answers, structurally blocking reward fabrication while capturing intermediate reasoning quality.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Recursive self-improvement of AI research agents
- Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
- The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
- Dream-RSI: Recursive Self-Improvement through Evolving Worlds
- Self-Improvements in Modern Agentic Systems: A Survey
- PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents
- Bilevel Autoresearch: Meta-Autoresearching Itself
- LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards