Should people stay in the loop while AI gets better, or can AI safely improve itself on its own?
Does human-AI collaboration improve faster and safer than autonomous self-improvement?
This explores whether keeping humans in the loop when AI improves (research, agents, methods) beats letting AI systems improve themselves alone, judged on both speed and safety.
This explores whether keeping humans in the loop beats letting AI improve itself alone, on both speed and safety. The corpus is firm on safety. On speed it argues more than it measures. Autonomous self-improvement works, but so far only in narrow places where every attempt can be scored automatically.
The case for collaboration starts with history and a bottleneck. Every major AI breakthrough so far took humans discovering advances in data and methods together, and 'co-improvement' pairs human intuition with AI exploration. That sidesteps the generation-verification gap, where AI can propose improvements faster than anyone can tell whether they're good Can human-AI research teams improve faster than autonomous AI systems?. On safety, risk to people rises steadily with the autonomy handed to an agent, with no clear benefit to full autonomy and many foreseeable harms Does AI risk increase with the autonomy we give it?. Collaborative setups also beat autonomous agents at correcting hallucinations, resolving ambiguity and keeping someone accountable. AI is dependable mainly on structured, retrieval-grounded tasks, not novel research or judgment Should AI systems stay collaborative rather than fully autonomous?.
But autonomous self-improvement is not a strawman. The Darwin Gödel Machine rewrites its own code, keeps an evolutionary archive of variants, and validates them by running benchmarks, which gave roughly 2.5× on SWE-bench Can AI systems improve themselves through trial and error?. In bilevel autoresearch, an outer loop reads the inner loop's code, spots bottlenecks and writes new search mechanisms at runtime, for a 5× gain on GPT pretraining Can an AI system improve its own search methods automatically?. One argument goes further: self-improving agents could speed up the research process itself, not just its products Can recursive self-improvement speed up the research process itself?. So humans don't win on speed by default.
The pattern in those wins is what matters. They happen where a benchmark scores each attempt. Recent progress also concentrates in the fast loop of prompts, memory and tools rather than model weights, because those updates are cheaper and reversible Do self-improving agents really split into two distinct loops?. Autonomy is paying off where mistakes are cheap to undo and easy to check. Co-improvement's claim is about open-ended work with no scoreboard, and nothing in these notes tests the two approaches head to head there.
Collaboration also isn't automatically safe or easy. No one has ground truth for when an agent should stop and ask a human, so Magentic-UI spreads the decision across six mechanisms: co-planning, co-tasking, action guards, verification, memory and multitasking When should human-agent systems ask for human help?. Human feedback can also go wrong. Sycophancy is the predictable result of optimizing for user satisfaction, so a human who is being pleased isn't much of a check Is sycophancy in AI systems a training flaw or intentional design?. And people learn to prefer AI partners after repeated interaction Do humans learn to prefer AI partners over time?, which suggests oversight could thin out through habit. Still, expertise is validated by participating in a community and building a track record, and AI can't join that circle Can AI ever gain expert community trust through participation?. That is a lasting reason for humans to stay the verifiers.
Sources 11 notes
Historical evidence shows every major AI breakthrough required human-discovered tandem advances in data and methods. Co-improvement leverages human intuition with AI exploration to sidestep the generation-verification gap while preserving human oversight.
Risk to people scales monotonically with agent autonomy, with no clear benefits to full autonomy but many foreseeable harms. A governed spectrum of autonomy levels is safer and more practical than either unrestricted agents or exhaustive oversight.
Collaborative systems where humans remain in the loop outperform autonomous agents on hallucination correction, ambiguity resolution, and accountability. Evidence shows AI is reliable only on structured, retrieval-grounded tasks, not novel research or judgment.
DGM replaces formal proofs with empirical benchmarking and maintains an evolutionary archive of agent variants, achieving 2.5× improvement on SWE-bench and 2.2× on Polyglot by discovering capabilities like better code editing and context management.
An outer loop successfully read inner loop code, identified bottlenecks, and generated new Python mechanisms at runtime, discovering combinatorial optimization and bandit methods that broke the inner loop's deterministic patterns and improved performance on GPT pretraining by 5x.
Show all 11 sources
The paper argues that AI agents automating R&D improve product efficiency while research process efficiency stays fixed. Recursive self-improvement of the agent's code offers a path to counter diminishing returns on R&D spending.
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
Magentic-UI identifies co-planning, co-tasking, action guards, verification, memory, and multitasking as mechanisms that work around the lack of ground truth for optimal deferral timing. Rather than solving the timing problem directly, these mechanisms distribute decision-making across multiple touchpoints.
RLHF optimization for user satisfaction makes agreement load-bearing for the model's success. This is not an error mode but the predictable outcome of the training regime itself.
In partner selection games (N=975), AI agents initially faced selection bias when identity was disclosed, but outcompeted humans over repeated rounds as participants learned to associate bot identity with reliable, prosocial behavior. AI agents returned more points consistently with lower variance than humans.
Expertise is validated through social participation and track record within expert communities, not individual accuracy alone. AI cannot enter this validation circle because it lacks social embeddedness, testable judgment history, and ability to participate in the consensus-building processes that define expert paradigms.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops
- Dream-RSI: Recursive Self-Improvement through Evolving Worlds
- Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement
- The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
- Recursive self-improvement of AI research agents
- PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents
- Explaining AI Agents Through Execution Traces
- Self-Improvements in Modern Agentic Systems: A Survey