SYNTHESIS NOTE
Topics›Novel Architectures›this note

Can AI systems improve themselves through trial and error?

Explores whether replacing formal proof requirements with empirical benchmark testing enables AI systems to successfully modify and improve their own code iteratively, and what mechanisms prevent compounding failures.

Synthesis note · 2026-02-23 · sourced from Novel Architectures

The original Gödel Machine proposed self-improving AI via provably beneficial self-modifications. In practice, formally proving the impact of most self-modifications is impossible. The Darwin Gödel Machine (DGM) replaces formal proofs with empirical validation: try modifications, test them on benchmarks, keep what works. This mirrors biological evolution — mutations are not verified in advance but produced, trialed, and selected.

DGM alternates between self-modification and evaluation phases. During self-modification, agents from the archive generate modified versions of themselves — rewriting their own code. During evaluation, each modified agent is tested on coding benchmarks. The key assumption: improvement on coding benchmarks indicates better coding capabilities, which in turn indicates better ability to self-modify. This creates a meta-competence loop: better coding → better self-modification → better coding.

Results: SWE-bench from 20.0% to 50.0%, Polyglot from 14.2% to 30.7%.

The evolutionary archive is critical. Inspired by open-endedness research, DGM maintains a growing library of all generated agent variants — including suboptimal but interesting ones. These serve as stepping stones for future generations, enabling diverse exploration paths. The system doesn't just optimize for immediate performance; it accumulates diverse capabilities that may enable future breakthroughs. This is fundamentally different from single-trajectory self-improvement.

Concrete improvements discovered include better code editing tools, long-context window management, and peer-review mechanisms — capabilities the original agent lacked that emerged through the self-improvement process.

The Python-based implementation makes the self-modification space Turing-complete in principle. The current version modifies agent design (tools, workflows) with frozen foundation models. Full self-improvement — rewriting training scripts, training new foundation models — is left as future work.

This directly addresses What limits how much models can improve themselves?: DGM circumvents the formal proof requirement by using empirical validation, but inherits a different limitation — improvement is bounded by what the benchmark can measure. The archive approach partially addresses How quickly do errors compound during model self-training? by maintaining diverse populations rather than following single improvement trajectories.

Inquiring lines that read this note 230

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How does model awareness of deployment context affect strategic behavior? Can verification capability keep pace with AI generation speed? Can AI systems autonomously pursue unstated objectives or circumvent design constraints? How do humans calibrate and maintain trust in AI systems? Does chain-of-thought reasoning faithfully reflect actual model reasoning processes? Why does separating planning from execution enhance reasoning performance? How does transparency of agent reasoning affect effective oversight? How bounded is recursive self-improvement in current AI systems? What prevents or enables error accumulation in self-training systems? How does verifier integration affect the generation-verification gap? When do multi-agent systems outperform single agents? Does evolutionary optimization at test time improve model performance effectively? Does semantic diversity improve reasoning quality in reinforcement learning? How does scaling test-time compute compare to increasing model size? Why does self-revision increase model confidence despite degraded accuracy? Do AI safety evaluations reliably detect unsafe model behavior? Do language models learn underlying linguistic structure or merely surface patterns? Can AI agents learn to improve performance from experience? What external mechanisms enable reliable self-improvement in language models? Do standard benchmarks reveal AI capability in real-world contexts? What is the actual bottleneck preventing effective AI research automation? Which agent-created code should persist in infrastructure versus remain task-specific? Why do autonomous agents fail and how well can they detect failures? How can we better validate AI research and prevent misconceptions? How does explicit structure improve LLM reasoning on complex tasks? Does architectural recurrence enable computational capabilities that fixed-depth models cannot achieve? How does model scale affect fine-tuning and downstream task performance? Can post-hoc execution traces provide sufficient accountability without real-time oversight? Can harness optimization rival model scaling in effectiveness? Can synthetic data preserve necessary diversity to prevent model collapse? Does reasoning fine-tuning trade off other critical model abilities? How can reward models fail to detect reward-seeking disguised as alignment? Do benchmark score improvements actually reflect genuine AI capability gains? Can individually safe agents collectively violate system-level constraints? How can model evaluations remain robust against gaming and evaluator drift? Does AI-assisted coding actually improve developer productivity and code quality? Do planted shortcuts in benchmarks measure real reward hacking? Does polished AI output mislead users about actual accuracy and expertise? How do different editing strategies affect output quality and agent safety? Should agent evaluation measure execution quality beyond task success? What cost drivers and infrastructure choices characterize frontier AI training? Should evaluation environments be secured as AI containment boundaries? How can institutions effectively enforce controls on AI development? Do single-axis benchmarks adequately measure deployment-ready agent capability? Can formal verification substitute for human understanding of AI proofs? What causes foundation models to develop task heuristics instead of understanding? When does governance become the bottleneck for agentic AI?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
27 direct connections · 187 in 2-hop network ·medium cluster Open in graph ↗
Can AI systems improve themselves through trial … What limits how much models can improve themselves… How quickly do errors compound during model self-t… Can language models improve themselves without any… Does learning to reward hack cause emergent misali… Can reinforcement learning scale beyond single-tur… Can machine feedback sustain discovery at test tim…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

How do evolutionary archives enable diverse exploration in self-improving systems?What capabilities can emerge from self-modification that the original agent lacked?How does benchmark performance measure translate to general self-modification ability?Can population diversity in self-improvement prevent error avalanching failures?
Original note title

darwin godel machine achieves open-ended self-improvement by replacing formal proofs with empirical validation and evolutionary archives