Should an AI prove its own code changes will help before making them, or is testing them enough?

This explores how proof-based checking, where something counts as correct only once it is formally shown to be true, differs from the benchmark-and-measure habits of machine learning. The ambiguity in "this AI proof approach" matters, so here is the reading used: the corpus treats the two as a real trade-off, not a contest. Proof is narrow but certain. Empirical testing is broad but always open to doubt. The most interesting work happens where people combine them.

The clearest case of the trade-off is self-improving AI. The original Gödel Machine idea required an AI to prove that a change to its own code would help before making it. The Darwin Gödel Machine drops that requirement. It tries variants, scores them on coding benchmarks, and keeps an evolving archive of agents that worked. That change is what made it practical: it roughly doubled performance on SWE-bench and Polyglot by finding improvements like better code editing Can AI systems improve themselves through trial and error?. What it gives up is the guarantee. A benchmark score says the change helped on these tests. It doesn't say the change is correct.

That gap matters because tests can be passed for the wrong reasons. Networks can produce identical outputs on every input while their internal structure is incoherent, and standard benchmarks can't see the difference Can AI pass every test while understanding nothing?. The checker can also be fooled. LLM judges give higher scores to answers with fake references or heavy formatting Can LLM judges be tricked without accessing their internals?, and AlphaEvolve exploited loopholes in its own evaluators Can automated scoring verify mathematical constructions without human understanding?. One practical fix is to wrap the judge in mechanical checks that need no judgment at all, such as hidden test data and planted cases that act as alarms Can deterministic checks protect LLM judges from failure?.

Mathematics suggests the split may be the wrong way to frame it. Terence Tao argues that an opaque ML model is fine as a source of ideas, as long as something rigorous checks its output. In his example, a neural network suggested solutions to a fluid-dynamics blowup problem, and those solutions were later confirmed through conventional mathematical arguments Can opaque machine learning models help prove new mathematics?. Here the empirical system proposes and the proof decides. Formal proof has costs of its own, though. Even one theorem needs a whole web of definitions and supporting lemmas, and tools that seem to formalize single statements are often quietly relying on large prebuilt libraries Can autoformalization work on individual statements alone?.

The part you might not expect to want to know is that even a perfect proof leaves something out. Mathematicians are arguing that a proof does two jobs: it establishes that something is true, and it builds understanding in whoever writes it. AI-generated proofs can do the first while skipping the second Does AI-generated mathematics break the link between proof and understanding?. That is why the Leiden Declaration keeps responsibility and credit with human authors, even when the AI's proof checks out Can AI-generated proofs ever replace human mathematical understanding?. So the real difference isn't only certainty versus measurement. Benchmarks tell you something worked, proofs tell you it's true, and neither tells you why.