An AI rewrote its own search code and got better results, but can anyone really explain why it works?

This explores whether people can really explain why an AI-discovered search strategy worked, as opposed to just seeing that it did. The short answer from the corpus: humans can often read what the AI did, but that is not the same as understanding why it worked. No note here directly tests whether people can explain an AI's search choices, so the answer has to be pieced together from nearby evidence.

The most concrete case is bilevel autoresearch Can an AI system improve its own search methods automatically?. An outer AI loop read the code of an inner search loop, found where it got stuck, and wrote new Python mechanisms (bandit methods and combinatorial optimization) that broke the inner loop's repetitive patterns, improving GPT pretraining results by 5x. Because the output is ordinary code, a human can inspect every line. Yet a readable mechanism plus a better score doesn't add up to an explanation. You can see that the AI added randomness to escape a rut without knowing whether that was the real cause or just one of several changes that happened to come together. SAGA pushes this further Can agents evolve their own objectives during search?: the AI rewrites not only its methods but also the goals it scores itself against. Once the target itself has moved, the question "why did it succeed?" gets harder to answer, because "succeed at what?" has shifted along the way.

The surprising lesson comes from mathematics. When an AI produced a formal Lean proof of Erdős Problem 728 Did an AI system truly solve Erdős Problem 728 autonomously?, the proof was checked by machine and is beyond dispute. Researchers still had to translate it into informal mathematics so people could follow it, and the note points out that nobody has tested whether readers actually understand it. So the case splits apart two things we usually treat as one: being *verified* correct and being *understood*. An AI result can be certainly right and still opaque.

There is a deeper reason to be careful. The "imposter intelligence" note Can AI pass every test while understanding nothing? shows that two networks can give identical outputs on every input while being organized very differently inside, and standard tests can't tell them apart. If success metrics can't reveal how a model is built internally, they also can't tell you why a strategy worked. The same gap appears in search agents: high benchmark scores routinely fail to predict whether real users are satisfied, because the benchmarks measure something narrower than real searching Why do search agents fail users despite strong benchmark scores?. A number going up gives you no causal story.

The human side matters too. The Rose-Frame work Why do people trust AI outputs they shouldn't? describes how people mistake a fluent account for the thing itself, and how confirmation bias amplifies that mistake. When an AI's success comes with a tidy explanation, whether from the system or from us, we tend to accept it. What you might not expect to take away: often the most useful question isn't "do I understand why this worked?" but "what would show me that my explanation is wrong?" That second question is the one these systems rarely answer for you.