How does FunSearch actually verify its discovered programs?

Romera-Paredes et al. describe FunSearch as "an evolutionary procedure based on pairing a pretrained LLM with a systematic evaluator," and report new cap set constructions "going beyond the best-known ones" and bin-packing heuristics that "improve on widely used baselines." The verification in that claim is the evaluate function: every candidate program is scored, and the evaluator "guards against confabulations and incorrect ideas." The paper calls the result "a new piece of verifiable knowledge," and the warrant for that word is the score. The excerpt does not show a human checking the constructions.

The mechanism the excerpt gives is a loop, not a single model call. The LLM is frozen: results were obtained "without any fine-tuning on our problems." The loop does the work through best-shot prompting, a fixed program skeleton so that only "the critical part" is evolved, and an island model that periodically discards "the programs in the worst half of the islands" and reseeds them from the best survivors. The output is a program that generates the solution rather than the solution itself, and such programs "tend to be more interpretable—facilitating interactions with domain experts—and concise." That interpretability sentence is the only place the excerpt touches human understanding. It is hedged with "tend to be," and it claims only that interpretability facilitates interaction with experts, not that anyone understood the output.

Against the nearest notes, FunSearch is the engineering version of the asymmetry that What limits how much models can improve themselves? formalizes. Its opening premise is that many problems are "easy to evaluate, despite being typically 'hard to solve,'" and the method is built on that gap without the excerpt measuring it. It also fills the verifier slot that Can AI systems improve themselves through trial and error? fills with benchmarks: empirical scoring in place of formal proof, applied candidate by candidate. Against Can search escape the entropy shell of language models?, FunSearch is an evolutionary search whose evaluator gives a dense, exact score on every candidate, the regime where BES's decomposition machinery is least needed, though the excerpt does not test that comparison. The foil is Can training and search gains add together in program evolution?: that note credits learning and search separately, while FunSearch has no learning component. The excerpt places its performance levers in the sampling trade-off, the skeleton and the islands, and reports that results "are not too sensitive to the exact choice of LLM."

The excerpt establishes less than its framing suggests. It shows that a score can verify a discovery against the stated evaluate function. It does not show that anyone understands the discovered programs, it measures no expert's comprehension, and it says nothing about the size of the improvements, which sit in results sections not included here. A verified program is correct relative to the evaluator, and whether the evaluator captures the intended problem is a separate check. At the strength the evidence allows, the evaluator licenses "correct" for each program, and the paper's interpretability sentence licenses at most a hypothesis that expert feedback is easier with code than with raw solutions. That hypothesis is untested in this excerpt, so verified and understood stay as two separate claims.