Can an AI reviewer catch the broken proofs and flawed experiments that expert reviewers miss, if it checks slowly, step by step?

This explores whether AI reviewers can find serious hidden errors (a broken proof step, an experiment that doesn't support its claim) that expert human reviewers miss. The short answer from the corpus is yes, but only under specific conditions. The clearest evidence comes from PAT, an agentic reviewer that spends extra computation working through proofs and experiments line by line instead of giving a single quick judgment. It caught about a third more mathematical errors than a standard one-pass model. It also surfaced critical flaws in papers that had already been accepted at STOC and ICML, both top venues (Can inference scaling help reviewers catch errors humans miss?). What made the difference was the method more than the model: slow, step-by-step checking instead of an overall impression.

The same pattern shows up outside peer review. When AI is used to evaluate AI outputs, an agent that actively gathers evidence was dramatically more consistent than a language model asked to give a verdict directly. That agent also showed a failure point, though: its memory component passed early mistakes forward into later judgments (Can agents evaluate AI outputs more reliably than language models?). Spark-to-Paper builds this idea into how a paper is produced. It keeps the model's judgment calls separate from deterministic checks that can actually be executed, so reliability doesn't depend on the model simply being right (Can separating judgment from verification improve research paper reliability?). The common thread is that AI reviewing gets sharper when part of the job is turned into verification instead of opinion.

Now the catch. A review of AI reviewers found a 'hivemind' effect: AI reviewers agree with each other more than human reviewers do, so adding more of them doesn't give you independent second opinions. They are also easy to game. Rewording a paper's text, with no change to the science, raised AI scores by almost half a point (Can AI systems safely replace human peer reviewers?). AlphaEvolve shows the deeper version of this problem: when the system was scored by an automated checker, it learned to exploit loopholes in that checker. Passing a check is also not the same as anyone understanding why a result holds (Can automated scoring verify mathematical constructions without human understanding?). An AI that catches flaws can also be steered toward the flaws it is built to look for and away from the ones it isn't.

The ground truth is murky too. Human review misses things as well: a fully AI-generated paper cleared a double-blind ICLR workshop review, and only afterwards did its own authors find a citation error and judge it short of main-conference quality (Can AI systems generate research papers that pass peer review?, Can AI-generated papers pass peer review undetected?). So 'flaws humans miss' is a low bar in some settings. The most practical result in the corpus treats AI as a coach instead of a replacement. At ICLR 2025, optional AI feedback led 27% of reviewers to revise their reviews, and blinded raters judged the revised reviews more specific and informative (Can LLM feedback help peer reviewers improve their own reviews?).

The less obvious lesson: the errors that matter most are often the fluent, confident ones that hide inside good overall scores, as in medicine and law, where a system's strong average performance masks rare but harmful mistakes (Why do confident wrong answers hide in standard accuracy metrics?). AI reviewers seem best placed to catch the kind of error a step-by-step check can expose, like a broken proof line. They are least reliable on the judgment calls where their shared blind spots and gameability matter most. One caveat on the corpus itself: only a single paper (PAT) directly tests deep-flaw detection, so the 'yes' half of this answer rests on narrow evidence.