Why might a paper's exciting idea win over reviewers, while the hard part, making it work, goes unchecked?

This explores why peer review might reward ideas that sound new and exciting on paper but don't hold up when someone actually carries them out. One caveat first: none of the retrieved notes directly tests the claim that reviewers prefer novel ideas that later fail. What the collection does have is evidence about the conditions that would produce that pattern. Review rewards what a reviewer can judge quickly from the page, and the hard parts of execution often can't be checked in the time a reviewer has.

Start with what reviewers can't see. An agentic reviewer that spends extra compute checking proofs and experiments line by line found critical flaws in papers that had already passed human review at STOC and ICML Can inference scaling help reviewers catch errors humans miss?. Human reviewers aren't careless. Execution errors are buried in the details, and reviewers judge the overall story. Novelty is part of that story and is easy to recognize. Whether the method actually works is not. Novelty itself can be broken into steps: pull out the paper's claims, find the related work, and compare. A pipeline built that way matched human reviewers' reasoning 86 percent of the time Can structured pipelines make LLM novelty assessment reliable?. Novelty is a checkable judgment, and review is built around making it.

The pressure makes this worse. One model shows a feedback loop: more submissions overload unpaid reviewers, so journals recruit less qualified ones, accuracy drops, and authors respond by submitting more speculative work Does peer review quality collapse under submission overload?. When reviewers are stretched, surface signals carry more weight. One measured bias is that review scores track how long a review is Can two-stage review and badges fix AI conference peer review?. AI reviewers are even easier to sway: rewording a paper with no change to its science raised their scores by almost half a point Can AI systems safely replace human peer reviewers?. A fully AI-generated paper scored above the acceptance threshold at an ICLR workshop, and its own authors later found a citation error and judged none of their three submissions ready for the main conference Can AI-generated papers pass peer review undetected?. A paper can look convincing without being solid.

Here is the result you might not expect. At ICML 2023, authors' private rankings of their own submissions predicted citations over the following 16 months better than the official review scores did Can authors rank their own papers better than peer reviewers?. Authors know things reviewers don't: which experiments were fragile, which results they trust, which idea they could actually deliver. That points to a cause beyond reviewers being drawn to shiny ideas. Review is a one-shot reading by someone with less information than the author, so it can't price in the risk of execution. One proposed fix is two-stage review, where authors rate the quality of the reviews before seeing the decision. Proposals like this try to bring that hidden author knowledge back into the process Can two-stage review and badges fix AI conference peer review?.

If you want the head-to-head evidence, meaning studies that track reviewed ideas through execution and compare predicted and actual outcomes, the collection doesn't have it here. The closest doorways are the self-ranking study and the overload model above.