Can two-stage review and badges fix AI conference peer review?

The paper's authors argue that the peer review crisis at major AI conferences cannot be placed on reviewers alone. They name three parties, authors, reviewers and "the System" (the venue and OpenReview), and say all three "share responsibility for the current problems." Author misconduct, they add, "can only be addressed through policy enforcement and detection tools," so they focus on reviewer accountability: a two-stage bi-directional review in which authors rate reviews, and a systematic reviewer reward system.

The first mechanism changes the order of release. Today all reviews and ratings reach authors at once. Under the proposal, each review's summary, strengths and clarifying questions come first, and authors grade them on the reviewer's comprehension and the constructiveness of the questions. Weaknesses and ratings follow, and authors cannot revise their evaluations afterward. The authors argue that rating a review before seeing its verdict blocks "retaliatory scoring." An LLM review, visible only to authors, is meant to be "a psychological deterrent" for reviewers and "a soft reference point" for spotting machine-written reviews. For rewards, reviewers would earn verifiable badges. The paper concedes that badges could reward lenient reviewing, and asks for a metric that rewards thoroughness over positivity.

The diagnosis rests on NeurIPS studies, as the paper reports them. In consistency experiments in 2014 and 2021, "16-23% of papers could have been either accepted or rejected based on the reviewing group." A 2022 NeurIPS study of review quality found "author-outcome bias" and "elongated review bias," the latter rating longer reviews higher "even when the two reviews contain the same information." The two-stage design targets the first, and capping the length of stage-one content targets the second. Against the nearest notes, this is a different lever from the one the ICML 2026 trial tested. Does banning LLM use in peer review change review outcomes? found rules barely moved outcomes, while this paper bets on visibility, which it does not test. Its LLM review is a reference for authors, not a reviewer, unlike Can inference scaling help reviewers catch errors humans miss?, where a model checks proofs and experiments line by line. The length bias also parallels the surface sensitivity in How much does rhetorical style shift AI review scores?, here measured in human reviewers.

The excerpt does not establish that the fix works. The paper reports no pilot. Its Section 5.1 asks for a large survey and a small-track pilot first, and Section 5.2 names the main obstacle as getting venues and OpenReview to build the feedback and reward systems, plus cost: ICML 2024 cut in-kind compensation for its top 10% of reviewers to a smaller pool. The claim that stage-one release "could prevent retaliatory scoring" is an inference from the bias finding, not a measured effect, and the NeurIPS results come without effect sizes. The 10,000-submissions figure carries no citation in the excerpt. The strongest support is for the diagnosis, that review is noisy and rating-biased. The prescription is a plausible design that a venue could test as a pilot before adopting it as policy.