Does banning AI from peer review actually keep it out of the reviews, or does the rule do little?

This explores whether a conference rule against LLM use changes how much AI ends up in peer reviews. The corpus suggests a rule on its own does little. The most direct evidence comes from a randomized experiment at ICML 2026. Reviewers were assigned either to a full ban or to a limited-use policy. Paper scores, accept/reject decisions and reviewer confidence came out almost the same under both. Substantial fractions of reviewers also broke whichever rule they were given Does banning LLM use in peer review change review outcomes?. So the policy text didn't reliably control behavior, and the stricter rule didn't produce measurably different reviews.

One reason is that the usage is large and hard to see. A population-level analysis of ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023 estimated that 6.5% to 16.9% of review text was substantially modified by LLMs. Rates were higher among reviewers who reported low confidence, submitted close to the deadline, or engaged less How much peer review text shows signs of LLM modification?. That pattern matters. AI use tracks time pressure and low engagement, and a ban doesn't reduce either one. Detecting individual cases is also unreliable. Even ML experts can't consistently tell LLM-written research text from human writing Can readers tell LLM abstracts from human ones?. A ban without dependable detection mostly relies on the honor system.

Conferences have responded by enforcing where they can verify. ICLR 2026 treated LLM-detector flags as one input for human area chairs, not as automatic verdicts, because the detectors are imperfect. The firm action was saved for something checkable: papers with confirmed fabricated references were desk-rejected How can conferences detect and handle LLM misuse in peer review?. This moves the question from 'did you use AI?', which is hard to prove, to 'is this output wrong in a way we can show?'

The worry about AI-written reviews goes beyond rule-breaking. AI reviewers show a 'hivemind' effect: they agree with each other more than human reviewers do. They are also easy to game. Rewriting a paper's text, with no change to the science, raised AI scores by 0.45 points Can AI systems safely replace human peer reviewers?. LLM judges also give higher scores to responses with authoritative-looking references or polished formatting, regardless of content Can LLM judges be fooled by fake credentials and formatting?. These are the failures a ban is meant to prevent, and they are why weak enforcement is a real problem.

The less obvious finding is that guiding how AI is used may work better than prohibiting it. In a randomized trial at ICLR 2025, reviewers got optional feedback from Claude-based agents on their own draft reviews. 27% of them revised, and blinded raters judged the revised reviews more informative and clearer Can LLM feedback help peer reviewers improve their own reviews?. A position paper argues that review failures come from authors, reviewers and venues together. It proposes two-stage review, where authors rate review quality before seeing decisions, plus badges that reward thorough reviewers Can two-stage review and badges fix AI conference peer review?. The pattern across the corpus: bans aim at the tool, but the measurable gains come from targeting incentives, verifiable errors and review quality.