A randomized test at a major AI conference found reviewer AI rules barely moved scores, and many reviewers broke them anyway.

This explores what happens when reviewers use AI against the rules: does it change decisions, who gets hurt, and how does the rest of the system react? The most direct evidence is surprising. In a randomized experiment at ICML 2026, some reviewers were told not to use LLMs and others were allowed limited use. The rule made almost no difference to scores, accept/reject decisions, or reviewer confidence, and large shares of reviewers broke whichever rule they were given Does banning LLM use in peer review change review outcomes?. So rule-breaking is common, and at least in measurable outcomes it doesn't obviously tilt results. Policy may be shaping what reviewers admit to more than what they actually do.

The bigger effect may be on authors, who now assume their paper will be read by a machine whatever the policy says. Researchers found hidden instructions in 18 arXiv manuscripts telling AI reviewers to give positive assessments. The authors were betting that reviewers would paste the paper into a chatbot Are hidden AI prompts in preprints a deceptive research practice?. That bet makes sense, because AI evaluators are easy to sway. They give higher scores to text with fake references or polished formatting Can LLM judges be tricked without accessing their internals?, and simply rewriting a paper's wording, with no change to the science, raised AI review scores by about half a point Can AI systems safely replace human peer reviewers?. Hidden AI use by reviewers therefore creates an opening that a dishonest author can exploit. A banned tool is still a target.

There is also a quieter problem: sameness. AI reviewers tend to agree with each other more than human reviewers do, a 'hivemind' effect Can AI systems safely replace human peer reviewers?. Peer review relies on several independent readers catching different things. If two of three 'independent' reviewers are secretly running the same model, the paper effectively gets one opinion counted twice, and nobody knows it.

Enforcement is hard in both directions. Accusing someone of using AI is unreliable. One study found that text accused of being AI-written lacked the features that actually separate AI writing from human writing, so accusations can end up wrongly punishing honest writers Do unfounded AI accusations harm human writers instead?. Openly admitting AI use also costs something: disclosure labels lower ratings slightly, though consistently Does disclosing AI assistance make readers trust articles less?. That study looked at news articles, not reviews, but it suggests why people might hide AI use instead of declaring it. A survey of 230 publications describes the whole situation as a linked arms race. More AI-produced papers lead to more automated review, which leads to manipulation, then defenses, then evasion, with each move setting off the next Does AI create a coupled arms race in research production and review?.

Some of the material suggests a ban may be the wrong tool altogether. A carefully designed AI reviewer that checks proofs line by line found serious errors that had already passed human review at top venues Can inference scaling help reviewers catch errors humans miss?. Proposals such as review-and-revise loops with built-in defenses against hidden prompts Can automated review loops handle AI-generated research at scale?, or systems that let authors rate how good their reviews were and reward thorough reviewers Can two-stage review and badges fix AI conference peer review?, treat the problem as one of design and incentives rather than policing. What the collection doesn't yet have is direct evidence from journals specifically, or data on whether hidden AI use has changed the fate of individual papers.