Many reviewers broke AI rules in a conference experiment, yet a full ban and limited use gave nearly identical scores.

This explores whether reviewers follow the rules conferences set on using LLMs, and whether those rules change anything. The short answer is that many reviewers don't follow them, and the rules may matter less than people expected. ICML 2026 ran a real experiment: some reviewers were told not to use LLMs at all, and others were allowed limited use. A substantial share of reviewers broke whichever rule they were given. The more surprising result is that the two policies produced almost the same paper scores, accept/reject decisions and reviewer confidence Does banning LLM use in peer review change review outcomes?. The policy barely moved the result it was meant to protect.

Measuring rule-breaking is harder than it sounds, because the most careful rule-breakers are the hardest to catch. ICML hid instructions inside submitted PDFs that an LLM would follow but a human reader wouldn't notice. These watermarks flagged 795 reviews, about 1% of the total, and led to 497 desk rejections. The program chairs said this mostly catches careless use: a reviewer who rewrote the output or removed the hidden text would go unnoticed How many peer reviewers secretly used LLMs despite the ban?. A different method gives a sense of the real scale. Instead of judging each review, researchers estimated how much LLM-modified text appears across a whole conference. They put it at roughly 6.5% to 16.9% of reviews at ICLR, NeurIPS and other venues in 2023–24. Rates were higher among reviewers who were less confident, rushed, or submitted near the deadline How much peer review text shows signs of LLM modification?. That points to time pressure, not laziness, as the main driver.

Enforcement is also hard because people can't reliably spot LLM writing by eye. Readers with ML expertise couldn't consistently tell LLM-written research abstracts from human ones Can readers tell LLM abstracts from human ones?. ICLR 2026 responded by treating detector flags as one piece of evidence for area chairs to weigh, not as automatic verdicts. Its firm enforcement went to something that can actually be checked: confirmed fabricated references led to desk rejection How can conferences detect and handle LLM misuse in peer review?. In practice, conferences are moving from policing whether an LLM was used to policing verifiable errors.

That shift makes sense once you see that unsupervised LLM use and approved LLM use look very different. When LLMs act as reviewers on their own, simulated LLM reviewers have rated LLM-written papers higher and marked down human papers that make critical arguments Do LLM reviewers favor papers written by other LLMs?. But in a study of more than 125,000 real reviews, the apparent favoritism disappeared once paper quality was taken into account. LLM-assisted reviewers were simply more lenient toward weaker papers, and LLM-written papers tended to be weaker Do LLM reviewers actually favor LLM-written papers?. When the conference itself offered LLM help openly, the results were good. At ICLR 2025, optional LLM feedback on draft reviews led 27% of reviewers to revise them, and independent raters judged the revised reviews more informative Can LLM feedback help peer reviewers improve their own reviews?.

The takeaway you might not have expected is that a ban mostly pushes LLM use out of sight, where it is harder to study and to improve. The ICML experiment suggests the choice between banning and limiting matters less than whether reviewers have the time and support to do the work well. The corpus says little about why individual reviewers decide to break the rules, or whether stricter penalties would change that. Those questions are still open.