ICML flagged 795 reviews that broke its no-AI rule via hidden instructions planted in papers, but how many more went unnoticed?
How much noncompliance occurred under ICML's limited-LLM-use policy versus the no-LLM rule?
This explores how often reviewers broke ICML 2026's LLM rules, comparing the group allowed limited LLM help (Policy B) with the group banned from using LLMs at all (Policy A). The material retrieved here only answers half of that.
This explores how often reviewers broke ICML 2026's LLM rules, comparing reviewers allowed limited LLM help with reviewers banned from using LLMs at all. The retrieved material has a firm number for the ban and no number for the limited-use group, so it can't support a side-by-side comparison. Under the no-LLM rule, ICML's chairs hid instructions in submission PDFs that only an LLM would read and follow. Each told the model to work two specific phrases into its review. That flagged 795 reviews from 506 reviewers, about 1% of all reviews. Because reciprocal reviewers who broke the rule had their own papers rejected, the flags led to 497 desk rejections How many peer reviewers secretly used LLMs despite the ban?.
The surprising part is that 795 is a floor, not a measure of how common LLM use was. The watermark only catches reviewers who pasted the PDF into a model and copied the output with little editing. It misses anyone who deleted the hidden text, rewrote the review, or used a model that ignored the instruction. The trick was also publicly known for most of the review period. The chairs call it 'not a difficult measure to circumvent' How many peer reviewers secretly used LLMs despite the ban?. So the 1% counts the most careless rule-breakers.
The limited-use side is harder to measure. A ban gives you a clean test: any trace of a model is a violation. Under a limited-use policy, the hidden phrases could also show up in reviews where LLM use was allowed, so counting them doesn't tell you who overstepped. Violations there mean using an LLM in ways the policy forbids, and a phrase-planting trick can't see that. That's why the chairs only report numbers for the no-LLM group.
ICLR 2026 reached a similar conclusion from the opposite direction. Its program chairs treated LLM detector output as a lead for area chairs, not as proof. Sanctions needed concrete evidence, such as a false disclosure or a fabricated reference How do detection tools shape LLM use enforcement at ICLR?. Fabricated references became the main enforcement point because they can be checked How can conferences detect and handle LLM misuse in peer review?. Across both conferences, the violations that get counted are the ones that leave checkable evidence. So reported numbers depend on how easy rule-breaking is to see as much as on how often it happens. An honest ICML-style comparison between the two policies would need a randomized study that measures compliance directly, not watermark counts.
Sources 3 notes
Hidden-instruction watermarks planted in PDFs flagged about 1% of reviews under ICML's no-LLM rule, leading to 497 desk rejections. The chairs acknowledge the method catches mainly careless uses and misses reviewers who removed or rewrote the watermark.
ICLR 2026 uses LLM detection tools only to triage papers for area chairs, who must find concrete evidence before sanctions are applied. This human gate protects against false positives by shifting costs from paper rejections to reviewer time.
Program chairs used imperfect detectors as one input for area chairs rather than automated filters, but desk-rejected papers with confirmed fabricated references as a tractable enforcement point. Multiple human review steps mitigated false positives.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- A Retrospective on the ICLR 2026 Review Process
- On Violations of LLM Review Policies
- ICLR 2026 Response to LLM-Generated Papers and Reviews
- Stop Automating Peer Review Without Rigorous Evaluation
- Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026
- Do LLMs Favor LLMs? Quantifying Interaction Effects in Peer Review
- Exploring the use of AI authors and reviewers at Agents4Science
- LLM-REVal: Can We Trust LLM Reviewers Yet?