How much peer review text shows signs of LLM modification?

The authors estimate that between 6.5% and 16.9% of text submitted as peer reviews to ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023 "could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates." The number is a population share for those four venues, not a count of reviews. The authors report no comparable change in Nature family journal reviews. Where they locate user behavior is in the circumstances: the fraction is higher in reviews that report lower confidence, were submitted close to the deadline, and come from reviewers "less likely to respond to author rebuttals."

The method is what the paper calls distributional GPT quantification. It starts from a stated problem: LLM detectors have "unstable performance," so instead of classifying each review and counting, the authors fit a maximum likelihood estimate of the mixture. Reviews known to be human-written supply one reference distribution, and reviews an LLM generates from the same review instructions and papers supply the other. The vocabulary is adjectives, because a word like "commendable" spikes in recent ICLR reviews and occurs more often in generated text, with technical keywords removed. The authors report that adverbs, verbs and nouns give similar results. They also claim the approach is "more than 10 million times" cheaper computationally than state-of-the-art detectors, with estimation error reduced by factors of 3.4 in-distribution and 4.6 out-of-distribution. Those are the authors' own comparisons.

Set against the nearest notes, this is the peer-review case of the population move in How fast did LLM writing adoption actually spread?, which measures four public-facing domains. Its reason for avoiding per-document judgment matches Can people reliably spot content made by AI?, though this excerpt tests no human judges. The corpus-level compression the authors describe, where generated text narrows "linguistic variation and epistemic diversity," is the aggregate form of the lexical gap in Can human judges detect measurable differences in AI text?. The reviewing setting also meets How much does rhetorical style shift AI review scores? from the other side: there, AI reviewers' scores move with rhetoric, while here human reviewers' text is partly LLM-modified. The excerpt does not connect the two.

The excerpt does not establish that reviewers wrote reviews with ChatGPT from scratch. The authors say so directly: "Our method does not constitute direct evidence that reviewers are using ChatGPT to write reviews from scratch." A reviewer who sketches bullet points and has an LLM flesh them out would also produce a high estimate. The estimate rests on one generator family; the authors report that GPT-3.5 data generalizes to GPT-4. The discussion states the share as "roughly 7-15% of sentences," a different unit from the abstract's share of text. The confidence, deadline and rebuttal links are associations, not tested causes. The implication is narrow: for four venues in 2023 and 2024 and one generator family, a corpus-level share of 6.5% to 16.9% is consistent with substantial LLM modification, and the corpus trend is a firmer claim than any label on a single review.