Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing

Journals and conferences have begun to screen submitted manuscripts for text written using large language models (LLMs). The reliability of this screening rests on benchmark evaluations against a fixed set of LLM versions, while the versions in actual use keep changing. Here we quantify how this LLM turnover affects the screening of scientific manuscripts. We paired 4,000 pre-ChatGPT abstracts from the Proceedings of the National Academy of Sciences with their rewrites by 23 LLM versions from three vendors, released between June 2023 and August 2026. We then trained detectors under maintenance scenarios ranging from a detector retrained on every new version to one trained once and never updated. Detectors trained only on a vendor’s past versions can collapse at the boundaries between model generations: calibrated to falsely flag 1% of human-written abstracts, they catch above 99% of rewrites just before the sharpest boundary and 3.8% just after it. Detectors trained on later versions can also miss rewrites of earlier ones. Vocabulary differences between versions largely track where detection transfers and where it fails. In the two screening scenarios we simulated, screens covering all 23 versions either flagged one in eight human-written abstracts or missed one in three rewrites of the newest version. Indeed, a commercial detector missed most rewrites of the version just after the sharpest boundary while flagging almost no human-written abstracts. Research-integrity policy should therefore treat the benchmark accuracy of a detector as provisional, to be re-verified with every LLM release, including earlier versions.

Introduction. Large language models (LLMs) have entered the process of scientific research, from the search and synthesis of prior literature [1] to the drafting and editing of manuscripts [2, 3], the writing of peer review reports [4], and the preparation of grant proposals [5]. LLM assistance is already widespread, with estimates of LLM-modified content reaching up to 22% in computer science papers by September 2024 [6] and at least 13.5% of biomedical abstracts published in 2024, rising to about 20% for some countries and fields and about 40% for some of their combinations [7]. Analyses by publishers likewise report a rapid rise in submitted manuscripts and peer review reports containing text produced by an LLM [8]. Journals, conferences, and editorial organizations ask authors to disclose such use [9–11], and the estimates above suggest that the use has spread faster than its disclosure [12]. These LLMs accelerate scientific production and lower language barriers, while they weaken the signals, such as the complexity of the writing, that readers have used to judge scientific work [13]. How science should respond to this trade is debated across disciplines [13,14]. Journals and conferences face it one submission at a time, ahead of any resolution of that debate. Journals and conferences have begun to screen individual submissions with automated detectors that estimate how much of a text was written or edited with an LLM. A leading conference in computer science analyzed every submission to one of its tracks with a commercial detector and rejected 178 of them without review in 2026, which amounted to 18.4% of the submissions to that track [11]. Similarly, a journal published by the National Academy of Sciences lists the same product among the tools it uses to assess submissions [15]. Guidance for medical journal editors now devotes a dedicated section to the use of AI in publishing [9]. Such detectors usually express this estimate as a score and return a verdict once the score passes a threshold [11,16]. The two kinds of mistakes such a verdict can make fall on different researchers. When a manuscript written by its authors is judged to involve an LLM, the authors may carry the burden of contesting the verdict and a rejection of the manuscript. When a manuscript largely written with an LLM passes as human writing, and the use was neither disclosed nor permitted, the burden can fall on reviewers and editors who assume the text reflects its authors. In practice, deployed detectors

Related work. The reliability of screening scientific manuscripts rests on benchmarks assembled from the LLM versions available when a detector is evaluated [2, 3, 22–26]. Detectors can lose their accuracy on LLMs absent from their training data [18, 27–29]. This loss extends to successive LLM versions within a single LLM family [30]. In the scientific literature, the period over which a released LLM is actively used has shortened with each release year [31]. A detector in service can be asked to judge a version that it was not built against, and how to adapt a detector to newly arriving models is an active line of work [32–35]. How this LLM turnover affects the screening of scientific manuscripts remains to be understood under the conditions a screen faces in service, where retraining takes effort and flags of human-written text must stay rare. Here we quantify how detection of scientific abstracts rewritten with an LLM behaves as newer LLM versions arrive. We paired 4,000 abstracts published before the release of ChatGPT [36] with their rewrites by 23 LLM versions released by three vendors between June 2023 and August 2026. We then trained detectors under maintenance scenarios that mirror practice, from retraining on every new version to training a detector once and never updating it. A detector trained on the version it screens caught almost all of its rewrites, whereas the same detector could miss most rewrites of a later generation from the same vendor. These drops concentrated at a few transitions between versions. The vocabulary difference between the rewrites of two versions largely tracks where detection transferred and where it failed. The two screening scenarios we simulated divided their errors differently, one flagging more human-written abstracts and the other missing more rewrites. A commercial detector deployed for screening shows the same pattern, missing most rewrites of the version at the sharpest of these transitions while flagging almost no human-written abstracts. Together, our results indicate that the reliability of screening is not a fixed property of a detector, and that it has to be re-earned as LLM versions turn over.

Method. For a given text, Pangram returns one of three labels: ‘fully AI-generated’, ‘AI-assisted, but not fully AIgenerated’, and ‘fully human-written’. Of the 5,000 human-written calibration abstracts, it labels 4,999 as fully human-written and one as fully AI-generated. The deployed detector thus showed a false positive rate of 0.02% (1 of the 5,000 calibration abstracts; exact 95% confidence interval 0.0005% to 0.11%), far below the 1% at which our detectors operate. In contrast, the share of AI-rewritten abstracts that Pangram misses depends strongly on the LLM version (see Fig. 4(a)). We count an AI-rewritten abstract as missed only when Pangram labels it fully human-written, the choice most favorable to the detector. The missed share ranges from 6.5% (26 of 400) for GPT-5 to 79.8% (319 of 400) for Muse-Glimmer. The missed share is not significantly related to the release date, where the Spearman rank correlation between the two across the 23 versions is ρ = 0.25 (p = 0.25), and some late versions, such as GPT-5, are among the easiest to catch. Moreover, the missed share differs across the disciplinary domains of the original abstracts (see Fig. 4(b)). Across all AI-rewritten abstracts, Pangram misses 45.2% in the Health Sciences, 33.4% in the Life Sciences, 31.4% in the Physical Sciences, and 26.5% in the Social Sciences. The misses of the deployed detector thus fall unevenly across disciplinary domains, as the flags of human-written text have been reported to do [20,21]. The deployed detector thus errs on the same side as the pooled scenario, with even fewer false flags and even more misses. At its single fixed operating point, it flagged 1 of the 5,000 human-written abstracts, and its misses reach four in five rewrites for Muse-Glimmer.

4.3 Trained detectors Our detectors are binary classifiers that label a given abstract as human-written or AI-rewritten. We build each detector by fine-tuning RoBERTa-base, a pretrained language model with 125 million parameters [51], on pairs of human-written and AI-rewritten abstracts. Fine-tuning a pretrained classifier such as RoBERTa is a standard supervised approach in the detection of AI-generated text [22, 23, 25, 29, 37]. Each abstract enters the detector as at most its first 512 tokens. Training data come from the 3,955 of the 4,000 sampled papers whose rewrites were retained for all 23 versions. This restriction ensures that every detector is trained and evaluated on the same underlying papers. We split these papers into 80% for training and 20% for evaluation at the paper level, which guarantees that the human abstract and every rewrite of one paper fall on the same side of the split (3,164 training and 791 evaluation papers). Each detector is trained on the human-written abstract of each training paper and on one AI rewrite per training paper, which keeps the two classes balanced. We fine-tune for 4 epochs with a batch size of 8 and the AdamW optimizer at a learning rate of 2 × 10−5 with linear decay. The detectors differ only in the set of model versions whose rewrites supply their training texts; the architecture, the training procedure, and every hyperparameter are held fixed across all experiments, except for the pooled detectors trained on all of the available rewrites (Supplementary Section S8.2). We repeat every experiment with 10 random splits and report means and standard deviations across the repetitions.

4.4 Detection scenarios We consider four maintenance scenarios that mirror how a detector could be kept up to date in practice. Each scenario specifies which LLM versions’ rewrites train the detector that screens a target LLM version v. In all four scenarios, the training versions come from the same vendor as v. In the “current version” scenario, the detector is trained on rewrites from v itself; this idealizes a detector built with full access to the version being screened. In the “past and current versions” scenario, the detector is trained on rewrites from v and from all versions released before v; this represents a detector retrained immediately whenever a new version appears.

Discussion. The results for the maintenance scenarios highlight that detection can collapse when an institution’s training data lag behind the newest LLM versions. We now focus on what happens when the institution holds the rewrites of all 23 LLM versions. We consider two screening scenarios that use them at once. In the union scenario, the institution runs the 23 version-specific detectors side by side, each at its own threshold of 1% FPR, and flags an abstract when at least one detector fires. In the pooled scenario, the institution trains a single detector on the rewrites of all 23 LLM versions pooled together and sets its threshold in the same way as for the other detectors. For each scenario, we examine both the share of human-written abstracts it flags and the share of AI-rewritten abstracts it misses. The union scenario catches nearly every AI-rewritten abstract. Even for Muse-Glimmer, the union catches 98.6% of the AI-rewritten abstracts on average, and more for every other version. The cost falls on the human side. Each of the 23 detectors alone holds its false positive rate at 1%. The abstracts falsely flagged by different detectors only partly overlap, and the false flags therefore accumulate across the 23 detectors. Over the 10 random splits, the union flags 11.9% ± 1.2% of the held-out human-written abstracts, that is, of the 791 original abstracts per split that no detector saw in training. An institution that screens with the union would thus falsely flag roughly one in eight genuinely human abstracts. The pooled scenario rarely flags human-written abstracts. The single detector flags 0.8% ± 0.3% of the held-out human-written abstracts over the 10 random splits, close to the 1% at which its threshold is set. For most LLM versions, the pooled detector also catches almost every AI-rewritten abstract, with a median miss rate of 2.8% across the 23 versions. The misses concentrate on a few LLM versions. The miss rate reaches 33.8% ± 3.4% for Muse-Glimmer, followed by 8.8% for Qwen2.5 and 6.6% for GPT-5.6 Terra. The pooled detector trains on the same number of rewrites as every other detector in this study, one per training paper drawn from the 23 versions. The pooled detector misses one in three Muse-Glimmer rewrites even though its training data contain Muse-Glimmer rewrites. In practice, LLM versions arrive one after another, and a screen at any moment can contain only the versions released up to that moment. To simulate this growth, we sort the 23 LLM versions by release date and build, for each k from 1 to 23, a union of the first k version-specific detectors and a pooled detector trained on the rewrites of the first k versions. The screens with k = 23 are the union and pooled scenarios above, and a screen with a smaller k is the same screen at an earlier moment. Figure 3 shows, for each k, the share of held-out human-written abstracts flagged and the share of Muse-Glimmer rewrites missed, where Muse-Glimmer is the last released of the 23 LLM versions. In the union scenario (Fig. 3(a)), every added detector adds false flags, and the flagged share climbs from 1.4% at k = 1 to 11.9% at k = 23. The missed share of the Muse-Glimmer rewrites falls as k grows and reaches 17.8% at k = 22. At k = 23, when the detector trained on the Muse-Glimmer rewrites joins the union, the missed share falls to 1.4%. In the pooled scenario (Fig. 3(b)), the flagged share stays near 1% at every k. The missed share of the Muse-Glimmer rewrites stays above 50% for pools of 12 to 22 versions. The same holds at the GPT-5 boundary: trained on the 11 versions released before GPT-5, the pooled detector catches 26.7% of the GPT-5 rewrites, as few as the detector trained on OpenAI’s earlier versions alone (26.6%; Fig. 1).