Training an AI-text detector on many old chatbot versions makes it steadier — but does it still catch the newest ones?

This explores whether training one AI-text detector on writing from many LLM versions makes it more robust, especially against newer models. The short answer from the collection: pooling makes a detector steadier, but it doesn't make it future-proof. What you gain is fewer false accusations. What you pay is missing the newest models.

The clearest test compared two ways of screening scientific abstracts rewritten by 23 different LLM versions Can one detector catch all LLM versions at once?. One way runs a separate detector for each version and flags a text if any of them fires. That catches 98.6% of AI rewrites, but it also wrongly flags about one in eight genuine human abstracts (11.9%). The other way is the pooled approach: one detector trained on all versions together. It keeps false flags near 1%, but it misses a third (33.8%) of rewrites from the newest version. So pooling doesn't remove the weakness. It moves it from wrongly accusing human writers to letting recent AI text through, and the people who pay for that are whoever relies on the detector to catch current tools.

The reason pooling can't fully solve this shows up in a companion finding Why do AI detectors suddenly fail when models update?. Detectors trained on past versions catch over 99% of rewrites right up to a new model release, then fall to 3.8% just after it. Detection doesn't wear down gradually. It falls off a cliff at each generation boundary, because each new version shifts its word choices enough that the learned patterns stop matching. No amount of pooling covers a version that didn't exist when the training data was gathered. Pooled data teaches the detector what LLMs have sounded like, not what the next one will sound like.

One mechanism in the collection may explain why each version leaves such a distinct and fragile fingerprint, though this is a connection across notes rather than something the detection papers test directly. RL post-training tends to lock onto one dominant output style from pretraining and suppress the others, and which style wins can depend on model scale rather than quality Does RL training collapse format diversity in pretrained models?. If each model release settles on its own house style in a way that's somewhat arbitrary, version-to-version changes in word choice would be hard to predict, which is exactly what defeats a detector trained on the past.

The collection doesn't yet test fixes such as retraining continuously after each release, or detectors that rely on signals other than word choice. Here's what it does establish: a detector's accuracy is tied to the model generation it was trained on, and pooling across versions mainly decides which kind of mistake you'd rather make.