Can we identify which LLM wrote a given text?

The authors construct a synthetic classification task: given a single text output, predict which LLM produced it. Fine-tuning a text-embedding model (an LLM2vec backbone, LoRA-tuned) on outputs from ChatGPT, Claude, Grok, Gemini and DeepSeek reaches "97.1% accuracy on the held-out validation data" in this five-way task, against a "20.0% chance-level guess." The signal is not limited to cross-vendor contrasts: fine-tuning on four sizes within the Qwen-2.5 family alone still reaches "a non-trivial 59.8% accuracy," and a 10-way classification mixing five chat APIs and five instruct models (including Phi-4) reaches 92.2%. Classification holds up even when prompt length and format are held constant, and stays "non-trivial" from only the first few tokens of a response.

The authors trace the signal to word-level distributions rather than surface typography. Letter-level shuffling of a response collapses accuracy to "49%-56%," near chance, and the underlying letter-frequency distributions look "almost identical" across models — letters contribute "minimally." Word-level signal, by contrast, survives external rewriting, translation, and summarization by another LLM, which the authors read as evidence the idiosyncrasies are "encoded in the semantic content," not just surface phrasing. A simple TF-IDF logistic regression over uni- and bi-grams gets most of the way there (85.5% on chat APIs, versus 95.1% for the fine-tuned embedding classifier), surfacing literal tics: ChatGPT favors openers like "certainly" and "below is" and formats enumerations with bold text and markdown headers, while Claude favors phrases like "according to the text" and "based on the text" and sticks to plain enumeration and bullet points.

This is a different detection problem than Can humans detect AI text if machines can measure it? and the lexical-diversity work it draws on: those findings classify human versus AI text; this one classifies AI versus AI, across five named commercial systems at once. The common thread is the same measurable-but-hidden structure — statistical regularities invisible to an ordinary reader but legible to a classifier — now shown to operate between models, not just between humans and models. It also sharpens Can imitating ChatGPT fool evaluators into thinking models improved?: if word-level style is this distinctive and this robust to paraphrase, style is not a thin veneer that imitation training erases by accident — it is a durable signature a model carries even through an external LLM's rewrite.

The study measures distinguishability among a handful of specific 2024-era chat, instruct and base models on UltraChat- and FineWeb-sourced prompts; it does not establish that idiosyncrasies are universal across all LLMs, architectures (the authors flag state-space and diffusion-based language models as untested), or future model versions, nor does it explain mechanistically how training produces these signatures — the authors list the causal question as open for future work. What it does establish is narrower but still consequential: today's leading chat models are reliably fingerprintable by word choice alone, which bears on provenance tracing, measuring model similarity, and detecting contamination from one model's synthetic data in another's training pipeline.