What specific phrases reveal which model wrote a response?

Beyond the headline classification accuracy, the paper isolates which concrete phrases and formatting habits a classifier — and a reader — can point to. Using TF-IDF over uni- and bi-grams of each model's responses, feeding a logistic regression N-way classifier on nothing but word and phrase frequency reaches "85.5% / 83.7% accuracy on chat APIs / instruct LLMs" — "close to" the accuracy of the full fine-tuned embedding classifier. Reading the resulting features back out, the authors report that "ChatGPT begins responses with 'certainly' and 'below is'," while "Claude usually references the original prompt using the phrases like 'according to the text' and 'based on the text.'" The same divergence shows up in markdown use: "ChatGPT tends to emphasize each key point within enumerations in bold and highlight a title with markdown headers, but Claude formats text with simple enumeration and bullet points."

The method is simple on purpose: treat each response as a TF-IDF "document," extract the words and phrases over-represented per model, and read off the highest-weighted features as that model's characteristic vocabulary and formatting defaults. Because this linear, interpretable method recovers nearly as much signal as the opaque fine-tuned classifier, the authors treat it as evidence that the idiosyncrasies are concrete, nameable habits — specific openers, specific hedge phrases, specific markdown conventions — rather than some diffuse statistical residue only a neural network can detect.

This gives a concrete face to Can we identify which LLM wrote a given text?: the word-level signal that survives rewriting and translation is, at least in part, these specific phrases and formatting tics, not only an abstract embedding-space pattern. It also complicates the style-versus-factuality framing in Can imitating ChatGPT fool evaluators into thinking models improved?: if style reduces to a short list of recurring openers and markdown habits, a model that imitates ChatGPT's "certainly" and bolded headers may acquire the surface signature cheaply — without that imitation closing the embedding-level, semantic-content gap the parent paper traces.

The excerpt reports this breakdown in detail only for ChatGPT and Claude ("Appendix B.3 provides characteristic phrases for other LLMs," not reproduced here), drawn from UltraChat-style instruction prompts against the specific 2024 model snapshots tested. It does not claim these exact phrases are stable across later model versions or different prompt distributions, only that TF-IDF recovers most of the five-way classification signal for the models tested. Whether "certainly" and bolded headers persist in subsequent ChatGPT releases, or whether other model pairs show equally legible fingerprints, is not shown here.