Can you actually tell AI chatbots apart just by the words they tend to use?
Which other LLM pairs show equally distinctive and readable fingerprints in their outputs?
This explores which language models, beyond any one pair, leave a recognizable 'signature' in what they write or how they behave, and whether someone can actually tell them apart. The short answer: the corpus doesn't rank specific model pairs by how distinctive they are, but it shows that model fingerprints are common and that they appear in more places than word choice.
This explores which language models, beyond any one pair, leave a recognizable 'signature' in what they write or how they behave, and whether someone can actually tell them apart. The corpus doesn't list matchups like 'model A vs. model B is the most distinctive pair.' What it does show is that being identifiable seems to be the normal state of today's models, not a quirk of a few. A simple classifier can tell five leading chat models apart with 97.1% accuracy using only the words they choose. The signal survives when the text is rewritten by another model, translated, or summarized, which suggests the fingerprint lies in what each model tends to say, not only in its phrasing Can we identify which LLM wrote a given text?.
Fingerprints also go beyond prose. When nine models took standard psychological questionnaires in Chinese and English, each produced a stable profile that repeated across runs. All nine also shared a common 'agreeable, stable' base layer that comes from alignment training. One detail stands out: the models differed in which questions they refused to answer, so what a model treats as 'not applicable to me' is part of its signature Do LLMs show reproducible psychological profiles when given standardized tests?. In practice, a fingerprint is a stack of layers: vocabulary, personality-style response patterns, and refusal boundaries.
The twist is that 'readable' depends on who is reading. Machines read these fingerprints easily. People mostly can't: readers with ML expertise couldn't reliably tell LLM-written research abstracts from human ones, and they tended to assume a human was involved Can readers tell LLM abstracts from human ones?. Models may be among the best readers. Training a model to recognize its own summaries made it prefer them more, in a roughly linear relationship. That is early evidence that self-recognition drives the self-preference bias that affects AI judges Do LLMs favor their own text because they recognize it?. Distinctive fingerprints are therefore more than a curiosity. When one model grades another, they become a source of bias.
Some model-specific differences show up only when you probe how a model behaves or look inside it. One framework tells fabrication from honest error by regenerating an answer several times. Made-up answers vary a lot from run to run, while honest mistakes come back the same each time. This is a behavioral fingerprint read from patterns, not from any single output Can we distinguish types of LLM falsehood by regeneration patterns?. Even safety tools respond to models differently. A cheap detector built from a model's internal activations caught more reward hacks than an LLM monitor in Kimi K3 but fewer in GLM 5.2 How do cheap vector detectors compare to expensive LLM monitors?. One caution: a single output at temperature zero is one fixed sample, not a reliable portrait of the model, so fingerprinting needs many samples Does setting temperature to zero actually make LLM outputs reliable?.
If you came looking for a ranked list of 'most fingerprintable pairs,' the corpus can't give you one. It does suggest a better question: not which pairs are distinct, but at which layer (words, values, refusals, failure patterns) two models differ, and whether the reader is a person, a classifier, or another model.
Sources 7 notes
A classifier identifies five leading LLMs with 97.1% accuracy based on word distributions alone. These idiosyncrasies survive external rewriting, translation, and summarization, suggesting they reflect semantic patterns, not surface phrasing.
Nine LLMs given seven psychological instruments in Chinese and English showed stable, model-specific response configurations reproducible across repeated administrations, while all models converged on a shared prosocial and stability-oriented pattern. Structured non-response patterns also distinguished models, suggesting the boundary of what each model treats as applicable is part of its behavioral signature.
Readers with ML expertise struggle to identify LLM-generated content reliably, tending to assume human involvement across all abstract types. However, LLM-edited abstracts received highest clarity ratings and were preferred 55% of the time when authorship was disclosed.
Fine-tuning LLMs to recognize their own summaries increased their preference for those summaries in a linear relationship, suggesting recognition capability drives self-preference bias. The authors present this as initial causal evidence, not proof.
Shanahan's framework distinguishes fabrication (high variation), good-faith error (low variation, stable), and role-played deception (low variation, context-dependent) using behavioral tests alone. This avoids mentalistic language while enabling differential diagnosis for safety.
Show all 7 sources
On DeepSWE, difference-of-means vectors caught 3.1% more hacks in Kimi K3 but 7.9% fewer in GLM 5.2 than LLM monitors at matched false positive rates. The method applies to existing forward passes, making it virtually free compared to running a separate monitor model.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- LLM or Human? Perceptions of Trust and Information Quality in Research Summaries
- LLM Evaluators Recognize and Favor Their Own Generations
- AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- Mapping the Increasing Use of LLMs in Scientific Papers
- Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling
- Your LLM, Your Style: Behavioral Mode Axes for LLM Behavioral Control
- Idiosyncrasies in Large Language Models