Could you unmask an AI by its spelling habits alone — or is the real giveaway what words it picks, not how it spells them?
Do letter-level patterns contribute meaningfully to LLM model identification?
This explores whether you can tell which AI model wrote a text by looking at spelling-level and character-level habits, or whether the clues are somewhere else, such as word choice, style or meaning.
This explores whether letter-level and character-level patterns are part of what makes one AI model's writing recognizable from another's. The short answer is that the collection has no study that tests letter-level features directly. What it does have points the other way: the clues that identify a model seem to sit higher up, in word choice and meaning, not in spelling.
The most direct evidence comes from work showing that a classifier can tell five leading chat models apart with 97.1% accuracy using word distributions alone Can we identify which LLM wrote a given text?. The surprising part is what happens when the text gets scrambled. The clues survive when another model rewrites the text, translates it or summarizes it. Translation into another language would wipe out nearly every letter-level pattern, yet the model's 'accent' still comes through. That suggests the fingerprint is in what a model tends to say and how it frames things, not in the characters it uses.
A second line of work suggests why letters may be a weak signal in the first place. Researchers who treat LLMs as next-word probability machines predicted that tasks needing letter-by-letter handling, such as counting letters or reciting the alphabet backwards, would be unexpectedly hard. Experiments confirmed it Can we predict where language models will fail?. Models work in chunks of words, not individual letters, so they have little fine control at the letter level. That makes it an unlikely place for a stable, model-specific habit to form.
Research on human writing shows the same split. Even GPT-2 could identify human authors with 95% accuracy from style patterns, but it could not explain why those choices mattered Can language models truly understand literary style?. Spotting who wrote something turns out to be easy, and surface patterns are enough to do it. This raises a question the collection doesn't answer: would letter-level features add anything once word-level features are already this strong? Probably very little, but no study here tests it.
There is also a twist that may affect how long these fingerprints last. Work on AI narrowing human expression finds that models reflect skewed training data, and that people who write with them start adopting the model's framings Do large language models narrow human expression and thought?. If people pick up a model's word habits, the habits that identify a model could start showing up in human writing too. Telling 'which model wrote this' apart from 'which human learned to write like this model' could get harder over time.
Sources 4 notes
A classifier identifies five leading LLMs with 97.1% accuracy based on word distributions alone. These idiosyncrasies survive external rewriting, translation, and summarization, suggesting they reflect semantic patterns, not surface phrasing.
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.
GPT-2 achieves 95% accuracy identifying authorship through style patterns alone, but lacks the evaluative framework to explain why those stylistic choices carry meaning. Detection without interpretation remains cataloguing, not criticism.
LLMs mirror skewed slices of human experience shaped by training data regularities, and widespread reliance on identical models amplifies convergence. Co-writing studies show users unconsciously adopt model stances and framings.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Homogenizing Effect of Large Language Models on Human Expression and Thought
- Six misconceptions about large language models: A minimal model and diagnostic taxonomy
- The Widespread Adoption of Large Language Model-Assisted Writing Across Society
- Idiosyncrasies in Large Language Models
- Mapping the Increasing Use of LLMs in Scientific Papers
- Argument Collapse: LLMs Flatten Long-Form Public Debate
- Semantic Structure in Large Language Model Embeddings
- From Human to Machine Psychology: A Conceptual Framework for Understanding Well-Being in Large Language Models