Idiosyncrasies in Large Language Models

In this work, we unveil and study idiosyncrasies in Large Language Models (LLMs) – unique patterns in their outputs that can be used to distinguish the models. To do so, we consider a simple classification task: given a particular text output, the objective is to predict the source LLM that generates the text. We evaluate this synthetic task across various groups of LLMs and find that simply fine-tuning text embedding models on LLMgenerated texts yields excellent classification accuracy. Notably, we achieve 97.1% accuracy on held-out validation data in the five-way classification problem involving ChatGPT, Claude, Grok, Gemini, and DeepSeek. Our further investigation reveals that these idiosyncrasies are rooted in word-level distributions. These patterns persist even when the texts are rewritten, translated, or summarized by an external LLM, suggesting that they are also encoded in the semantic content. Additionally, we leverage LLM as judges to generate detailed, open-ended descriptions of each model’s idiosyncrasies. Finally, we discuss the broader implications of our findings, including training on synthetic data, inferring model similarity, and robust evaluation of LLMs. Code is available at github.com/locuslab/llm-idiosyncrasies.

Introduction. As the adoption of generative models such as LLMs accelerates, it becomes increasingly important to understand the origin and provenance of such generated content. While a great deal of past work has focused on the classification of human-written and AI-written content (Krishna et al., 2023; Mitchell et al., 2023; Sadasivan et al., 2025), there has been little work on classifying between content generated by different LLMs, either between the outputs of entirely different models or between those of different variants of the same model family. If possible, the ability to distinguish between source models in this manner would be valuable for a number of applications: it could shed light on the relative uptake of different LLMs, beyond what is reported by individual companies, and on the nature of data used to build different models. Additionally, it could offer insights into what features of generated text are most “unique” to each LLM.

In this paper, we investigate whether LLMs exhibit idiosyncrasies that enable their outputs to be reliably differentiated. Inspired by recent studies on dataset bias in computer vision (Liu & He, 2025; Zeng et al., 2024), which showed that images from different large-scale datasets can be accurately distinguished by standard neural networks, we consider a similar synthetic classification task to assess the separability of responses generated between different LLMs. Specifically, we sample a large number of text outputs from each LLM using the same set of prompts and then train a classifier to recognize which model generates a specific text. Figure 1 provides an overview of our framework. The illustrated example on ChatGPT, Claude, Grok, Gemini, and DeepSeek presents a five-way classification problem.

We find that a classifier based upon simple fine-tuning text embedding models on LLM outputs is able to achieve remarkably high accuracy on this task. This indicates the clear presence of idiosyncrasies in LLMs. The observation is highly robust over a large variety of LLM combinations. For instance, trained on the combined set of texts from ChatGPT, Claude, Grok, Gemini, and DeepSeek, a model can achieve 97.1% classification accuracy on the held-out validation data, compared to a 20.0% chance-level guess. Within the same model family, we obtain a non-trivial 59.8% accuracy across 4 model sizes in Qwen-2.5 series (Qwen et al., 2024). Further, we observe strong out-of-distribution generalization of these classifiers when tested on responses from prompts outside the training distribution.

We observe several interesting properties of this task. When controlling the length and format of outputs through prompt instructions, we still obtain high classification accuracy. Furthermore, for post-trained LLMs, the classifier demonstrates non-trivial accuracy even with only the first few tokens of the generated text. However, when classifying generations

Related work. Dataset classification. Torralba & Efros (2011) introduced the “Name That Dataset” experiment a decade ago to highlight the bias present in visual datasets at that time. Recently, Liu & He (2025) revisited this problem (termed dataset classification) and found that current large-scale, supposedly more diverse visual datasets are still very biased. Zeng et al. (2024) further identified structural and semantic components in images as key contributors to these biases. You et al. (2024) and Mansour & Heckel (2024) applied the dataset classification framework to study bias in synthetic images and LLM pretraining datasets, respectively. While the synthetic task shown in Figure 1 is conceptually similar to dataset classification, we focus not on training datasets but on the distinctive characteristics inherent to LLMs. Shortly after our initial arXiv preprint, Suzuki et al. (2025) demonstrated that subtle variations in the training process can result in distinguishable outputs across LLMs.

Human vs. machine-generated texts. Many prior works have studied the problem of determining whether a text is authored by a human or an AI system (Mitchell et al., 2023; Wu et al., 2023; Cai et al., 2025). Model-free approaches typically use linguistic properties such as n-gram frequencies (Badaskar et al., 2008; OpenAI, 2019), entropy (Lavergne et al., 2008; Gehrmann et al., 2019) or negative probability curvature (Mitchell et al., 2023; Bao et al., 2024). Other works leverage neural network features to perform this task, such as fine-tuning BERT (Uchendu et al., 2021; Ippolito et al., 2020). Russell et al. (2025) found that experienced chatbot users are good at distinguishing between AI- and human-written articles. Neural authorship attribution (Uchendu et al., 2020; Antoun et al., 2023; Huang et al., 2024) seeks not only to identify machine-generated text but also to attribute it to specific text generators. In this work, we focus on the distinguishability between LLMs rather than between AI vs. human.

Characteristics of machine-generated texts. Beyond detection, it is also important to understand the distinctive properties of machine-generated texts. N-gram frequencies have long served as a basic signal for such purpose. McGovern et al. (2024) showed that LLM outputs contain unique lexical and syntactic fingerprints that distinguish them from human writing. Other works have examined stylometric features in AI-generated content (Zaitsu & Jin, 2023). Most recently, Chakrabarty et al. (2024) studied the human-AI alignment in the writing process, finding that professional writers can effectively identify and edit undesirable idiosyncrasies common in LLM-generated text.

Method. Large Language Models (LLMs) share several core characteristics. The majority of them are based on the Transformer architecture (Vaswani et al., 2017), which is shared by all models we consider in this paper. Second, they are trained using an auto-regressive objective (Radford et al., 2019), where they predict the next token in a sequence based on preceding context. Lastly, their training datasets significantly overlap, often incorporating vast and diverse sources such as Common Crawl, Wikipedia, and Stack Overflow. Given these similarities, it is natural to ask: do LLMs speak in the same way? If not, how can we effectively measure the degree of their differences?

To address these questions, we construct a synthetic task focused on classifying outputs from different LLMs. Consider N LLMs, denoted as f1, . . . , fN, where each fi takes an input prompt p and outputs a text completion o. For a given dataset D of prompts, the outputs produced by each LLM fi are denoted as Oi. We approach this problem with a straightforward setup. For N output sets Oi, we formulate a N-way classification task, where the objective is to predict which LLM produced each output. If outputs of different LLMs were drawn from the same distribution, classification accuracy would not be better than random chance. Thus, we use the classification performance of this synthetic task as a measure of idiosyncrasies in LLMs.

Our task is formulated as a sequence classification problem, for which fine-tuning BERT-style models is a common approach (Sun et al., 2019). In this work, we fine-tune a more recent and competitive sequence embedding model based on decoder-only Transformers: LLM2vec (BehnamGhader et al., 2024). We attach a N-way classification head to the extracted embeddings and use LoRA-based fine-tuning (Hu et al., 2022) to the model weights. Input sequences are truncated to a maximum length of 512 tokens. We report the classification accuracy on a held-out validation set. Additional training details are provided in Appendix A.2.

We describe the LLMs we use to generate the output datasets O1,··· ,N. For a comprehensive and fair comparison across model families, we categorize three groups of LLMs:

1. Chat APIs (“chat”): This category includes state-ofthe-art LLMs that are primarily accessible via APIs. We consider GPT-4o (OpenAI, 2024), Claude-3.5- Sonnet (Anthropic, 2024), Grok-2 (xAI, 2024), Gemini- 1.5-Pro (Google, 2024), and DeepSeek-V3 (DeepSeek- AI, 2024). For simplicity, we refer to them as ChatGPT, Claude, Grok, Gemini and DeepSeek. Their architectures and weights remain proprietary and undisclosed, with the exception of DeepSeek.

2. Instruct LLMs (“instruct”): These models are trained to generate high-quality responses from human instructions. We consider four LLMs of similar sizes across different families: Llama3.1-8b (Dubey et al., 2024), Gemma2-9b (Riviere et al., 2024), Qwen2.5-7b (Qwen et al., 2024) and Mistral-v3-7b (Jiang et al., 2023). We will refer to them as Llama, Gemma, Qwen and Mistral. 3. Base LLMs (“base”): These are base versions of instruct LLMs. They are obtained by pretraining on extensive text corpora without any post-training stage.

Throughout the paper, we refer to these three categories as “chat”, “instruct”, and “base” respectively. For a given prompt dataset, we collect 11K text sequences, splitting them into 10K and 1K as training and validation sets, respectively. The same split is used across all LLMs. For chat APIs and instruct LLMs, we generate outputs from UltraChat (Ding et al., 2023), a diverse dialogue and instruction dataset. For base LLMs, we synthesize new texts using prompts from FineWeb (Penedo et al., 2024), a highquality LLM pretraining dataset. More details on response generation are in Appendix A.1.

Discussion. Text shuffling. To decouple the effects of words and letters from other factors, we remove special characters in LLM-generated responses, such as punctuations, markdown elements, and excessive white spaces. This ensures that each response consists solely of words separated by a white space. Additionally, we apply two shuffling strategies to the preprocessed text: word-level and letter-level shuffling. These transformations disrupt the natural order and force the classifier to learn patterns from raw text statistics. Table 8 presents the classification results.

In contrast, shuffling at the letter level results in a substantial drop in accuracy (49%-56%), approaching chance-level performance. This indicates that letter-level statistics alone are not sufficient for predicting LLM identities. To qualitatively visualize distinctions in letter distributions across models, Figure 5 (right) shows the frequency distribution of letters in responses generated by chat APIs. Different LLMs share almost identical letter distributions, indicating that letters contribute minimally to idiosyncrasies in LLMs.

Characteristic phrases. We use Term Frequency-Inverse Document Frequency (TF-IDF) to highlight characteristic phrases inside LLM-generated responses that reflect each model’s word choices. Formally, we treat each LLM response as a document and then extract TF-IDF features on all uni-gram and bi-gram words. We then train a N-way logistic regression model to predict the origin of responses on the extracted features. This simple linear classifier achieves 85.5% / 83.7% accuracy on chat APIs / instruct LLMs, close to 95.1% / 93.8% achieved with fine-tuning embedding models on responses without special characters (Table 8).

Figure 4a illustrates these characteristic phrases with example responses from ChatGPT and Claude. While ChatGPT begins responses with “certainly” and “below is”, Claude usually references the original prompt using the phrases like “according to the text” and “based on the text”. Moreover, Figure 7 reveals noticeable differences in the distribution of first word choices among chat APIs. Appendix B.3 provides characteristic phrases for other LLMs.

We seek to understand how each LLM formats their responses, particularly in markdown. To this end, we focus on common markdown elements used by LLMs: (1) bold text, (2) italic text, (3) header, (4) enumeration, (5) bullet point, responses without bold texts or headers. On the contrary, other LLMs exhibit lower values at zero and thus decorate text with these formatting elements more often.

Figure 4b visualizes how ChatGPT and Claude structure their responses in markdown. Interestingly, ChatGPT tends to emphasize each key point within enumerations in bold and highlight a title with markdown headers, but Claude formats text with simple enumeration and bullet points. More analysis for other models can be found in Appendix B.4.

Rewriting. One potential reason for the high classification accuracy is the unique writing style (e.g., word choice, sentence structure) of each LLM. To isolate this factor, we leverage another LLM (e.g., GPT-4o mini) to rewrite LLM responses. Our rewriting approaches include (see Appendix C for example responses after rewriting):

Open-ended language analysis. In this part, we focus on studying the semantic difference in responses generated by LLMs. We employ another LLM (e.g., ChatGPT) as a judge to provide open-ended, descriptive characterizations for each LLM’s outputs. The results with other LLM judges for our language analysis are available in Appendix B.6.

Conclusion. We demonstrate the presence of idiosyncrasies in Large Language Models (LLMs) and investigate a synthetic task designed to quantify their extent. We find that simply finetuning pretrained text embedding models on LLM outputs leads to exceedingly high accuracy in predicting the origins of the text. This phenomenon persists across diverse prompt datasets, LLM combinations, and many other settings. We also pinpoint concrete forms of these idiosyncrasies within LLMs. We hope our work encourages further research into understanding idiosyncrasies in LLMs.

We conclude by outlining several directions for future work:

• It is worth investigating whether our observations generalize to LLMs beyond the Transformer architecture, such as state space models (Gu & Dao, 2024) and diffusionbased language models (Nie et al., 2025).

• Understanding how the training process results in these idiosyncrasies remains an important open question.

• Third, our setup does not consider scenarios where the list of source LLMs could be large and even unknown beforehand, which require further investigation. Our initial results suggest it is promising: a 10-way classification problems involving 5 chat APIs and 5 instruct LLMs including Phi-4 achieves 92.2% accuracy.

• It is interesting to examine how these idiosyncrasies relate to model distillation (Hinton et al., 2015), a technique that has become increasingly prevalent in practice.