Could sorting AI answers by what they actually do, not the words they use, expose when a model is stuck in a rut?
Can functional similarity clustering reveal when models over-rely on specific themes?
This explores whether grouping a model's outputs or internals by what they do, rather than by the words they share, can show when a model keeps falling back on the same few themes, phrases, or answers.
This explores whether grouping a model's outputs or internals by what they do (not by the words they use) can show when it keeps falling back on the same few themes. Note up front that the collection has no paper testing 'functional similarity clustering' as a named method. It does have several studies that use grouping or classification to catch exactly this kind of over-reliance, and together they suggest the idea works. They also show where it can mislead you.
The clearest case of over-reliance comes from strategy advice. Across 15,000 simulations, six LLMs recommended the same side of every strategic trade-off. Changing the industry barely moved them, but changing the order of the options moved them more Do LLMs consistently favor the same strategic choices regardless of context?. The models were recombining fashionable vocabulary instead of analyzing the situation. That is what clustering by function would catch: on the surface the answers look varied and fitted to each context, but grouped by what they actually recommend, they collapse into one cluster. Something similar shows up at the level of phrasing. A simple classifier using only habitual openers like 'certainly' or 'according to the text' can tell which model wrote a response 85.5% of the time What specific phrases reveal which model wrote a response?. Habits that strong are measurable, and grouping outputs makes them visible.
Clustering also works inside the model's reasoning. When reasoning chains are pruned token by token, the tokens fall into six functional categories, and models consistently keep the symbolic-computation tokens while dropping filler Which tokens in reasoning chains actually matter most?. Grouping by function shows what the model actually leans on, which the raw text can hide. Routing systems apply the same idea at larger scale. Avengers-Pro groups incoming queries into semantic clusters and sends each one to whichever model does best on it Can routing beat building one better model?. That only works because models have uneven, cluster-specific strengths and weak spots, so the clusters double as a map of where each model is strong or weak.
The caveat is that over-reliance doesn't always show up where you'd look for it. Two models can score identically and still have very different internal organization, and one of them can be fragile in ways standard evaluation won't reveal Can models be smart without organized internal structure?. Inside the network, a handful of huge activations act as hidden bias terms that pull attention toward specific tokens no matter what the input is Do hidden massive activations act as attention bias terms?. That is a built-in fixation, and clustering outputs would never surface it. Training can also plant tendencies through data that has nothing to do with the trait Can language models transmit hidden behavioral traits through unrelated data?, which means grouping by topic could miss the source entirely.
One more twist: what counts as a 'theme' depends on the domain. Preference tuning makes code output less varied but creative writing more varied Does preference tuning always reduce diversity the same way?. So a tight cluster of code answers may just mean the model is converging on correct solutions, while the same tightness in strategy advice signals a rut. Clustering can tell you that a model keeps returning to the same place. Whether that's a problem depends on whether the task has one right answer.
Sources 8 notes
Across 15,000 simulations, six LLMs recommended the same strategic choice in every tension tested. Industry context shifted bias only 11%, while option order—a framing artifact—shifted results 19%, revealing that models recombine trend-coded vocabulary rather than analyze context.
TF-IDF analysis reveals ChatGPT favors openers like 'certainly' and 'below is' plus bolded headers, while Claude prefers 'according to the text' and plain bullet points. A linear classifier using only these phrases reaches 85.5% accuracy, nearly matching opaque fine-tuned models.
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
Avengers-Pro achieves 7% higher accuracy than GPT-5-medium by routing queries to optimal models per semantic cluster, or matches its performance at 27% lower cost. Ten 7B models with routing previously surpassed GPT-4.1 and 4.5, suggesting selection is a stronger lever than scaling.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Show all 8 sources
A very small number of input-agnostic activations with values up to 100,000× larger than others act as indispensable implicit bias terms and concentrate attention probability onto specific tokens. This phenomenon appears across model sizes and Vision Transformers.
Research demonstrates that behavioral traits propagate between models via filtered data bearing no semantic relationship to the trait. The effect is model-specific, fails across different architectures, and persists despite rigorous filtering—indicating the mechanism embeds statistical signatures rather than semantic content.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
- Do LLMs Encode Functional Importance of Reasoning Tokens?
- Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
- Massive Activations in Large Language Models
- Evaluating the Diversity and Quality of LLM Generated Content