Does word frequency correlate with semantic abstraction?
Explores whether LLMs' preference for high-frequency language also pulls them toward more abstract, general meanings—and whether this shapes how they handle expert knowledge.
The companion paper "LLMs are Frequency Pattern Learners in NLI" measured WordNet hyponym-hypernym pairs (e.g., "whisper" → "talk") and found hypernyms — the more general concepts — occur more frequently than their hyponyms. Hypernym frequency exceeds hyponym frequency systematically. Combined with Adam's Law's finding that LLMs prefer high-frequency phrasing across tasks, this yields a non-obvious correlation: when an LLM prefers a higher-frequency paraphrase, it is also preferring a more abstract paraphrase. Frequency is not just a register property; it is also a generalization-gradient property.
This sharpens Does fine-tuning on NLI teach inference or amplify shortcuts?. Fine-tuning on NLI does not just amplify a frequency preference — it amplifies a preference for inferences that move from specific to general (the upward semantic-entailment direction WordNet calls generalization). The model is not learning entailment; it is learning the surface signal of generalization, which happens to correlate with entailment in the kinds of sentences NLI corpora contain.
The implication for the Knowledge Custodian frame is uncomfortable. Expert knowledge lives in the hyponyms — the specific cases, the qualifying conditions, the rare technical terms. When LLMs prefer high-frequency paraphrases at parse time, they drift up the generalization gradient: away from the specific cases that distinguish an expert from a competent generalist, and toward the abstract concepts that any reasonably literate reader could state. This is the same direction Do LLMs compress concepts more aggressively than humans do? identifies in concept representations. The compression is not random — it has a direction, and the direction is from specific toward abstract, from rare toward common, from distinctive toward median. An expert who prompts in their own register is asking the model to comprehend in a region the model is bad at; the model's "help" is to gently flatten the request back toward the register where it performs well, which is exactly the register that erases what the expert was trying to say.
Inquiring lines that read this note 50
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does AI writing assistance affect reader perception of writer authenticity? What determines when personalization helps versus harms model behavior?- What level of abstraction makes interest journeys feel personally relevant to users?
- Why does semantic memory abstraction outperform raw episodic recall for personalization?
- Can meaning-level metrics like Semantic Entropy avoid length bias?
- Does bias measurement through subjective word lexicons match expert judgment of slop?
- Can statistical learning from language alone capture all aspects of cultural competence?
- Does generalization frequency explain why models favor upward semantic movement?
- Do LLMs learn linguistic generalizations or just surface-level frequency patterns?
- What other semantic relations benefit from explicit surface markers in text?
- What semantic classifier design avoids lexical variation without genuine conceptual distinctness?
- Does selective suppression of linguistic relations enable human meaning-making?
- How does semantic entanglement interact with personality dimension shifts during finetuning?
- Why does assembling knowledge from multiple sources build better understanding than reading summaries?
- Does RLHF training suppress exploratory and qualifying language?
- Can preference tuning or RLHF reduce epistemic diversity alongside lexical diversity?
- How does RLHF training encode sociocognitive biases against non-native English speakers?
- Can knowledge density explain why LLM writing feels coherent but fatiguing?
- How much semantic meaning survives when LLMs paraphrase poetry and literary text?
- How do spoken expert discussions shape what LLMs cannot learn?
- Do metaphors work by decoupling meaning from linguistic associations?
- Do all semantic steering effects follow predictable patterns based on feature alignment?
- Do LLMs learn abstract grammar or culturally situated discourse patterns instead?
- Can humans suppress frequency bias through attention and intention?
- Why do users prefer community sources over encyclopedic references?
- Do newer LLM generations create worse detector bias through increased linguistic divergence?
- Do vocabulary drift patterns appear across different document types and domains?
- Why do cognitive metaphors change based on available technology?
- What does ascetical perception training accomplish that media literacy cannot?
- Why does semantic diversity matter more than surface lexical diversity?
- Does semantic diversity in output space compete with reward-component diversity?
- Why do frequent words rank higher in taxonomic abstraction hierarchies?
- How does co-occurrence statistics alone produce hierarchical concept organization?
- What role does recursive generalization play in semantic drift during learning?
- Do LLMs track surface wording more than semantic meaning in moral judgment?
- Does LLM vocabulary become the cultural lexicon for how we think?
- Do LLMs trained on Wikipedia content count as indirect readership?
Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does fine-tuning on NLI teach inference or amplify shortcuts?
When LLMs are fine-tuned on natural language inference datasets, do they learn genuine reasoning abilities or become better at exploiting statistical patterns in the training data? Understanding this distinction matters for assessing model capabilities.
frequency bias amplification has a directional gradient
-
Do LLMs compress concepts more aggressively than humans do?
Do language models prioritize statistical compression over semantic nuance when forming conceptual representations, and how does this differ from human category formation? This matters because it may explain why LLMs fail at tasks requiring fine-grained distinctions.
same compression dynamic at representational level
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- LLMs are Frequency Pattern Learners in Natural Language Inference
- Adam's Law: Textual Frequency Law on Large Language Models
- From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
- Semantic Structure in Large Language Model Embeddings
- Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
- AI-Augmented Strategic Decision-Making Under Time Constraints: An Experimental Study on Mental Representations and Strategic Foresight
- Six misconceptions about large language models: A minimal model and diagnostic taxonomy
- LLM Augmentations to support Analytical Reasoning over Multiple Documents
Original note title
frequency tracks the generalization gradient — hypernyms outnumber hyponyms so frequent phrasing is also more abstract phrasing