Do LLMs compress concepts more aggressively than humans do?

An information-theoretic framework drawing from Rate-Distortion Theory and the Information Bottleneck principle quantitatively compares how LLMs and humans balance compression against semantic fidelity. The comparison uses seminal cognitive psychology datasets (Rosch typicality ratings, McCloskey & Glucksberg category membership) as human baselines.

Where they converge: LLM-derived clusters significantly align with human-defined conceptual categories. Broad category structure — that robins and sparrows are birds, that chairs and tables are furniture — is captured reliably. Some encoder models achieve surprisingly strong alignment with human categorical structure, sometimes outperforming much larger models, suggesting factors beyond scale matter for human-like abstraction.

Where they diverge: LLMs fail to capture fine-grained semantic distinctions crucial for human understanding. Correlations between LLM item-to-category-label similarities and human typicality judgments are generally modest. Items humans perceive as highly typical (robin as prototypical bird) are not consistently represented as substantially more similar to the category label embedding than atypical items (penguin as bird).

The fundamental divergence in strategy: LLMs exhibit a strong bias toward aggressive statistical compression — maximally reducing representational complexity. Human conceptual systems prioritize adaptive nuance and contextual richness, even at the cost of lower compressional efficiency. Humans preserve distinctions that matter for situated action (the difference between a robin and a penguin matters for different reasons in different contexts), while LLMs collapse these distinctions in favor of statistical regularity.

This finding refines the debate around Can text-trained models compress images better than specialized tools?. LLMs are excellent compressors — but compression is not comprehension. The compression strategy differs fundamentally from how humans organize concepts. Human categorization isn't optimized for compression; it's optimized for adaptive action in context. The "cost" of preserving nuance (lower compressional efficiency) is paid because nuance has survival value.

This connects to Does semantic grounding in language models come in degrees? by providing the information-theoretic mechanism behind weak causal grounding: if LLMs compress away the fine-grained distinctions that ground causal reasoning (the specific weight, texture, and behavior of a robin versus a penguin), causal grounding requires exactly the nuance that compression eliminates.

The literary language dimension: Literary language is where the compression-nuance divergence becomes most consequential. Literary prose and poetry are maximally nuanced — every word choice is deliberate, ambiguity is preserved intentionally, connotation carries as much weight as denotation. LLM compression preserves denotation (what a text literally says) but destroys connotation (what a text means through association, implication, and resonance). This is testable: having LLMs paraphrase poetry and measuring which dimensions of meaning survive versus collapse would quantify the gap between understanding what a text says and understanding what a text means. Since Can LLMs truly understand literary meaning or just mechanics?, the compression-nuance trade-off is one of four converging mechanisms that explain the mechanics-meaning gap.