Can encoder models match human conceptual structure better than larger language models?
This reads the question as: does scaling up a language model actually buy you human-like conceptual structure — or can smaller/differently-built models (like encoders) capture meaning that bigger LLMs miss?
This explores whether bigger language models genuinely capture how humans organize concepts, or whether size mostly buys statistical fluency. Worth flagging up front: the corpus doesn't contain a head-to-head benchmark of encoder models against larger LLMs on conceptual structure — so the direct comparison your question asks for isn't settled here. What the collection does have is a strong, repeated finding that scale alone does not deliver human conceptual structure, which reframes the question in a useful way: the issue isn't just 'encoder vs. LLM,' it's that statistical learning of any size tends to track surface form rather than meaning.
The sharpest evidence is that even top-tier large models systematically prefer the more textually frequent phrasing over a semantically identical rare paraphrase, across math, translation, and commonsense tasks Do language models really understand meaning or just surface frequency?. That points to models tracking pretraining mass, not meaning-recognition — and bigger doesn't fix it. The same pattern shows up in grammar: large models like Llama3-70b consistently misidentify embedded clauses and complex nominals, with errors that worsen predictably as syntactic depth increases Why do large language models fail at complex linguistic tasks?. So scale captures surface regularities but not the recursive structure humans use.
There's also a deeper structural diagnosis: 'potemkin understanding,' where a model explains a concept correctly, fails to apply it, and even recognizes its own failure — a triple incompatible with human cognition, suggesting explanation and execution run on disconnected pathways Can LLMs understand concepts they cannot apply?. And on inference specifically, LLMs predict entailment based on whether a hypothesis was attested in training rather than whether the premise supports it Do LLMs predict entailment based on what they memorized?. These are not gaps a few more parameters close.
Where scale does seem to matter is a representational-capacity threshold: smaller models plateau on argument-scheme classification while only larger ones cross meaningful accuracy, hinting that some conceptual tasks genuinely need representational room Can large language models classify argument schemes reliably?. But architecture, not just size, drives this — deep-and-thin models compose abstract concepts through layers better than wide ones at the same parameter count Does depth matter more than width for tiny language models?. That's the closest the corpus comes to your intuition: how a model is built can beat how big it is for conceptual composition.
The quietly provocative thread underneath all this: one note argues LLMs operationalize Saussure's 'langue' — they learn meaning as purely relational structure compressed from text, with no external referents Can language models learn meaning without engaging the world?. If meaning really is relational, then a model that compresses relational structure efficiently might match human conceptual organization regardless of size — which is exactly the case for asking whether a leaner encoder could rival a giant decoder. The corpus suggests the right question isn't 'is it bigger?' but 'does its architecture compress conceptual relationships, or just frequency?'
Sources 7 notes
LLMs show consistent preference for higher-frequency surface forms over semantically equivalent rare paraphrases across math, machine translation, commonsense reasoning, and tool calling. This suggests models track statistical mass from pretraining rather than meaning-recognition as their primary mechanism.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
McKenna et al. (2023) identified attestation bias: LLMs predict entailment based on whether the hypothesis appears in training data, not whether the premise actually supports it. Random premise experiments show models maintain high entailment predictions when hypotheses are attested, proving they respond to memorized propositions rather than premise-hypothesis relationships.
Zero-shot prompting fails uniformly across models. Few-shot with scheme descriptions helps, but only larger models exceed F1 0.55, with Claude reaching 0.65. Smaller models plateau around 0.53, suggesting a representational capacity threshold.
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.
Research shows LLMs learn culturally situated discourse patterns by compressing relational structure from text, demonstrating that fluent language generation requires no external referents or embodied grounding.