LLMs learn language only from text, with no body and no senses — so how do they form anything like real concepts?
How do LLMs build concepts without embodied experience or mental models?
This explores how language models end up with something that works like concepts when they learn only from text, with no body, no senses, and possibly no internal picture of the world, and whether that premise even holds.
This explores how LLMs come to handle concepts when all they learn from is text, with no body and no direct contact with the world. The corpus pushes back on half of the question. 'Without embodied experience' holds up. 'Without mental models' is more contested than it sounds.
Start with the simplest account. A model can learn meaning purely from how words relate to other words. One note frames this through the linguist Saussure, who argued that a word's meaning comes from its differences from other words, not from pointing at things. On this view, LLMs are almost a working demonstration: they compress the web of relations between words and produce fluent, culturally specific language without ever touching a referent Can language models learn meaning without engaging the world?. A second argument goes further. The text LLMs learn from was written by people who *did* have bodies and *did* interact with the world, so the regularities in that text carry traces of real cause and effect. Extracting them gives the model a kind of secondhand grounding. It ends up with structured world representations that are real, but it can't check them against reality or update them as things change Can large language models develop genuine world models without direct environmental contact?.
What do those 'concepts' look like inside the model? Interpretability research suggests there are layers of them. At the bottom, concepts show up as directions in the model's internal space. Above that are factual links between those concepts. At the top are compact circuits that encode actual principles. The key finding is that the higher levels don't replace the lower ones. Real understanding sits right next to shallow shortcuts, so you get a patchwork rather than a clean ladder Do language models understand in fundamentally different ways?. Related work argues that most of this conceptual work happens in hidden internal states, not in the reasoning the model writes out, so its visible chain-of-thought is only a partial view of what's going on Where does LLM reasoning actually happen during generation?.
That patchwork explains a strange failure that wouldn't happen with a unified mental model. Models can correctly explain a concept, fail to apply it, and then correctly notice that they failed. No human mind works quite like that. It suggests that explaining and doing run on separate tracks that aren't connected Can LLMs understand concepts they cannot apply?, and it's one of several repeatable ways LLM knowledge breaks down that go beyond simply being wrong How do LLMs fail to know what they seem to understand?. Models trained on psychological data even reproduce human-like quirks, such as how we update beliefs and divide experience into events. But they compress more aggressively than people do and lose contextual nuance along the way How do language models learn to think like humans?.
The twist worth taking away: whether to call any of this 'concepts' or 'beliefs' is now an active philosophical question, and some philosophers argue for a modest yes. That means attributing simple belief-like states to models while withholding claims about consciousness, much as we do with animals Can we defend modest mental attributions to large language models? Can we describe LLM beliefs without assuming consciousness?. Influence also runs the other way. As LLM vocabulary spreads, people start describing their own minds in its terms, with memory as 'retrieval' and creativity as 'recombination' How does LLM vocabulary spread beliefs about human thinking?. So asking how LLMs form concepts without a body may end up changing how we think humans form them.
Sources 10 notes
Research shows LLMs learn culturally situated discourse patterns by compressing relational structure from text, demonstrating that fluent language generation requires no external referents or embodied grounding.
LLMs form structured world representations by extracting regularities from training data produced by causally grounded humans. This constitutes indirect causal grounding mediated through text, though the chain has gaps that limit real-time verification and model updating.
Mechanistic interpretability reveals conceptual understanding (features as directions), state-of-world understanding (factual connections), and principled understanding (compact circuits). Crucially, higher tiers coexist with lower-tier heuristics rather than replacing them, creating a patchwork of capabilities.
Evidence from CoT faithfulness tests, feature steering, and layer analysis suggests latent-state dynamics drive reasoning, while surface chain-of-thought serves as a partial interface. Hidden reasoning processes should be the default focus of study.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Show all 10 sources
LLMs show repeatable, empirically documented failure modes—from Potemkin understanding (correct explanation + failed application) to reasoning collapse under implicit constraints. These failures reveal gaps between statistical pattern-tracking and actual epistemic competence.
LLMs trained on psychological data exhibit cognitive phenomena mirroring humans: asymmetric belief updating, event segmentation matching human consensus, and individual-level variation. However, they compress information more aggressively than humans do, sacrificing contextual nuance for statistical efficiency.
Both robustness and etiological deflationist arguments beg the question against inflationism. A graded approach ascribing metaphysically undemanding states like beliefs and desires—while withholding consciousness claims—mirrors how we treat non-human animals.
Chalmers introduces quasi-interpretivism to ascribe belief-like states to LLMs based on behavioral interpretability without committing to phenomenal consciousness. The approach works well for sub-personal functional states but overreaches when applied to relational or normative states like speech-acts.
LLM features get projected onto humans through two mechanisms: analogical transfer (memory as retrieval, creativity as recombination) and metaphorical availability (LLM vocabulary becoming psychologically salient). This pattern propagates the bias without requiring explicit endorsement.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Debate Over Understanding in AI's Large Language Models
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Word Meanings in Transformer Language Models
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
- Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
- Mechanistic Indicators of Understanding in Large Language Models
- From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning