Does an AI think in letters, words, or phrases — or in something that doesn't match how we read text?
What linguistic units do learned concepts correspond to in a language model?
This explores whether the concepts a language model learns line up with the units of text it reads and writes (tokens, words, multi-word phrases) or with something else.
This explores whether a model's learned concepts line up with the units of text it reads and writes: tokens, words, phrases. The corpus suggests they don't sit at one unit. Concepts start at the token level, often stretch across several tokens, and end up as directions in the model's internal space that aren't tied to any particular string. No note here maps concepts onto morphemes, words, or sentences directly, so this answer is pieced together from several angles.
Tokens are the entry point, not the concept. Circuit tracing in Claude models finds four tiers of features: token-level inputs, abstract concepts, functional operations, and outputs How do language models organize features across processing layers?. Concepts form above the token layer, and larger models build richer abstract features. This fits a quirk of how models are trained. Next-token prediction sees a multi-word idea one fragment at a time. Two lines of work try to fix that. Concept-aware fine-tuning adds multi-token prediction and finds it forms coherent semantic entities, even beating standard fine-tuning Can models learn multi-token concepts during fine-tuning?. A model trained to predict learned concepts alongside tokens reached the same final loss as OLMo-3-7B with about half the training tokens Can models learn faster by predicting their own concepts?. If explicit multi-token supervision helps this much, the natural unit of a concept is often a span of text, not a single token.
Words still shape the geometry. The nested structure of concepts (broad categories containing narrower ones) needs no dedicated mechanism. It falls out of the spectral structure of word co-occurrence statistics, and word2vec reproduces it Where does hierarchical structure in language models come from?. So words are not the concepts, but how words appear together in text sets the shape of concept space.
Once formed, concepts behave less like chunks of text and more like directions. Mechanistic work describes conceptual understanding as features that are directions in activation space Do language models understand in fundamentally different ways?. Twenty-eight semantic axes in LLM embeddings collapse into three components that match human judgments of good–bad, strong–weak, and active–passive. Pushing on one feature moves the aligned ones with it Do LLM semantic features organize along human evaluation dimensions?. A concept in this picture is a direction along which many words and phrases vary. It is not a dictionary entry.
The label on a unit doesn't reliably tell you what the concept inside it does. Models can encode facts that never causally affect what they generate Do language models actually use their encoded knowledge?. Low-resource cultures like Ethiopia and Algeria are represented internally through high-resource cultural proxies, even when the surface answers are right Do LLMs represent low-resource cultures through dominant cultural proxies?. A name or word in the input can therefore be handled by a concept borrowed from somewhere else.
Sources 8 notes
Circuit tracing in Claude models reveals features progress from token-level inputs to abstract concepts to functional operations to outputs. Larger models develop richer abstract features, suggesting scaling enables higher-level conceptual reasoning rather than pattern memorization.
CAFT successfully brings multi-token prediction to post-training via self-distilled auxiliary heads, outperforming next-token fine-tuning on tasks like protein design. CAFT LoRA even outperforms full next-token fine-tuning, suggesting models learn more effectively in multi-token settings.
An 8.9B model trained to predict both tokens and learned concepts from its hidden states matched OLMo-3-7B's final loss using only 51.3% of training tokens and outperformed it by 2.45 points downstream. This suggests explicit supervision of multi-token semantic structure improves compute efficiency.
LLM hierarchical representations arise as a direct mathematical consequence of corpus statistics, not from hierarchy-specific mechanisms. Spectral analysis of word co-occurrence matrices predicts and reproduces the same nested geometry found in trained embeddings and word2vec models.
Mechanistic interpretability reveals conceptual understanding (features as directions), state-of-world understanding (factual connections), and principled understanding (compact circuits). Crucially, higher tiers coexist with lower-tier heuristics rather than replacing them, creating a patchwork of capabilities.
Show all 8 sources
Twenty-eight semantic axes in LLM embeddings reduce to three principal components matching human EPA structure. Intervening on one feature predictably shifts aligned features proportionally, creating unavoidable off-target effects that reflect how meaning is fundamentally organized.
Multiple studies confirm that language models can encode facts in their representations while those facts fail to causally affect downstream outputs. Encoding and usage are distinct processes.
Mechanistic interpretability analysis reveals that low-resource cultures like Ethiopia and Algeria are structurally represented through high-resource cultural proxies in internal model states, not just output. This architectural bias persists even when models can produce correct surface-level answers.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
- Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
- Word Meanings in Transformer Language Models
- Semantic Structure in Large Language Model Embeddings
- From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
- Large Concept Models: Language Modeling in a Sentence Representation Space
- Six misconceptions about large language models: A minimal model and diagnostic taxonomy
- Mechanistic Indicators of Understanding in Large Language Models