How do language models encode syntactic relations geometrically?
Do LLM embeddings use distance alone or also direction to represent syntax? Understanding whether neural networks can spontaneously develop symbolic-compatible geometric structures.
The symbol-vector divide has been a core challenge in cognitive science since Smolensky (1987): syntactic trees are symbolic structures that seem incompatible with the vectorial representations of neural networks. The Structural Probe (Hewitt & Manning 2019) made partial progress — it showed that the existence of syntactic links between words is encoded in the distance between their corresponding embeddings. But whether the type and direction of syntactic relations were represented remained unknown.
The Polar Probe answers this: syntactic relations are coded by the relative direction between nearby embeddings, not just their distance. Using both distance and direction (a polar coordinate system), the Polar Probe recovers syntactic relation types and directions with nearly 2x the accuracy of the distance-only Structural Probe.
Three key findings:
Complete syntactic encoding. The polar coordinate system captures existence, type, AND direction of syntactic relations — the full specification of a dependency tree is encoded in the geometry of LLM activations.
Low-dimensional subspace. This encoding exists in a low-dimensional subspace of intermediate layers across many LLMs, and becomes increasingly precise in frontier models. This is not a brute-force representation but a compressed, structured one.
Nested consistency. Similar syntactic relations are coded similarly across nested levels of syntactic trees. The encoding is not ad hoc for each syntactic instance but systematic — a genuine coordinate system.
The resolution of the symbol-vector divide is significant: LLMs don't need explicit symbolic mechanisms to represent symbolic structures. They spontaneously learn a geometry that explicitly represents the main symbolic structures of linguistic theory. This doesn't mean LLMs "understand" syntax in a human sense, but it demonstrates that connectionist architectures can natively develop symbolic-compatible representations — the two paradigms are not incompatible.
This connects to Do transformer static embeddings actually encode semantic meaning? at a different structural level: static embeddings encode semantic features, while intermediate activations encode syntactic relations. Together they suggest LLM representations are far richer and more structured than the "statistical patterns" dismissal implies.
Inquiring lines that read this note 58
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do mechanistic features and representations drive model behavior?- How do embedding dimension limits constrain what concept models can represent?
- How do functional features differ from representational abstract features?
- What makes linear decodability a reliable signal of compositionality?
- What role does a model's representational structure play in learning?
- Can representation engineering cleanly isolate single features in entangled semantic space?
- Do language models and multimodal models show similar attractor-based interpretability?
- Can geometric structure in representations exist without supporting functional mechanisms?
- Can representation analysis methods detect complex features models compute with?
- How do semantic features in representations become steerable task-specific directions?
- Can neural networks represent symbolic structures without explicit mechanisms?
- Does the linear representation hypothesis reflect networks or reflect our analysis tools?
- Why does gradient descent discover compositional structure without explicit pressure?
- How should we rethink the symbolism versus connectionism debate in light of LLMs?
- How do concept vectors in neural networks relate to human-language chess explanations?
- How does syntactic encoding relate to semantic feature representation?
- How does the distance between natural language and formal notation affect translation accuracy?
- What compression explains why syntax fits in low-dimensional subspaces?
- Can speech embeddings carry articulatory structure that text cannot?
- How do hidden embeddings preserve more information than discrete tokens?
- How does iconicity detection work within static embeddings before any attention?
- How do encode-decode contractive biases create stable attractors in latent space?
- What spectral signatures distinguish hierarchy-driven geometry from corpus-driven geometry?
- Can spectral eigenvector ordering serve as a model-agnostic interpretability probe?
- What physical structure does a Gaussian-regularized latent space actually encode?
- What temporal and spatial constraints does Space-Time U-Net solve?
- Can generative reconstruction preserve latent manifold structure better than geometric compression?
- How do speech encoders learn articulatory physics without phonetic labels?
- Can speech encoders learn vocal tract physics without phonetic labels?
- What mathematical limits constrain embedding-based retrieval systems?
- Do multi-vector or cross-encoder models escape these dimensional constraints?
- Why is a combinatorial framework better than family resemblance classification?
- Why do single vectors fail at capturing negation and word order?
- Why do leading embedding eigenvectors align with WordNet taxonomy structure?
- Why do unit-sphere spaces fail at distinguishing word order and negation?
- Can single-vector embeddings capture non-commutative relationships like word order?
- What other semantic relations benefit from explicit surface markers in text?
- How do static embeddings and contextualized representations divide semantic labor?
- Does language convey meaning purely through relational structure without external grounding?
- How should meaning spaces be systematically modeled across different applications?
- What other structural limits exist at the language-formal boundary?
- What geometric structure do language models actually use during inference?
- Do language models encode deep syntactic structure or only surface-level patterns?
- Why do surface generalizations fail on unusual syntactic structures?
- Can language models execute iterative numerical methods in latent space?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do transformer static embeddings actually encode semantic meaning?
Explores whether the fixed word embeddings that enter transformer networks contain rich semantic information or serve only as shallow placeholders. This addresses a longstanding debate in philosophy of language about whether word meanings are stored or constructed.
semantic features in static embeddings complement syntactic features in intermediate activations
-
Why do neural networks fail at compositional generalization?
Exploring whether the binding problem from neuroscience explains neural networks' inability to systematically generalize. The binding problem has three aspects—segregation, representation, and composition—each creating distinct failure modes in how networks handle structured information.
polar coordinate encoding is evidence against the strong version: systematic structure IS represented, even if binding problems remain at the compositional level
-
Can neural networks learn compositional skills without symbolic mechanisms?
Do neural networks need explicit symbolic architecture to compose learned concepts, or can scaling alone enable compositional generalization? This asks whether compositionality is an architectural feature or an emergent property of scale.
convergent: symbolic-like structure emerges without explicit symbolic mechanisms
-
Do neural networks naturally learn modular compositional structure?
Explores whether neural networks decompose compositional tasks into distinct subroutines without explicit symbolic design. This challenges the longstanding view that neural networks are fundamentally non-compositional.
related: modular structure emerges from training
-
Where does hierarchical structure in language models come from?
Do LLMs build hierarchical concept geometry through dedicated mechanisms, or does it emerge naturally from word co-occurrence patterns in training data? Understanding the source matters for interpreting what representations actually reveal about model computation.
contrasts: this note reads symbolic-compatible geometry as spontaneously learned, but distributional theory shows such structure can be a co-occurrence shadow not a learned mechanism
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- A polar coordinate system represents syntax in large language models
- Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
- Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
- On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models
- Semantic Structure in Large Language Model Embeddings
- Large Linguistic Models: Investigating LLMs' metalinguistic abilities
Original note title
a polar coordinate system in llm activations encodes both type and direction of syntactic relations — resolving the symbol-vector divide