How do language models encode syntactic relations geometrically?
Do LLM embeddings use distance alone or also direction to represent syntax? Understanding whether neural networks can spontaneously develop symbolic-compatible geometric structures.
The symbol-vector divide has been a core challenge in cognitive science since Smolensky (1987): syntactic trees are symbolic structures that seem incompatible with the vectorial representations of neural networks. The Structural Probe (Hewitt & Manning 2019) made partial progress — it showed that the existence of syntactic links between words is encoded in the distance between their corresponding embeddings. But whether the type and direction of syntactic relations were represented remained unknown.
The Polar Probe answers this: syntactic relations are coded by the relative direction between nearby embeddings, not just their distance. Using both distance and direction (a polar coordinate system), the Polar Probe recovers syntactic relation types and directions with nearly 2x the accuracy of the distance-only Structural Probe.
Three key findings:
Complete syntactic encoding. The polar coordinate system captures existence, type, AND direction of syntactic relations — the full specification of a dependency tree is encoded in the geometry of LLM activations.
Low-dimensional subspace. This encoding exists in a low-dimensional subspace of intermediate layers across many LLMs, and becomes increasingly precise in frontier models. This is not a brute-force representation but a compressed, structured one.
Nested consistency. Similar syntactic relations are coded similarly across nested levels of syntactic trees. The encoding is not ad hoc for each syntactic instance but systematic — a genuine coordinate system.
The resolution of the symbol-vector divide is significant: LLMs don't need explicit symbolic mechanisms to represent symbolic structures. They spontaneously learn a geometry that explicitly represents the main symbolic structures of linguistic theory. This doesn't mean LLMs "understand" syntax in a human sense, but it demonstrates that connectionist architectures can natively develop symbolic-compatible representations — the two paradigms are not incompatible.
This connects to Do transformer static embeddings actually encode semantic meaning? at a different structural level: static embeddings encode semantic features, while intermediate activations encode syntactic relations. Together they suggest LLM representations are far richer and more structured than the "statistical patterns" dismissal implies.
Inquiring lines that use this note as a source 51
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- How do embedding dimension limits constrain what concept models can represent?
- Can neural networks represent symbolic structures without explicit mechanisms?
- How does syntactic encoding relate to semantic feature representation?
- What compression explains why syntax fits in low-dimensional subspaces?
- What mathematical limits constrain embedding-based retrieval systems?
- What other semantic relations benefit from explicit surface markers in text?
- How do functional features differ from representational abstract features?
- How should meaning spaces be systematically modeled across different applications?
- What makes linear decodability a reliable signal of compositionality?
- What happens when you tightly couple two representations together?
- Can speech embeddings carry articulatory structure that text cannot?
- Do multi-vector or cross-encoder models escape these dimensional constraints?
- Can steering vectors prove that representations are genuinely organized?
- How does the symbol grounding problem apply to artificial language systems?
- How do hidden embeddings preserve more information than discrete tokens?
- How does the distance between natural language and formal notation affect translation accuracy?
- Do language models encode deep syntactic structure or only surface-level patterns?
- Why do surface generalizations fail on unusual syntactic structures?
- Can LLMs reliably generate novel working architectures without structured representations?
- Does the linear representation hypothesis reflect networks or reflect our analysis tools?
- What role does a model's representational structure play in learning?
- Can representation engineering cleanly isolate single features in entangled semantic space?
- Why do language models reproduce human EPA structure despite different architecture?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- How does iconicity detection work within static embeddings before any attention?
- How do static embeddings and contextualized representations divide semantic labor?
- What other behavioral properties exist as linear directions in activation space?
- Why is a combinatorial framework better than family resemblance classification?
- What makes modernized N-gram embeddings composable with transformer architectures?
- Why do single vectors fail at capturing negation and word order?
- What other structural limits exist at the language-formal boundary?
- How do encode-decode contractive biases create stable attractors in latent space?
- Do language models and multimodal models show similar attractor-based interpretability?
- Why do leading embedding eigenvectors align with WordNet taxonomy structure?
- Can geometric structure in representations exist without supporting functional mechanisms?
- What spectral signatures distinguish hierarchy-driven geometry from corpus-driven geometry?
- Why do unit-sphere spaces fail at distinguishing word order and negation?
- Why does gradient descent discover compositional structure without explicit pressure?
- Can language models execute iterative numerical methods in latent space?
- Does language convey meaning purely through relational structure without external grounding?
- Can spectral eigenvector ordering serve as a model-agnostic interpretability probe?
- Can representation analysis methods detect complex features models compute with?
- What geometric structure do language models actually use during inference?
- Does Gemma's transformer explicitly exploit the inherited hierarchical geometry?
- What physical structure does a Gaussian-regularized latent space actually encode?
- Can single-vector embeddings capture non-commutative relationships like word order?
- What makes regularization an implicit factor in embedding geometry?
- What temporal and spatial constraints does Space-Time U-Net solve?
- How should we rethink the symbolism versus connectionism debate in light of LLMs?
- How does serializing screen layout to text preserve spatial relationships?
- How do semantic features in representations become steerable task-specific directions?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do transformer static embeddings actually encode semantic meaning?
Explores whether the fixed word embeddings that enter transformer networks contain rich semantic information or serve only as shallow placeholders. This addresses a longstanding debate in philosophy of language about whether word meanings are stored or constructed.
semantic features in static embeddings complement syntactic features in intermediate activations
-
Why do neural networks fail at compositional generalization?
Exploring whether the binding problem from neuroscience explains neural networks' inability to systematically generalize. The binding problem has three aspects—segregation, representation, and composition—each creating distinct failure modes in how networks handle structured information.
polar coordinate encoding is evidence against the strong version: systematic structure IS represented, even if binding problems remain at the compositional level
-
Can neural networks learn compositional skills without symbolic mechanisms?
Do neural networks need explicit symbolic architecture to compose learned concepts, or can scaling alone enable compositional generalization? This asks whether compositionality is an architectural feature or an emergent property of scale.
convergent: symbolic-like structure emerges without explicit symbolic mechanisms
-
Do neural networks naturally learn modular compositional structure?
Explores whether neural networks decompose compositional tasks into distinct subroutines without explicit symbolic design. This challenges the longstanding view that neural networks are fundamentally non-compositional.
related: modular structure emerges from training
-
Where does hierarchical structure in language models come from?
Do LLMs build hierarchical concept geometry through dedicated mechanisms, or does it emerge naturally from word co-occurrence patterns in training data? Understanding the source matters for interpreting what representations actually reveal about model computation.
contrasts: this note reads symbolic-compatible geometry as spontaneously learned, but distributional theory shows such structure can be a co-occurrence shadow not a learned mechanism
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- A polar coordinate system represents syntax in large language models
- Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
- Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
- On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models
- Semantic Structure in Large Language Model Embeddings
- Large Linguistic Models: Investigating LLMs' metalinguistic abilities
Original note title
a polar coordinate system in llm activations encodes both type and direction of syntactic relations — resolving the symbol-vector divide