When a model reasons through several steps, is it traveling a path through a hidden map of meaning?

This explores whether the *shape* of a model's internal representation space — how entities cluster, the angles between them, the curve of the path a forward pass traces — is what enables reasoning across multiple steps. The corpus suggests geometry isn't decoration on top of reasoning; it's where the reasoning is visibly happening. The clearest evidence comes from watching multi-hop skill emerge during training: successful cross-step reasoning shows up as *cosine clustering* of entity representations, meaning the model literally pulls related concepts into the same angular neighborhood before it can compose them How do transformers learn to reason across multiple steps?. Reasoning, in other words, has a geometric fingerprint you can measure.

That fingerprint isn't only about *where* things sit but *how the model moves through* the space. Measuring the intrinsic geometry of the forward pass shows that genuine reasoning and analogy tasks carve high-curvature paths (0.71–0.83 rad) while shallow lexical tasks trace nearly straight lines (0.27–0.31 rad) — the harder the composition, the more the trajectory bends Does transformer reasoning leave a geometric signature in representation space?. And the geometry is richer than distance alone: a polar-coordinate probe finds LLMs encode *both* the type and the direction of a relation using distance *and* angle between embeddings, nearly doubling accuracy over distance-only methods How do language models encode syntactic relations geometrically?. Direction matters because multi-hop reasoning is directional — A relates to B relates to C — and angular structure is how the model keeps the arrows pointing the right way.

Here's the twist a curious reader might not expect: if geometry carries the reasoning, you can *intervene* on it. Because distinct kinds of thinking occupy distinct regions of activation space, researchers extract a single steering vector that shifts chain-of-thought toward brevity — cutting length 67% with no retraining, just by nudging the model along a direction in that space Can we steer reasoning toward brevity without retraining?. The same logic runs the other way in Soft Thinking, which refuses to collapse a reasoning step down to one discrete token and instead keeps a probability-weighted blend of concept embeddings, letting several candidate paths coexist as a superposition in the continuous space before the model commits Can we explore multiple reasoning paths without committing to one token?. Both treat the embedding geometry as a control surface, not a readout.

There's a strand that pushes reasoning *entirely* into geometry. Meta's Large Concept Model abandons token-by-token generation to reason over whole sentence embeddings in a language-agnostic space, planning at the concept level before decoding Can reasoning happen at the sentence level instead of tokens?, while SoftCoT delegates continuous 'soft thought' generation to a small assistant so the frozen backbone can reason in embedding space without forgetting what it knew Can continuous reasoning avoid forgetting in instruction-tuned models?. Looped architectures make a related bet from a different angle: re-applying the same layers in recurrent depth lets a model refine its position in representation space over many passes, buying compositional generalization that raw parameter scale can't Can models learn by looping instead of growing larger?.

But geometry inside the model isn't the only place multi-hop reasoning lives — and the contrast is instructive. A whole family of systems externalizes the hops into *explicit graph structure* rather than trusting the embedding manifold: HippoRAG runs Personalized PageRank over a knowledge graph to traverse multi-hop paths in a single retrieval Can knowledge graphs enable multi-hop reasoning in one retrieval step?, and hypergraph memory binds three-or-more entities into one relation so joint constraints survive across steps Can hypergraphs capture multi-hop reasoning better than graphs?. The reason both exist is the corpus's cautionary note: chain-of-thought that lives purely in learned geometry degrades predictably once you leave the training distribution, producing fluent paths that don't actually connect Does chain-of-thought reasoning actually generalize beyond training data?. So the honest answer is that embedding geometry is *how* transformers do multi-hop reasoning natively — clustering, angle, and curvature are the mechanism — but that same geometry is also why the reasoning can look right while being wrong, which is exactly why explicit graph structure keeps getting bolted back on.