Trajectory Geometry of Transformer Representations Across Layers

Understanding how transformer representations evolve across layers not merely what they encode remains an open problem in mechanistic interpretability. We recast the transformer forward pass as a discrete population trajectory through a high-dimensional representation manifold, drawing on geometric tools from computational neuroscience. Rather than probing for pre-specified features, we characterize the intrinsic geometry of these trajectories using five metrics computed directly in the ambient space: trajectory length, curvature, a semantic convergence index, layerwise cosine similarity, and representational stability. Across three model families (GPT-2, TinyLlama, Qwen2.5) and five semantically controlled prompt families, we report four principal findings. First, semantically related prompts undergo statistically significant trajectory convergence in middle-to-late layers, with peak convergence indices of 0.41–0.58 across architectures (p < 0.001, Mann-Whitney U), consistent with attractor-like dynamics. Second, reasoning and analogy tasks produce trajectories of significantly greater curvature than lexical variation tasks (0.71–0.83 rad vs. 0.27–0.31 rad), suggesting that mean curvature encodes computational complexity.

Introduction. While modern transformers [1] achieve remarkable performance across diverse natural language tasks, the internal mechanisms governing their representations remain largely opaque. Existing interpretability approaches operate primarily in two regimes: mechanistic analyses that trace individual attention heads and circuit-level computations [2, 3], and static analyses that probe layer-wise embeddings for pre-specified linguistic features [4, 5]. Both paradigms treat each layer as an independent snapshot, ignoring the continuous geometric structure of how representations evolve across the depth of the network. We propose a complementary perspective: the transformer forward pass as a discrete population trajectory through a high-dimensional representation manifold. Rather than probing for what is encoded at a given layer, we ask how representations travel from input to output, characterizing the geometry of the path itself.

Discussion / Conclusion. Our four findings collectively support a coherent picture: the transformer forward pass implements a structured geometric flow through representation space, with distinct computational phases, task-sensitive path geometry, and learned disambiguation dynamics. We discuss the theoretical implications, connections to prior work, practical consequences, and limitations of this view. Implications for Interpretability and Alignment: Beyond descriptive geometry, our findings suggest several actionable directions. First, the layer-resolved map of computational phases provides a principled basis for layer selection in representation engineering: interventions targeting semantic content should be applied in Phase II (elaboration), while output-vocabulary interventions belong in Phase III.