Emergent Analogical Reasoning in Transformers

Analogy is a central faculty of human intelligence, enabling abstract patterns discovered in one domain to be applied to another. Despite its central role in cognition, the mechanisms by which Transformers acquire and implement analogical reasoning remain poorly understood. In this work, inspired by the notion of functors in category theory, we formalize analogical reasoning as the inference of correspondences between entities across categories. Based on this formulation, we introduce synthetic tasks that evaluate the emergence of analogical reasoning under controlled settings. We find that the emergence of analogical reasoning is highly sensitive to data characteristics, optimization choices, and model scale. Through mechanistic analysis, we show that analogical reasoning in Transformers decomposes into two key components: (1) geometric alignment of relational structure in the embedding space, and (2) the application of a functor within the Transformer. These mechanisms enable models to transfer relational structure from one category to another, realizing analogy. Finally, we quantify these effects and find that the same trends are observed in pretrained LLMs.

Introduction. Recent years have witnessed remarkable progress in the reasoning capabilities of large language models (LLMs), particularly in constructing chains of intermediate reasoning before the final answer (Wei et al., 2022; Kojima et al., 2022; OpenAI et al., 2024; Google DeepMind, 2025; DeepSeek- AI et al., 2025). These developments have renewed interest in a key question: how do LLMs realize reasoning? Much of recent research on understanding reasoning frames reasoning as compositional reasoning, where complex reasoning arises from sequentially composing simpler primitives. For example, given the facts Beyond compositional reasoning, humans exhibit a qualitatively different form of reasoning, analogy. Rather than producing conclusions by chaining local steps, analogy identifies shared relational structure across distinct domains, enabling a form of “leap” (Gentner, 1983; Holyoak & Thagard, 1996; Bartha, 2013).

Discussion / Conclusion. Summary. In this work, we focus on analogy, an underexplored aspect of reasoning in LLMs. We introduce a synthetic task designed to analyze analogical reasoning in Section 2. In Section 3, we investigate its training dynamics and their relationship to data characteristics, optimizers, and model scale. We then discover the internal mechanisms in Section 4, and show that closely related mechanisms can also be observed in pretrained LLMs in Section 5. We approach reasoning from a novel perspective based on analogy, while remaining closely connected to existing work. Analogy and Language Models. In cognitive science, analogy is formalized by Structure-Mapping Theory, which characterizes analogy as a mapping that preserves higherorder relations rather than surface features (Gentner & Markman, 1997; Gick & Holyoak, 1983). Building on this, in natural language processing, analogy has traditionally been studied through four-term lexical analogies (A:B :: C :D) (Turney et al., 2003; Mikolov et al., 2013a; Pennington et al., 2014; Ethayarajh et al., 2019; Ushio et al., 2021; Lewis & Mitchell, 2025; Musker et al., 2025).