Does GPT-4 retrieve analogies differently than humans do?

A controlled experiment pitted GPT-4 against business-school Master's students on a "matching problem" in analogical reasoning: each subject saw two source stories (the classic Gick & Holyoak radiation problem, and a new "Dolphins" story illustrating survivorship bias) and two target business problems, where only one source correctly applied to each target and the other acted as a placebo. The paper reports: "GPT-4 achieves high recall—retrieving all plausible analogies—but suffers from low precision, frequently applying incorrect analogies based on superficial surface features. Humans, by contrast, exhibit high precision but low recall, selecting fewer analogies yet with stronger causal alignment." Without a hint, GPT-4 showed "essentially complete absence of the evidence for analogical reasoning" (zero precision, recall, and F1); with a hint, its recall jumped to 1.0 but precision dropped as it over-applied analogies to both targets.

The mechanism the authors give rests on the standard two-stage account of analogical reasoning: retrieval, where a source analogy "comes spontaneously to mind" via cue-dependent, often superficial similarity, and mapping, where the candidate is checked against the target for structural or causal correspondence. GPT-4's errors are retrieval errors mistaken for matches — it finds surface links ("dolphin→sea→city factory with port") rather than testing causal structure, and sometimes invokes both stories at once regardless of fit, a pattern the authors call a "demand effect" (using an available analogy because it's available). Human errors, by contrast, are "subtler misinterpretations of causal logic" — people who attempt a causal mapping get it wrong in more sophisticated ways, but attempt it far less often than they should, "missing to spot an opportunity for analogical transfer" even after a hint.

This is a domain-specific instance of the pattern in Do foundation models learn world models or task-specific shortcuts?: there, transformers predict orbital trajectories and arithmetic answers accurately using task-specific heuristics rather than an internalized causal model (Newtonian mechanics, an addition algorithm); here, GPT-4 retrieves analogies accurately using surface-feature similarity rather than causal mapping. Both cases show similarity/pattern-matching substituting for structural understanding, and both only surface the gap when the task forces a discrimination that heuristics can't settle — multiple candidate analogies here, out-of-distribution data there. The finding also echoes the distinction in Can matching human actions prove an LLM simulation explains behavior?: retrieving a plausible-looking analogy (a correct-seeming output) is not the same as having performed the causal mapping that would justify it, just as matching an action is not the same as matching the reasoning behind it.

The study does not establish how this trade-off would hold outside a two-source, two-target lab task: the human sample is explicitly "a sample of strategy novices" (business students, not senior executives or cross-cultural cohorts), and the authors note the task environment was "deliberately simplified" relative to real strategic search, where the space of candidate analogies is far larger and less curated than a forced binary choice. The paper's own proposed implication — a "potentially productive division of labor" with AI as analogy generator and humans as causal evaluators — is offered as a suggestion the data point toward, not a result demonstrated under realistic organizational conditions.