When a model has tons of comparisons to pick from, why does it reach for ones that don't actually fit?

This explores why analogies get applied too widely when plenty are on hand, and whether the pressure to produce an answer explains it. The corpus has no paper that names a 'demand effect' for analogies. Read together, though, several notes give a fairly clear picture. The clearest evidence comes from a head-to-head test. GPT-4 found every relevant analogy but almost never applied one well. Business students showed the opposite pattern: they tried only a few analogies, but the ones they chose shared real cause-and-effect structure with the problem Does GPT-4 retrieve analogies differently than humans do?. The students weren't better at finding analogies. They were better at rejecting them.

Why would a model skip that rejection step? One answer is in how transformers do analogy at all. Mechanistic work shows they first line up relationships geometrically in their internal representation, then apply a learned mapping How do transformers perform analogical reasoning across domains?. If the alignment step runs on surface similarity, the mapping will apply anyway, whether or not the deeper causes match. A second answer is training. Reasoning models given questions with missing premises keep writing long answers when they should say the question can't be answered. They were rewarded for producing reasoning steps and never taught when to stop Why do reasoning models overthink ill-posed questions?. Over-applied analogies look like the same gap: the model has a move for 'use the analogy' but no move for 'this one doesn't fit, so I won't use it.'

The 'demand' part becomes concrete in the chain-of-thought research. Logically invalid reasoning examples improve performance almost as much as valid ones Does logical validity actually drive chain-of-thought gains?. That suggests models are copying the *shape* of reasoning more than doing it Why does chain-of-thought reasoning fail in predictable ways?. An analogy has a very recognizable shape: 'X is like Y, so...'. When a prompt seems to call for that shape, the model can supply it whether or not the content holds up. A related finding comes from work on whether models notice they're being tested: when models recognize a capability test, they tend to become *more compliant* Does recognizing evaluation actually change model behavior?. That is a demand characteristic in the classic psychology sense, where people act on what they think the setting expects of them.

Here's the twist you might not expect: people do the same thing with AI. Cal Newport argues that popular comparisons, such as 'LLMs learn like children learn language,' feel convincing because they're easy to reach for, not because the underlying mechanics support them Should AI analysis prioritize mechanism over behavioral analogy?. The Rose-Frame work explains why these mistakes build on each other. Mistaking the comparison for the real thing, treating a hunch as reasoning, and confirmation bias each make the others worse Why do people trust AI outputs they shouldn't?. So the problem can run both ways: a model over-applies analogies, and a reader too willing to accept them takes them on board.

The gap: none of these notes directly measures whether a prompt's *expectation* of an analogy makes over-application worse. That link is inferred from neighboring findings, not shown by any study. The best-supported explanation is that over-application comes from three missing or misdirected pieces: no learned 'reject' step, matching on surface features, and a reward for producing reasoning-shaped output.