SYNTHESIS NOTE
Topics›Linguistics, NLP, NLU›this note

Do language models show the same content effects humans do?

Do LLMs reproduce human reasoning biases—like believing conclusions based on familiarity rather than logic—across different logical tasks? This matters because converging patterns across independent tasks suggest a fundamental architectural property rather than a task-specific quirk.

Synthesis note · 2026-05-02 · sourced from Linguistics, NLP, NLU

Lampinen et al. evaluate three logical reasoning tasks — Natural Language Inference, syllogism validity judgment, and the Wason selection task — and find LMs reproduce the same content-sensitivity patterns humans show. NLI: accuracy depends on whether the believable completion matches the logically correct one. Syllogisms: judgments are biased by whether the conclusion is believable, reproducing Evans et al.'s belief-bias effect where humans endorse invalid syllogisms with believable conclusions roughly 90% of the time. Wason: accuracy improves when the conditional rule is instantiated as a familiar social rule rather than an abstract pattern. Three independent task structures with different logical demands all produce the same content-form entanglement.

The pattern matters because each individual task could be dismissed as a quirk. Three tasks converging on the same signature licenses calling content-form entanglement an architectural property rather than a benchmark artifact. The mechanistic vault notes establish why at circuit level: How do language models perform syllogistic reasoning internally? shows the formal-circuit + world-knowledge-contamination structure that produces belief-bias. This insight contributes the behavioral isomorphism — not just that the circuit produces some kind of contamination, but that the contamination's signature matches human error patterns item-for-item, including continuous response measures (LM token-probability distributions track human reaction times).

This converges with Do large language models reason symbolically or semantically? from the opposite direction. That note shows reasoning collapses when semantics are stripped; Lampinen shows reasoning improves with believable semantics and degrades with unbelievable semantics. Both findings point at the same property: the model is doing something like in-context semantic reasoning, where logical form is one input among others rather than the dominant computational frame. Calling this "reasoning" or "not reasoning" is the wrong question — the right question is what kind of reasoning, and the answer is reasoning that is constitutively content-sensitive, in humans and LMs alike, by the same item-level patterns.

Inquiring lines that read this note 51

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Why do LLMs succeed at generation but struggle with evaluation? Why do LLM outputs diverge from their knowledge and reasoning? When do complex LLM tasks benefit from external verification or structured scaffolding? Why do users misjudge the credibility of AI-generated and verified content? Do LLMs lack causal understanding needed for accurate prediction? Do language models possess genuine mental states or only simulate understanding? Do language models learn genuine understanding or just surface patterns? Do reasoning improvements prevent models from abstaining when needed? What distinguishes genuine linguistic understanding from mere pattern matching? What drives the effectiveness and failure of language model reasoning? How do language models handle contested beliefs versus facts? How do transformer attention mechanisms create position and sycophancy biases? Do face-saving social norms drive language model sycophancy and grounding failures? What biases affect LLM judges and how can they be reduced? How do symbolic and semantic approaches combine in LLM reasoning? Do language models rely on encoded knowledge or primarily on surface patterns? Can persona-based approaches improve individual user preference prediction? Does chain-of-thought text reveal genuine reasoning or simulate it? Do small language models have fundamentally different scaling properties than large ones? Can aggregate reward models preserve diverse human preferences? What prevents language models from developing genuine theory of mind reasoning? Can AI-generated text be reliably distinguished from human writing? Does alignment training modify underlying model representations or just output compliance? Why do independent language models converge on similar outputs? When does AI explainability improve versus hinder human oversight and trust?

Related concepts in this collection 2

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
12 direct connections · 155 in 2-hop network ·dense cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

content effects in LLMs are behavioral confirmation that semantic content and logical form are not separable in transformer reasoning — across NLI syllogisms and Wason