SYNTHESIS NOTE
Do large language models reason symbolically or semantically?
Can LLMs follow explicit logical rules when those rules contradict their training knowledge? Testing whether reasoning operates independently of semantic associations reveals what computational mechanisms actually drive LLM multi-step inference.
The "In-Context Semantic Reasoners" paper tests a fundamental question about what drives LLM reasoning by systematically decoupling semantics from the reasoning process across deduction, induction, and abduction tasks. The findings are clear: when semantics are consistent with commonsense, LLMs perform well; when semantics are removed or made counter-commonsense, performance collapses even when correct rules are provided in context.
The experimental design is precise. By replacing relation labels with shuffled alternatives ("motherOf" → "sisterOf", "female" → "male"), the researchers create tasks where the in-context rules are logically valid but semantically counter-intuitive. LLMs cannot follow these counter-commonsense rules despite having them explicitly in the prompt. The model's parametric knowledge — its compressed commonsense from training — overrides the in-context logical structure.
This reveals a specific computational mechanism: LLMs create "superficial logical chains" through semantic token associations, not through symbolic manipulation. The connections between tokens that enable multi-step reasoning are semantic connections, not logical ones. When those semantic connections support the correct answer, reasoning appears to work. When they conflict, reasoning fails regardless of what the prompt says.
The implication is that LLM reasoning is fundamentally bounded by training distribution semantics. Since Can large language models translate natural language to logic faithfully?, the failure is bidirectional: LLMs can neither translate TO formal logic faithfully nor reason FROM formal logic when it conflicts with semantic priors. Since Do foundation models learn world models or task-specific shortcuts?, the semantic dependency IS the heuristic — the model uses semantic similarity as a proxy for logical validity.
This connects to the Dual Process Theory framework: human System II symbolic reasoning operates independently of semantic content, but LLM "reasoning" remains entangled with System I semantic associations. The paper's suggestion — integrating LLMs with external non-parametric knowledge bases and improving in-context knowledge processing — implicitly acknowledges that the LLM alone cannot escape this limitation.
Retort implication — rules out a class of anthropomorphization: The finding constrains what we can say about LLM behavior in other domains. Any account that treats LLMs as agents who "reverse-engineer" justifications for conclusions they have committed to — the standard anthropomorphization of sycophancy, rationalization, or motivated reasoning — presupposes the semantic competence this note shows LLMs lack. If reasoning collapses when semantics are decoupled, there is no separable reasoning faculty available to perform a post-hoc rationalization. What looks like reverse-engineering is pattern-matching within semantic associations. This rules out a whole class of AI commentary that treats LLMs as dishonest agents who could have reasoned correctly but chose not to.
Metaphor as paradigmatic semantic decoupling: Metaphor is the literary instantiation of this finding. A metaphor works by using one domain's vocabulary to illuminate another — "time is money," "argument is war," "memory is a jar of flies." The decoupling between the source domain's semantics and the target domain's meaning is the defining feature of metaphorical language. Since LLM reasoning collapses when semantics are decoupled from their typical packaging, and metaphor is decoupled semantics, this predicts a specific failure mode: LLMs should handle conventional metaphors (lexicalized, semantically consistent with commonsense) better than novel literary metaphors (where the mapping between domains is unexpected and requires conceptual reasoning beyond semantic association). The Diplomat dataset (Diplomat: A Dialogue Dataset for Situated PragMATic Reasoning) suggests treating all figurative language as a unified pragmatic reasoning task — but the semantic-decoupling finding predicts that this unified approach will hit a wall at the novelty threshold where metaphors stop relying on conventional semantic associations.
Inquiring lines that read this note 258
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do language models establish social grounding in human dialogue?- Can LLMs infer situational context the way humans do pragmatically?
- How does semantic grounding differ between human minds and language models?
- How does implicit meaning processing limit LLM pragmatic reasoning?
- Can LLMs infer implicit meaning without surface linguistic markers?
- Can training LLMs to form ad-hoc conventions improve their pragmatic reasoning?
- Can LLMs identify implicit metaphoric mappings that require pragmatic inference?
- Why do LLMs fall for and deploy logical fallacies with equal confidence?
- Can evidence density alone shift an LLM from generation to reasoning?
- How much of LLM reasoning failure stems from missing knowledge versus signal weighting?
- Should LLM reasoning be studied as latent state trajectories rather than surface text?
- How does training data distribution constrain LLM moral reasoning patterns?
- What internal mechanisms explain LLM reasoning and representation limits?
- Do LLMs understand implicit warrants in reasoning chains?
- Why can LLMs identify argument structure but not check warrants?
- Why do LLMs fail when asked to use counter-commonsense rules explicitly?
- Why can't LLMs reason from first principles or initial commitments?
- How does context complexity affect LLM performance on temporal reasoning tasks?
- Can LLMs improve at simple deduction through different training approaches?
- Does LLM reasoning always match the outputs it generates?
- Why do LLMs fail at counterfactual reasoning despite factual knowledge?
- Does compressing Walton's schemes into nine categories make LLM classification easier?
- Why do LLMs explain correct reasoning but then choose greedy actions?
- Can irrelevant information reliably expose the limits of LLM reasoning?
- How faithful are natural language explanations from LLMs really?
- Can LLMs simultaneously reason and optimize their own modules?
- Why don't LLM explanations predict what models would actually do?
- Why do LLMs fail inter-annotator agreement tests on argument evaluation?
- Why does hypothesis attestation bias exist separately from frequency bias in NLI?
- Why do LLMs inherit causal biases from their training data?
- Which knowledge types do LLMs handle better than humans in reasoning tasks?
- Can LLMs simulate belief revision in social systems without modeling thought?
- Can implicit association tests reveal LLM biases beneath trained responses?
- Do independent LLM outputs converge enough to create artificial hiveminds?
- How does surface salience compete with background knowledge in model inference?
- How do logical forms of prompts influence what language models can derive?
- Can prompt-based debiasing overcome entrenched LLM model priors?
- Can LLM-generated descriptions of schemes outperform formal dictionary definitions for prompting?
- How do different LLM integration paradigms affect inheritance of pretraining biases?
- How does the outer loop escape its own LLM's knowledge boundaries when discovering mechanisms?
- How does LLM-PKG compare to mining product relations directly from interaction data?
- How do LLMs and knowledge graphs work together in different integration patterns?
- Can explicit linkers replace vector similarity for multi-step question answering?
- Can knowledge graph structure alone generate sufficient training signals for domain reasoning?
- Does epistemic drift operate the same way across all languages?
- What distinguishes inductive inference from negative evidence versus positive patterns?
- How do transformers perform multi-hop reasoning across distant training documents?
- Can symbolic mechanisms improve transformer compositional abilities?
- Can explicit stack mechanisms extend what formal languages transformers can learn?
- Can transformers abstract relational structure without explicit symbolic machinery?
- Why do simple length heuristics outperform sophisticated semantic methods?
- Why do reasoning models fail when input length increases even below context limits?
- Can explicit numerical signals override learned linguistic defaults in fine-tuned models?
- Why do language models substitute parametric knowledge over retrieved context mid-reasoning?
- How do LLMs infer information that was explicitly censored?
- Can neural networks learn that A implies B in reverse?
- Why does monological training prevent models from overriding statistical priors?
- What makes some contexts learnable as rules versus requiring model retraining?
- How do training associations override context information in language models?
- How does the frame problem differ between symbolic and statistical reasoning systems?
- Can symbolic solvers rescue language models from logical reasoning failures?
- How do deterministic symbolic solvers improve the reliability of language model reasoning?
- Can symbolic solvers reliably replace LLM reasoning for logical tasks?
- What makes natural language reasoning more practical than formal languages for multi-framework codebases?
- Do computational systems need formal argument analysis for explainability?
- How does tool-based reasoning expand what language models can do?
- How does syntactic encoding relate to semantic feature representation?
- Can LLMs improve at metaphor if they handle decoupled semantics better?
- How does the symbol grounding problem apply to artificial language systems?
- Why do LLMs fail at semantic generalization despite grammatical accuracy?
- Do metaphors work by decoupling meaning from linguistic associations?
- Why do LLMs choose surface-order quantifier scope over contextually correct readings?
- What specific linguistic features cause LLMs to fail at trivial entailment?
- How does bidirectional entailment distinguish semantic equivalence from token similarity?
- Why does cross-text analogical reasoning fail when semantics decouple from symbols?
- Why does removing semantic content collapse reasoning in language models?
- What semantic information is necessary to preserve for sound LLM reasoning?
- Do language models learn surface patterns instead of underlying linguistic principles?
- How deeply are ideological structures represented in large language models?
- Does generalization frequency explain why models favor upward semantic movement?
- Is relevant knowledge encoded in LMs but not causally active in generation?
- What reveals the epistemic limits of language models?
- Do LLMs rely on surface heuristics instead of learning recursive grammar rules?
- Can complexity-stratified testing reveal whether LLMs understand grammatical structure?
- How does structural depth in sentences predict LLM annotation accuracy?
- What separates pattern matching from genuine language understanding?
- Do LLMs learn linguistic generalizations or just surface-level frequency patterns?
- Do LLMs learn surface patterns instead of genuine linguistic structure?
- How do pretrained language models represent inferential patterns versus lexical and positional cues?
- What other structural limits exist at the language-formal boundary?
- How do corpus statistics shape the abstraction hierarchy in language model representations?
- What geometric structure do language models actually use during inference?
- What are the stages of inference inside language models?
- Why does explicit theory injection work better than example-based learning for reasoning tasks?
- How does business logic specification replace annotated training datasets?
- Do reasoning languages like Prolog follow the same two-constraint transfer pattern?
- Why do open-source models trained on proprietary outputs still fail at reasoning?
- Why does NLI fine-tuning amplify frequency bias instead of teaching inference?
- Do reasoning models perform genuine logical evaluation or pattern matching?
- Why do models learn reasoning form instead of actual abstract inference?
- Why does distillation transfer reasoning patterns with few examples?
- What makes reasoning-specific post-training different from standard parameter scaling?
- How much does training composition affect syntactic versus reasoning performance?
- How do single training examples activate reasoning capabilities in language models?
- Why might rationales that predict common text patterns fail on hard novel reasoning?
- Can reasoning learned from language modeling actually transfer to knowledge-intensive domains?
- What kinds of reasoning tasks reveal the ceiling of text-only training?
- How do humans use associative reasoning without causal connections?
- Why does LLM compression eliminate causal grounding in conceptual representations?
- Can LLMs reason through semantics without understanding causal mechanisms?
- How does semantic association differ from mechanistic causal reasoning?
- Why do LLMs reason fluently about causality but lack causal rigor?
- Do LLMs show stronger reasoning about causality than about temporal ordering?
- Can autoregressive models learn faithful translation to logical representations without semantic loss?
- Why do diffusion LLM answer tokens converge in confidence long before reasoning stabilizes?
- What circuit mechanisms produce belief bias in syllogistic reasoning?
- Why do non-factive verbs and triggers both fool language models?
- Can we distinguish between semantic and symbolic reasoning in language models?
- How do humans and LMs differ on multi-hop reasoning?
- Where do humans and language models actually diverge in reasoning ability?
- What causes snowball errors to accumulate across reasoning steps in language models?
- Do sparse arithmetic circuits explain all language model reasoning abilities?
- Can small models solve complex tasks using externalized reasoning graphs?
- What neuroscience evidence suggests language networks are not optimized for reasoning?
- Can long-context models handle compositional reasoning requiring structured logic?
- Why do language models struggle with formal logical reasoning and joins?
- How does inductive reasoning from partial evidence enable hypothesis formation?
- Can language models reason without relying on surface level pattern matching?
- Can explicit optimal algorithms prevent reasoning model collapse at high complexity?
- How do recursive language models rethink where to store reasoning?
- What sparse mechanistic structures drive reasoning traces in language models?
- Why do reasoning models fail at learning hidden rules from sparse exceptions?
- Why do language models produce unfaithful chain of thought explanations?
- Can instance-adaptive reasoning happen without sequential token dependencies?
- Why do smaller LLMs fail at zero-shot argument scheme classification?
- What implicit premises do language models skip even with correct surface reasoning?
- Why do smaller models lose reasoning faithfulness more than larger models?
- Why do long-context language models struggle with compositional reasoning tasks?
- What evidence shows that reasoning chains encode token-level functional structure?
- Does reasoning efficiency transfer to tasks without ground truth dependency graphs?
- Can language models reason without relying on learned semantic patterns?
- Why do language models imitate reasoning form without abstract inference capability?
- Can reasoning chains work without logical validity?
- What makes symbolic operations different from general knowledge questions?
- Why does semantic decoupling specifically break LLM reasoning abilities?
- Why do LLMs generate logical forms without preserving semantic content?
- Can LLMs translate between natural language and formal logic faithfully?
- Why can LLMs interpret formal logic better than they generate it?
- How does structural complexity affect LLM performance differently than inferential complexity?
- How does semantic reasoning differ from symbolic reasoning in language models?
- Can LLMs reliably generate novel working architectures without structured representations?
- Do LLMs lack architectural scaffolding for compositional reasoning?
- What makes deductive reasoning so brittle in language models overall?
- How does structural complexity in sentences degrade LLM reasoning systematically?
- What makes structural logic correlate so strongly with contextual consistency?
- How does in-context semantic reasoning differ from symbolic reasoning in concept fusion?
- Why do recursive belief models require different training than logical derivation?
- Can language models perform purely symbolic reasoning when semantics are removed?
- Why does augmenting symbolic reasoning outperform replacing it entirely?
- Why do LLMs struggle to translate natural language into logical formalizations?
- Can language models perform genuine symbolic reasoning without semantic grounding?
- Does structured decomposition improve LLM reasoning in other compound tasks?
- Why do LLM descriptions of argument schemes work better than formal definitions for classification?
- Why does augmenting natural language with formal representations outperform full formalization?
- How do LLMs translate informal prose into logically correct formal specifications?
- Can we use LLM language without adopting LLM assumptions?
- How do LLMs lose information when translating natural language to formal logic?
- Why do LLMs fail at faithful autoformalisation of reasoning problems?
- How does neuro-symbolic design differ from pure LLM reasoning?
- How do semantic and symbolic reasoning capabilities differ in language models?
- Can language models translate theorems faithfully without semantic loss?
- Can explicit connectives compensate for missing intentional tracking in LLMs?
- Why do explicit discourse connectives help LLMs but implicit relations cause failures?
- How do embedding contexts like presupposition triggers affect LLM entailment reasoning?
- Why do LLMs perform better on explicit discourse connectives than implicit relations?
- Can LLMs compute how presuppositions project through embedded clauses?
- Why do explicit linguistic markers override semantic computation in models?
- Why do embeddings measure semantic association instead of task relevance?
- Why do unit-sphere spaces fail at distinguishing word order and negation?
- Why does explicit reasoning degrade passage reranking performance?
- What does pass@k reveal about base model reasoning capacity?
- Do LLMs compute scalar implicature differently across conversational contexts?
- Why do true and false LLM outputs use the same mechanism?
- Can language models acquire meaning from distributional patterns alone without joint attention?
- How does subject-predicate distinction emerge from formal linguistic analysis?
- How do LLMs compress specific expert knowledge into median abstraction?
- Why does knowledge storage separate from reasoning circuits in neural networks?
- Does more inference compute help reasoning models match specialized domain performance?
- Where does inference compute stop substituting for model capacity?
- Can latent reasoning in continuous space scale beyond supervised reasoning tasks?
- Can latent reasoning mechanisms and recursive tracking mechanisms be combined effectively?
- Do reflection tokens and symbolic tokens serve different roles in reasoning?
- Can continuous latent reasoning match discrete chain-of-thought without training modifications?
- Can latent reasoning achieve the same substitution without tokens?
- How does latent reasoning recursion compare to chain-of-thought reasoning?
- Is verbalized chain-of-thought necessary for language model reasoning?
- Do latent sequence vectors outperform per-token latent iterative computation for reasoning?
- Which game type reveals minimax reasoning in language models?
- Why do rare complex structures in training data harm LLM generalization?
- What latent mechanisms do LLMs use when they cannot execute iterative methods?
- Do distributed relational tasks consistently underperform local classification across NLP domains?
- Why do LLMs fail at iterative numerical computation in latent space?
- How do explicit reasoning traces help models construct valid syntactic trees?
- How do we verify that stated beliefs actually follow from underlying motifs?
- Why do language model reasoning chains look fluent when they deviate from the task?
- Do LLMs rely on surface statistical patterns instead of causal structure?
- Can LLM semantic representations exist without causally influencing their generation output?
- How can we probe LLM representations in channels that training did not target?
- Do language models need words to think or just latent structure?
- How should we rethink the symbolism versus connectionism debate in light of LLMs?
- What empirical evidence supports the Learning Law on real language models?
- What prevents LLM representations from causally influencing generation outputs?
- Can targeted activation steering surface latent reasoning in base models?
- How does an instruction-following LLM activate latent retrieval knowledge?
- Do base models contain latent reasoning that minimal training can unlock?
- Do base models truly possess latent reasoning capability?
- Does latent reasoning capability exist in base models before any training?
- How much training data is truly necessary to unlock latent model reasoning?
- Does the base model already contain latent reasoning capability?
- Can models possess latent reasoning capability that training signals fail to unlock?
- What mechanisms activate latent reasoning capabilities already present in base models?
- Can structured workflows unlock latent reasoning abilities that raw models don't show?
- Can minimal training signals unlock latent reasoning capability in base models?
- Can minimal training signals unlock reasoning already latent in pretrained representations?
- What latent reasoning capability do base models already possess before training?
- How does training data format shape whether models reason in parallel or sequentially?
- Does training data format shape which reasoning strategies LLMs develop?
- Why does hierarchical formal language training improve token efficiency more than natural language?
- Can standard next-token prediction capture complex multi-step human reasoning directly?
- Can you control LLM reasoning strategy without fine-tuning the model?
- Can models reason at inference without specialized internal training?
- Why do reasoning-optimized models show no resistance advantage on agreement tasks?
- Can we detect redundant reasoning steps during model inference instead of training?
- Does adding reasoning to models degrade other capabilities like rule inference?
- Can reinforcement learning close the gap between LLM reasoning and action?
- Can base models spontaneously produce reasoning traces without any RL training?
- What makes hierarchical reasoning effective for taxonomy induction?
- How does scaling and training data enable compositional behavior without symbolic mechanisms?
- Why does compositional reasoning fail to explain cross-domain transfer?
- How does evidence retrieval affect compositional reasoning in language models?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do large language models reason symbolically or …
Can large language models translate natural langua…
Do foundation models learn world models or task-sp…
Why do language models ignore information in their…
Does semantic grounding in language models come in…
Why do neural networks fail at compositional gener…
Do LLMs actually have world models or just facts?
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Can large language models translate natural language to logic faithfully?
This explores whether LLMs can convert natural language statements into formal logical representations without losing meaning. It matters because faithful translation is essential for any AI system that reasons formally or verifies specifications.
bidirectional semantic dependency: fails translating TO logic and reasoning FROM logic
-
Do foundation models learn world models or task-specific shortcuts?
When transformer models predict sequences accurately, are they building genuine world models that capture underlying physics and logic? Or are they exploiting narrow patterns that fail under distribution shift?
semantic associations are the heuristic mechanism
-
Why do language models ignore information in their context?
Explores why language models sometimes override contextual information with prior training associations, and whether providing more context can solve this problem.
same mechanism: parametric knowledge overrides in-context information
-
Does semantic grounding in language models come in degrees?
Rather than asking whether LLMs truly understand meaning, this explores whether grounding is actually a multi-dimensional spectrum. The question matters because it reframes the sterile understand/don't-understand debate into measurable, distinct capacities.
functional grounding through semantic associations explains why reasoning works within commonsense boundaries
-
Why do neural networks fail at compositional generalization?
Exploring whether the binding problem from neuroscience explains neural networks' inability to systematically generalize. The binding problem has three aspects—segregation, representation, and composition—each creating distinct failure modes in how networks handle structured information.
the binding problem may explain WHY semantic decoupling collapses reasoning: without compositional binding mechanisms, removing semantic content removes the only glue holding multi-step inference together; semantic associations serve as a substitute for genuine compositional binding
-
Do LLMs actually have world models or just facts?
The term 'world model' conflates two different capabilities: factual representation versus mechanistic understanding. Understanding which one LLMs actually possess matters for assessing their reasoning reliability.
semantic reasoning operates on factual world representation (Sense 1) but cannot perform mechanistic reasoning (Sense 2) when logic must override semantic priors
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
- Language models show human-like content effects on reasoning tasks
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering
- Can Large Language Models Reason and Optimize Under Constraints?
Original note title
llms are in-context semantic reasoners not symbolic reasoners — when semantics are decoupled reasoning collapses