Why does an AI answer logic puzzles by vibe and association instead of actually following the rules — even when the rules are right there?
Why do LLMs reason semantically rather than symbolically?
This explores why LLMs lean on what words mean, and what sounds plausible, instead of following formal logical rules, and whether that habit is a flaw to fix or a strength to work with.
This explores why LLMs lean on meaning and plausibility instead of mechanically following logical rules, and what that means for anyone who wants them to reason reliably. The clearest evidence is simple: give a model a logic problem where the words have been stripped of their usual meaning, such as nonsense terms or facts that contradict common sense, and its performance collapses, even when the correct rules are written right there in the prompt Do large language models reason symbolically or semantically?. The model isn't really manipulating symbols. It draws on associations learned in training about how things usually go together, so its reasoning is tied to the kinds of situations it has seen before.
The surprising part is that the symbolic machinery does exist inside the model. Interpretability work on syllogisms ("All A are B, all B are C, so...") found a real three-step internal circuit that handles the logical form regardless of content, and it shows up across different architectures How do language models perform syllogistic reasoning internally?. Other attention heads carrying world knowledge run alongside it and pull the conclusion toward what sounds believable instead of what logically follows. That contamination gets stronger as models get bigger. So scale doesn't push models toward cleaner logic. It can make the semantic pull stronger. This fits a broader picture in which a model's deeper, principled understanding sits next to shallow shortcuts instead of replacing them, giving a patchwork of abilities Do language models understand in fundamentally different ways?. It also helps explain a stranger failure: a model can correctly explain a concept, fail to apply it, and then recognize that it failed Can LLMs understand concepts they cannot apply?. Knowing the rule and using the rule seem to run on separate tracks.
Why semantic instead of symbolic? Part of the answer is where the reasoning actually happens. The evidence suggests it mostly unfolds in the model's hidden internal states, and the chain-of-thought text you see is only a partial window onto that process Where does LLM reasoning actually happen during generation?. Those internal states are built from meaning-rich representations. Nothing in that space works like a clean variable you can swap out. Some researchers lean into this instead of fighting it. Meta's Large Concept Model reasons over whole-sentence meanings instead of individual tokens, and treats semantic abstraction as the main design idea Can reasoning happen at the sentence level instead of tokens?.
The practical response isn't to force LLMs into pure logic. Fully translating a problem into formal symbols tends to lose information, while plain language lacks structure. Adding just a few symbolic elements to natural language beats both, with reported gains of 4-8% Why does partial formalization outperform full symbolic logic?. Another option is a division of labor: the LLM translates the problem into a formal representation, a deterministic solver does the actual logic, and the solver's error messages catch translation mistakes better than the model checking its own work Can symbolic solvers fix how LLMs reason about logic?. Lighter versions of the same idea also help. You can wrap the model in an explicit algorithm that controls each step Can algorithms control LLM reasoning better than LLMs alone?, or prompt it to state the unstated premises an argument depends on Can structured argument prompts make LLM reasoning more rigorous?.
The takeaway you might not expect: semantic reasoning isn't the model failing to reach logic. It's a different engine with its own failure pattern. On deep, multi-step problems that pattern shows up as wandering instead of systematic search, and success drops off exponentially as problems get deeper Why do reasoning LLMs fail at deeper problem solving?. The same tie to familiar meaning may also explain why models struggle with the kind of creative reasoning that requires breaking from conventional framings Can LLMs reason creatively beyond conventional problem-solving?. The best current designs treat semantics as the model's strength and add structure from outside where rigor matters.
Sources 12 notes
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
LLMs implement a content-independent three-stage reasoning mechanism—recitation, middle-term suppression, mediation—that works across architectures. However, additional attention heads encoding world knowledge systematically bias conclusions toward semantically plausible rather than logically valid answers, with contamination increasing at larger scales.
Mechanistic interpretability reveals conceptual understanding (features as directions), state-of-world understanding (factual connections), and principled understanding (compact circuits). Crucially, higher tiers coexist with lower-tier heuristics rather than replacing them, creating a patchwork of capabilities.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Evidence from CoT faithfulness tests, feature steering, and layer analysis suggests latent-state dynamics drive reasoning, while surface chain-of-thought serves as a partial interface. Hidden reasoning processes should be the default focus of study.
Show all 12 sources
Meta's Large Concept Model operates on sentence embeddings rather than tokens, reasoning in a language-agnostic space before decoding to any target language. This hierarchical approach with paragraph-level planning produces more coherent output than flat token generation.
QuaSAR and Logic-of-Thought both achieve 4-8% accuracy gains by enriching natural language with selective symbolic elements rather than replacing it. Full formalization loses semantic information; pure language lacks structure. Augmentation preserves both.
Logic-LM divides cognitive labor by having LLMs formulate symbolic representations while deterministic solvers execute inference and provide machine-verifiable error messages. This structured feedback loop catches translation errors better than LLM self-critique, improving faithful reasoning without requiring perfect formalization.
LLM Programs embed LLMs within explicit algorithms that manage control flow and state, presenting only step-specific context to each LLM call. This information hiding addresses capability and context window limits while treating complex reasoning as modular, debuggable sub-tasks.
Applying Toulmin's argument model as explicit prompting steps (CQoT) improves LLM reasoning by forcing models to identify warrants and backing rather than skipping implicit premises. The method catches failures that standard chain-of-thought prompting allows.
Current reasoning models lack the three properties of systematic exploration: validity, effectiveness, and necessity. This causes success probability to drop exponentially with problem depth, making medium problems solvable but deep problems catastrophically harder.
Research identifies combinational, exploratory, and transformational reasoning as distinct creative modes grounded in cognitive science. Existing LLM reasoning methods address only conventional problem-solving, leaving creative paradigms unaddressed and potentially explaining diversity collapse in ideation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
- Reasoning LLMs are Wandering Solution Explorers
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Large Language Model Reasoning Failures
- LLM Reasoning Is Latent, Not the Chain of Thought