Why does partial formalization outperform full symbolic logic?
Explores whether injecting some symbolic structure into natural language reasoning works better than completely formalizing problems. Matters because it could reveal the optimal balance between structure and semantics for LLM reasoning.
Two independent approaches converge on the same principle: injecting partial symbolic structure into natural language context outperforms both pure NL reasoning and full symbolic formalization. The key is augmentation, not replacement.
QuaSAR (Quasi-Symbolic Abstract Reasoning) guides LLMs through four steps: (1) Abstraction — identify relevant predicates, variables, constants; (2) Formalization — reformulate using a mix of symbols and NL; (3) Explanation — solve using quasi-symbolic representations; (4) Answering — extract the final answer. The model formalizes only what's relevant, keeping everything else in NL. Result: up to 8% accuracy improvement on MMLU-Redux and GSM-Symbolic, with enhanced robustness on adversarial variations.
Logic-of-Thought (LoT) takes a different path to the same destination: extract propositional logic from the input, expand via logical reasoning laws (double negation, contraposition, transitivity), translate the expanded logic back to NL, then inject as additional context alongside the original prompt. Result: +4.35% on ReClor (with CoT), +3.52% on RuleTaker (with CoT+SC), +8% on ProofWriter (with ToT).
Both approaches solve the same problem differently. Full neuro-symbolic methods (Logic-LM, LINC, SatLM) translate the ENTIRE problem to formal logic, which inevitably loses information — the LoT paper documents specific cases where "Harry is a person" and "Walden is a book" are lost during extraction, causing symbolic solvers to fail. QuaSAR and LoT avoid this by keeping the original NL context intact and adding formal elements as enrichment.
The theoretical grounding is illuminating. QuaSAR draws on Kitcher's unificationist account of explanation: explanations work by subsuming observations under recurring argument patterns through abstraction. Replacing concrete entities with abstract symbols creates reusable reasoning patterns — the same pattern can explain why objects fall AND why celestial bodies attract. This is partial formalization as cognitive tool, not as logical translation.
Since Can large language models translate natural language to logic faithfully?, full formalization is a dead end. Since Do large language models reason symbolically or semantically?, removing semantics breaks reasoning. The partial approach threads the needle: add enough structure to bypass content bias while preserving enough semantics for the model to reason.
Inquiring lines that read this note 65
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
When does architectural design matter more than raw model capacity? How effectively do deterministic tools improve language model reasoning on formal tasks?- How does the frame problem differ between symbolic and statistical reasoning systems?
- Can symbolic solvers rescue language models from logical reasoning failures?
- Would hybrid systems combining LLMs with symbolic solvers overcome the retraction limitation?
- Which constraint types do reasoning models handle best?
- How do deterministic symbolic solvers improve the reliability of language model reasoning?
- Why do semi-formal templates improve verification accuracy over unstructured reasoning?
- Can partial formal verification work without full formalization of language semantics?
- Why does moving verifier synthesis to the LLM extend verification beyond math and code domains?
- Can symbolic solvers reliably replace LLM reasoning for logical tasks?
- Can completeness scaffolding substitute for actual code execution in reasoning?
- What makes natural language reasoning more practical than formal languages for multi-framework codebases?
- Can completeness scaffolding work for domains beyond code verification?
- Why does formalizing the Kepler conjecture cost eleven years of work?
- What are the five structure types and which tasks does each one suit best?
- Which knowledge structure types best fit different query types?
- Can we distinguish between semantic and symbolic reasoning in language models?
- Why do language models struggle with formal logical reasoning and joins?
- What graph structures would enable transformational creative reasoning in LLMs?
- What makes tarot and periodic tables resist meaningful scientific integration?
- How do progressive abstraction chains differ from branching reasoning topologies?
- Do reasoning languages like Prolog follow the same two-constraint transfer pattern?
- Why does structured stochasticity help reasoning more than naive randomness?
- What makes symbolic operations different from general knowledge questions?
- Why do LLMs generate logical forms without preserving semantic content?
- Can LLMs translate between natural language and formal logic faithfully?
- How does the distance between natural language and formal notation affect translation accuracy?
- Why can LLMs interpret formal logic better than they generate it?
- How does structural complexity affect LLM performance differently than inferential complexity?
- How does semantic reasoning differ from symbolic reasoning in language models?
- Do LLMs lack architectural scaffolding for compositional reasoning?
- How does structural complexity in sentences degrade LLM reasoning systematically?
- What makes structural logic correlate so strongly with contextual consistency?
- How does in-context semantic reasoning differ from symbolic reasoning in concept fusion?
- Why do format and structure matter more than actual content in reasoning?
- Can language models perform purely symbolic reasoning when semantics are removed?
- Why does augmenting symbolic reasoning outperform replacing it entirely?
- Why do LLMs struggle to translate natural language into logical formalizations?
- Does structured decomposition improve LLM reasoning in other compound tasks?
- Can LLMs successfully translate natural language into formal solver specifications?
- What makes language an effective parameterization for procedural knowledge?
- Why does augmenting natural language with formal representations outperform full formalization?
- How do LLMs translate informal prose into logically correct formal specifications?
- How do LLMs lose information when translating natural language to formal logic?
- How does neuro-symbolic design differ from pure LLM reasoning?
- How do semantic and symbolic reasoning capabilities differ in language models?
- What makes structured informal reasoning preferable to full formalization?
- Why does comparison reasoning generalize better than composition reasoning?
- What makes structured stochasticity more effective than unstructured randomness in reasoning?
- Why do LLMs struggle with negation and exception handling?
- Why can't LLMs reason from first principles or initial commitments?
- What structural framework prevents LLM explanations from becoming just plausible fiction?
- What types of math proofs benefit most from proof-by-contradiction framing?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can large language models translate natural language to logic faithfully?
This explores whether LLMs can convert natural language statements into formal logical representations without losing meaning. It matters because faithful translation is essential for any AI system that reasons formally or verifies specifications.
full formalization fails; partial avoids the failure
-
Do large language models reason symbolically or semantically?
Can LLMs follow explicit logical rules when those rules contradict their training knowledge? Testing whether reasoning operates independently of semantic associations reveals what computational mechanisms actually drive LLM multi-step inference.
preserving semantics is necessary; adding partial structure is sufficient
-
Can symbolic solvers fix how LLMs reason about logic?
LLMs excel at understanding natural language but fail at precise logical inference. Can pairing them with deterministic symbolic solvers—using solver feedback to refine attempts—overcome this fundamental weakness?
full symbolic offloading is the other extreme; this is the productive middle ground
-
Can structured argument prompts make LLM reasoning more rigorous?
Does requiring language models to explicitly check warrants, backing, and rebuttals—rather than reasoning freely—improve reasoning quality and catch failures that standard step-by-step prompting misses?
CQoT is another form of partial structure injection; both add formal elements without full formalization
-
Do formal language prototypes improve reasoning across different domains?
Can training language models on abstract reasoning patterns in Prolog and PDDL help them generalize to new reasoning tasks? This tests whether shared logical structures underlie seemingly different problem domains.
ProtoReasoning takes the augmentation approach at the training level: Prolog/PDDL prototypes alongside natural language, not replacing it; the 4-6% cross-domain improvement confirms that partial formal augmentation transfers better than full formalization, extending the augmentation principle from inference-time (this note) to training-time
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions
- Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering
- Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
- Faithful and Robust LLM-Driven Theorem Proving for NLI Explanations
- Large Language Models as Planning Domain Generators
- Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models
- From Language to Logic: A Bi-Level Framework for Structured Reasoning
Original note title
partial symbolic abstraction preserves information completeness that full formalization loses — augmentation outperforms replacement for logical reasoning