Why does partial formalization outperform full symbolic logic?
Explores whether injecting some symbolic structure into natural language reasoning works better than completely formalizing problems. Matters because it could reveal the optimal balance between structure and semantics for LLM reasoning.
Two independent approaches converge on the same principle: injecting partial symbolic structure into natural language context outperforms both pure NL reasoning and full symbolic formalization. The key is augmentation, not replacement.
QuaSAR (Quasi-Symbolic Abstract Reasoning) guides LLMs through four steps: (1) Abstraction — identify relevant predicates, variables, constants; (2) Formalization — reformulate using a mix of symbols and NL; (3) Explanation — solve using quasi-symbolic representations; (4) Answering — extract the final answer. The model formalizes only what's relevant, keeping everything else in NL. Result: up to 8% accuracy improvement on MMLU-Redux and GSM-Symbolic, with enhanced robustness on adversarial variations.
Logic-of-Thought (LoT) takes a different path to the same destination: extract propositional logic from the input, expand via logical reasoning laws (double negation, contraposition, transitivity), translate the expanded logic back to NL, then inject as additional context alongside the original prompt. Result: +4.35% on ReClor (with CoT), +3.52% on RuleTaker (with CoT+SC), +8% on ProofWriter (with ToT).
Both approaches solve the same problem differently. Full neuro-symbolic methods (Logic-LM, LINC, SatLM) translate the ENTIRE problem to formal logic, which inevitably loses information — the LoT paper documents specific cases where "Harry is a person" and "Walden is a book" are lost during extraction, causing symbolic solvers to fail. QuaSAR and LoT avoid this by keeping the original NL context intact and adding formal elements as enrichment.
The theoretical grounding is illuminating. QuaSAR draws on Kitcher's unificationist account of explanation: explanations work by subsuming observations under recurring argument patterns through abstraction. Replacing concrete entities with abstract symbols creates reusable reasoning patterns — the same pattern can explain why objects fall AND why celestial bodies attract. This is partial formalization as cognitive tool, not as logical translation.
Since Can large language models translate natural language to logic faithfully?, full formalization is a dead end. Since Do large language models reason symbolically or semantically?, removing semantics breaks reasoning. The partial approach threads the needle: add enough structure to bypass content bias while preserving enough semantics for the model to reason.
Inquiring lines that read this note 72
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does explicit structure improve LLM reasoning on complex tasks?- What structural constraints matter more than model depth for CF?
- Would hybrid systems combining LLMs with symbolic solvers overcome the retraction limitation?
- How does structural complexity affect LLM performance differently than inferential complexity?
- Do LLMs lack architectural scaffolding for compositional reasoning?
- Does structured decomposition improve LLM reasoning in other compound tasks?
- Can LLMs successfully translate natural language into formal solver specifications?
- Can symbolic solvers reliably replace LLM reasoning for logical tasks?
- How does neuro-symbolic design differ from pure LLM reasoning?
- How does the frame problem differ between symbolic and statistical reasoning systems?
- Can symbolic solvers rescue language models from logical reasoning failures?
- Why do language models struggle with formal logical reasoning and joins?
- Which constraint types do reasoning models handle best?
- Why does structured stochasticity help reasoning more than naive randomness?
- Can problem structure and representation format be mismatched intentionally?
- How should emotional states integrate into symbolic reasoning systems?
- Can we distinguish between semantic and symbolic reasoning in language models?
- What are the five structure types and which tasks does each one suit best?
- Which knowledge structure types best fit different query types?
- What graph structures would enable transformational creative reasoning in LLMs?
- Do reasoning languages like Prolog follow the same two-constraint transfer pattern?
- What makes symbolic operations different from general knowledge questions?
- How does semantic reasoning differ from symbolic reasoning in language models?
- What makes structural logic correlate so strongly with contextual consistency?
- How does in-context semantic reasoning differ from symbolic reasoning in concept fusion?
- Why do format and structure matter more than actual content in reasoning?
- Why does augmenting symbolic reasoning outperform replacing it entirely?
- What makes language an effective parameterization for procedural knowledge?
- How do progressive abstraction chains differ from branching reasoning topologies?
- What makes natural language reasoning more practical than formal languages for multi-framework codebases?
- How do semantic and symbolic reasoning capabilities differ in language models?
- What makes structured informal reasoning preferable to full formalization?
- Can LLMs improve at metaphor if they handle decoupled semantics better?
- Why do LLMs generate logical forms without preserving semantic content?
- Can LLMs translate between natural language and formal logic faithfully?
- How does the distance between natural language and formal notation affect translation accuracy?
- How does structural complexity in sentences degrade LLM reasoning systematically?
- Can language models perform purely symbolic reasoning when semantics are removed?
- Why do LLMs struggle to translate natural language into logical formalizations?
- How do LLMs translate informal prose into logically correct formal specifications?
- How do LLMs lose information when translating natural language to formal logic?
- Why does GraphRAG prioritize corpus completeness while LogicRAG prioritizes query adaptivity?
- How does MaxSim reranking differ from structural verification at the token level?
- Why do LLMs struggle with negation and exception handling?
- Why can't LLMs reason from first principles or initial commitments?
- Why can LLMs interpret formal logic better than they generate it?
- What structural framework prevents LLM explanations from becoming just plausible fiction?
- Why do LLMs reason semantically rather than symbolically?
- Why do only context-sensitive formal languages transfer effectively to natural language?
- Why does augmenting natural language with formal representations outperform full formalization?
- How do deterministic symbolic solvers improve the reliability of language model reasoning?
- Why do semi-formal templates improve verification accuracy over unstructured reasoning?
- Can partial formal verification work without full formalization of language semantics?
- Why does moving verifier synthesis to the LLM extend verification beyond math and code domains?
- Can completeness scaffolding work for domains beyond code verification?
- Can natural language to formal logic translation ever be fully trustworthy?
- How do completeness scaffolds force explicit step-by-step derivation?
- Can completeness scaffolding substitute for actual code execution in reasoning?
- What types of math proofs benefit most from proof-by-contradiction framing?
- Why does formalizing the Kepler conjecture cost eleven years of work?
- Why does formalizing obvious steps take longer than formalizing key insights?
- Can proof assistants verify the full lattice construction argument formally?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can large language models translate natural language to logic faithfully?
This explores whether LLMs can convert natural language statements into formal logical representations without losing meaning. It matters because faithful translation is essential for any AI system that reasons formally or verifies specifications.
full formalization fails; partial avoids the failure
-
Do large language models reason symbolically or semantically?
Can LLMs follow explicit logical rules when those rules contradict their training knowledge? Testing whether reasoning operates independently of semantic associations reveals what computational mechanisms actually drive LLM multi-step inference.
preserving semantics is necessary; adding partial structure is sufficient
-
Can symbolic solvers fix how LLMs reason about logic?
LLMs excel at understanding natural language but fail at precise logical inference. Can pairing them with deterministic symbolic solvers—using solver feedback to refine attempts—overcome this fundamental weakness?
full symbolic offloading is the other extreme; this is the productive middle ground
-
Can structured argument prompts make LLM reasoning more rigorous?
Does requiring language models to explicitly check warrants, backing, and rebuttals—rather than reasoning freely—improve reasoning quality and catch failures that standard step-by-step prompting misses?
CQoT is another form of partial structure injection; both add formal elements without full formalization
-
Do formal language prototypes improve reasoning across different domains?
Can training language models on abstract reasoning patterns in Prolog and PDDL help them generalize to new reasoning tasks? This tests whether shared logical structures underlie seemingly different problem domains.
ProtoReasoning takes the augmentation approach at the training level: Prolog/PDDL prototypes alongside natural language, not replacing it; the 4-6% cross-domain improvement confirms that partial formal augmentation transfers better than full formalization, extending the augmentation principle from inference-time (this note) to training-time
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions
- Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering
- Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
- Faithful and Robust LLM-Driven Theorem Proving for NLI Explanations
- Large Language Models as Planning Domain Generators
- Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models
- From Language to Logic: A Bi-Level Framework for Structured Reasoning
Original note title
partial symbolic abstraction preserves information completeness that full formalization loses — augmentation outperforms replacement for logical reasoning