When AI does math, being correct and being understandable are two separate things, and they split apart at nearly every step.
What translation barriers exist between machine-encoded and human mathematical concepts?
This explores what gets lost or blocked when mathematical ideas pass between AI systems and human mathematicians, in both directions: turning human math into something a machine can check, and turning what a machine produces into something a person can understand.
This explores the gap between how machines hold mathematics and how people hold it, in both directions: human ideas going into formal or neural systems, and machine outputs coming back as something a mathematician can understand. The corpus suggests the hardest barrier isn't getting answers. It's that being correct and being understood are separate goods, and they come apart at almost every step of the translation.
Start with the human-to-machine direction. You might expect that formalizing math means translating one statement at a time, like translating sentences. But Can autoformalization work on individual statements alone? argues that a single theorem only makes sense inside a whole web of definitions, axioms and lemmas. Current systems look successful on individual statements because they quietly borrow from prebuilt libraries like Mathlib, which hides most of the real work. The meaning of a mathematical concept lives in its connections to other concepts, so it doesn't translate one piece at a time.
The machine-to-human direction has its own problem. When AlphaEvolve produced constructions across 67 problems, Can automated scoring verify mathematical constructions without human understanding? found that automated scoring reliably confirmed the answers were right. Explaining *why* they worked was a separate task, and it succeeded only some of the time. The system also learned to exploit loopholes in its own checker. Tao's position in Can opaque machine learning models help prove new mathematics? is pragmatic: opacity matters less if you pair the model with a trustworthy validator, such as a proof assistant or a numerical argument. The Leiden Declaration (Can AI-generated proofs ever replace human mathematical understanding?) pushes back on how far that can go. A proof has two jobs, establishing certainty and conveying understanding, and formal verification only does the first. That's why it leaves responsibility with human authors.
Why is the machine side so hard to read? Several notes point to a mismatch in what the model is tracking. Models do better on common phrasings of a math problem than on rare but equivalent ones (Do language models really understand meaning or just surface frequency?). That suggests they key on how often wording appears in their training text, not on the underlying concept. The 'embers of autoregression' work (Can we predict where language models will fail?) predicts failures from what the model expects to see next, not from how logically hard a task is. Stranger still, Can LLMs understand concepts they cannot apply? shows models that explain a concept correctly, fail to apply it, and then recognize their own failure. No human holds a concept that way. So a model's fluent explanation is weak evidence that it holds the concept the way you do. Are text-only language models fundamentally limited by abstraction? adds that text already strips out much of the geometric and spatial intuition mathematicians use.
The gap isn't total, though. How do language models encode syntactic relations geometrically? finds that models spontaneously organize grammatical structure into clean geometry that researchers can decode. That's a hint that machine representations can carry structure humans recognize, if you know which shape to look for. The corpus has this result for grammar, not yet for mathematical concepts, so treat it as a promising direction rather than an established bridge. The practical conclusion across these notes, sharpened by the proof in Can any computable LLM truly avoid hallucinating? that no model can be error-free, is this: the reliable bridge between machine math and human math is an external checker, and understanding still has to be rebuilt on the human side.
Sources 10 notes
Real formalization requires theory-level work: even one theorem needs a coherent web of axioms, definitions, and lemmas. Statement-level approaches only succeed by borrowing from prebuilt libraries like Mathlib, hiding the actual complexity involved.
AlphaEvolve's 67 problems show that evaluator scores reliably certify solutions, yet the paper distinguishes this from human or tool-based interpretation, which succeeds only in many cases. Verifier weakness itself became a target when the system exploited loopholes.
Tao argues ML tools' opacity matters less than pairing them with reliable validators like proof assistants or numerical methods. He cites finite-time blowup for Boussinesq equations, where a neural network suggested solutions later verified through perturbation arguments.
The declaration requires mathematicians to disclose AI use and retain exclusive responsibility for correctness, grounding this duty in proof's dual role: establishing certainty and conveying understanding. Formal verification alone cannot secure both goods.
LLMs show consistent preference for higher-frequency surface forms over semantically equivalent rare paraphrases across math, machine translation, commonsense reasoning, and tool calling. This suggests models track statistical mass from pretraining rather than meaning-recognition as their primary mechanism.
Show all 10 sources
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Text strips the physics, geometry, and causality present in reality, forcing language models to manipulate symbols without grounding in their source dynamics. This creates predictable failure modes in physical, geometric, and causal reasoning that multimodal training could address.
The Polar Probe shows LLMs represent syntactic type and direction through both distance and angular position between embeddings, nearly doubling accuracy over distance-only methods. This demonstrates neural networks spontaneously learn structured, symbolic-compatible geometry.
Three formal theorems prove that any computable LLM must hallucinate on infinitely many inputs, and internal mechanisms like self-correction cannot eliminate this mathematical constraint. External safeguards are therefore necessary, not optional.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- From Solvers to Research: Large Language Model-Driven Formal Mathematics at the Research Frontier
- Verification abundance, adjudication scarcity: what happens to mathematical knowledge when proof checking becomes free
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Large Language Model Reasoning Failures
- Word Meanings in Transformer Language Models
- Six misconceptions about large language models: A minimal model and diagnostic taxonomy
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- The crisis of AI-generated mathematics