How does bidirectional entailment distinguish semantic equivalence from token similarity?
This explores bidirectional entailment as a trick for telling 'these two answers mean the same thing' apart from 'these two answers just share words' — and what the corpus says about whether that trick actually holds up inside LLMs.
This explores bidirectional entailment — the idea that two statements are *semantically* equivalent if each one logically entails the other, no matter how differently they're worded — as a way to escape mere token overlap. The premise is elegant: surface similarity counts shared words, but mutual entailment is supposed to test shared meaning. "The cat sat on the mat" and "a feline rested upon the rug" share almost no tokens yet entail each other both ways; bidirectional entailment is meant to catch that. The corpus doesn't have a paper dedicated to the technique itself, but it has something more useful — a stack of evidence about why the entailment judgments such a method *depends on* are exactly where LLMs leak token-level signal back in.
The sharpest warning comes from Do language models really understand meaning or just surface frequency?: across math, translation, commonsense, and tool use, models systematically favor the higher-frequency surface form over a semantically identical rare paraphrase. That is the precise failure bidirectional entailment claims to neutralize — and it shows the model's own meaning-recognition is contaminated by statistical mass from pretraining. If your equivalence check is run *by* an LLM, it may quietly score the common phrasing as 'more entailed' for reasons that have nothing to do with meaning.
It gets worse when you look at how LLMs actually compute entailment. Do LLMs predict entailment based on what they memorized? shows attestation bias: models predict entailment based on whether the hypothesis looks like something they memorized, not on whether the premise supports it — they'll affirm entailment even from a *random* premise. And Why do embedding contexts confuse LLM entailment predictions? finds that constructions which should flip an entailment ('he pretended that…', 'she failed to…') get read as surface cues instead of structural operators. So the entailment signal a bidirectional method leans on is itself partly token similarity in disguise.
The deeper reason sits in Do large language models reason symbolically or semantically?: when you strip semantic content away from a reasoning task, performance collapses even with correct rules supplied. Models reason through semantic associations, not symbolic manipulation — which means 'does A entail B' is computed by association strength, not logical form. Bidirectional entailment asks for exactly the symbolic, direction-sensitive judgment these systems are weakest at.
The thing worth taking away: bidirectional entailment is a *good idea built on a shaky foundation*. Its logic is sound — meaning is relational, and Can language models learn meaning without engaging the world? argues LLMs really do encode meaning as relational structure compressed from text, which is why they can recognize paraphrase at all. But the same relational, frequency-shaped machinery that lets them spot equivalence also pulls their entailment verdicts back toward surface form. The method distinguishes meaning from tokens in principle; in practice it inherits whatever token bias the underlying entailment judge carries. If you want the technique to do real work, the corpus suggests the open problem is auditing the *judge*, not trusting the *trick*.
Sources 5 notes
LLMs show consistent preference for higher-frequency surface forms over semantically equivalent rare paraphrases across math, machine translation, commonsense reasoning, and tool calling. This suggests models track statistical mass from pretraining rather than meaning-recognition as their primary mechanism.
McKenna et al. (2023) identified attestation bias: LLMs predict entailment based on whether the hypothesis appears in training data, not whether the premise actually supports it. Random premise experiments show models maintain high entailment predictions when hypotheses are attested, proving they respond to memorized propositions rather than premise-hypothesis relationships.
LLMs treat presupposition triggers and non-factive verbs as surface cues rather than computing their opposite semantic effects on entailments. This structural failure persists across prompts and models, suggesting models rely on surface patterns instead of structural analysis.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
Research shows LLMs learn culturally situated discourse patterns by compressing relational structure from text, demonstrating that fluent language generation requires no external referents or embodied grounding.