An AI can describe a rule almost perfectly but still break it when actually using it — why would knowing and doing split apart?

This explores whether an LLM that has every fact and rule it needs will actually carry the rule out correctly, or whether knowing a rule and applying it are two separate abilities. The corpus says that complete information is often not enough. One study finds that models can explain a principle correctly about 87% of the time but act on it correctly only about 64% of the time. The authors call this a 'split-brain' pattern, where the pathway that describes a rule and the pathway that carries it out don't fully connect Can language models understand without actually executing correctly?. They argue the gap comes from how the model is built, not from missing knowledge, so adding more information to the prompt won't close it.

The reason may be that LLMs don't follow rules the way a logic engine does. When researchers strip away familiar meaning, for example by replacing real-world concepts with arbitrary symbols while keeping the correct rules in the prompt, performance collapses Do large language models reason symbolically or semantically?. The models seem to lean on associations they learned in training rather than working through the rule step by step. A rule applied to familiar content can look faithful, while the same rule applied to unfamiliar content falls apart. A related result: models write logic that looks correct but means the wrong thing, with errors clustering around words like 'all' versus 'some' and around which part of a sentence a qualifier applies to Can large language models translate natural language to logic faithfully?. Even turning a decision rule into precise form is a weak point.

There is a less obvious trap. Setting temperature to zero makes a model give the same answer every time, which can look like reliable rule-following. But that answer is still just one sample from the model's range of possible outputs, so it can be wrong every time Does setting temperature to zero actually make LLM outputs reliable?. Consistency is not the same as faithfulness. 'Complete information' also has limits when the rules contradict each other. Work on learning at test time finds that models can't settle conflicts between rules on their own, because the right choice depends on context outside the system, so a human has to step in Can LLMs learn reliably at test time without human oversight?.

The practical response in the corpus is to stop asking the model to be the rule-follower. One approach wraps LLM calls inside an ordinary program that handles the control flow and shows each call only the context relevant to its step Can algorithms control LLM reasoning better than LLMs alone?. Another puts simple automated checks around an LLM judge, such as running clear-cut checks first and planting known cases as alarms, so the model's judgment is never trusted to enforce the rules itself Can deterministic checks protect LLM judges from failure?. The pattern that emerges: let LLMs propose and interpret, and let ordinary code apply the rule.

The corpus doesn't contain a direct benchmark of LLMs applying formal decision rules (such as eligibility criteria or decision tables) with all inputs given. The answer here is pieced together from nearby evidence rather than from one study that tests exactly this.