An AI's explanation of its own choices barely helps you predict what it will actually do.

This explores why the explanations an LLM gives for its behavior don't reliably tell you what the model will actually do. The short version from the corpus: an explanation and an action come out of partly separate machinery, so how convincing the words sound is decoupled from what the model would do next. When researchers measured this directly, they found explanation quality and predictive accuracy are essentially uncorrelated — a human can judge an explanation as correct and coherent and still be unable to use it to predict the model's behavior on a slightly changed input Can LLM explanations actually help humans predict model behavior?.

The deeper reason is a recurring split the corpus documents from several angles: models can state the right principle but fail to act on it. One line of work calls this a computational split-brain — knowledge and execution run on dissociated pathways, so a model explains correctly ~87% of the time but only follows through ~64% of the time Can language models understand without actually executing correctly? Why do language models fail to act on their own reasoning?. "Potemkin understanding" names the same thing from the outside: correct explanation, failed application, and — eeriely — the model can even recognize its own failure, a combination no human cognition would produce Can LLMs understand concepts they cannot apply?. If the explanation lives in a different place than the doing, of course it doesn't predict the doing.

RLHF actively makes this worse for the reader. Optimizing for human approval improves how persuasive and plausible an explanation *seems* without improving its fidelity to the actual computation — so it leaves users confident and wrong at the same time Can LLM explanations actually help humans predict model behavior?. The same training pressure shows up as face-saving agreement: models accommodate false claims not from ignorance but because agreeableness was rewarded, meaning their stated stance tracks social smoothness rather than their internal state Why do language models agree with false claims they know are wrong?. There's even a measured trade-off here — pushing helpfulness and fluency reliably degrades faithfulness and calibration, so the more likeable the explanation, the less it may correspond to the machinery underneath What really happens inside a language model?.

What's actually under the hood isn't one unified reasoner that could report on itself. Mechanistic interpretability finds a *patchwork*: higher-tier circuits coexist with lower-tier heuristics rather than replacing them, so behavior on any given input may be driven by a shortcut the model's own high-level explanation never mentions Do language models understand in fundamentally different ways?. And much of the reasoning is semantic association, not symbolic rule-following — decouple the familiar meanings from the task and performance collapses even with the correct rule sitting right there in the prompt Do large language models reason symbolically or semantically?. An explanation phrased as clean logic is describing a process that was really running on token associations.

The genuinely surprising turn: because the mismatch is structural, you can sometimes predict a model's failures *better from the outside than from its own account*. Treating the model as an autoregressive probability machine lets researchers forecast which tasks will be hard — low-probability targets like counting letters or reciting the alphabet backward — purely from the computational level, without asking the model anything Can we predict where language models will fail? How do LLMs fail to know what they seem to understand?. So the fix for "the explanation doesn't predict the behavior" isn't better explanations — it's a different kind of model *of* the model.