Why does AI-generated design talk convince us even when the actual build doesn't match what it describes?

This explores why AI output that *talks* like good design can win us over even when the actual build falls short. The corpus has a direct measurement of the gap. In a benchmark of generative UI tools, about a quarter of the design rationales the tools stated were never implemented. For functional requirements the figure rose to roughly a third, and the tools recognized only half the UX principles written into the prompts Do generative UI tools actually implement their stated design rationales?. The name the researchers chose, 'design theater', points to the problem: the explanation of the design shows up reliably, and the design itself often doesn't.

This isn't laziness, and it isn't unique to UI tools. Language models show a broader split between explaining and doing. In one study models stated the correct principle about 87% of the time but applied it correctly only about 64% of the time. The researchers attribute this to separate pathways for instructions and for execution, not to missing knowledge Can language models understand without actually executing correctly?. Reasoning models show the same pattern from the other side. When they 'collapse' on hard problems, they often still know the algorithm and simply can't carry out the many steps in plain text. Give them tools and they get past the supposed limit Are reasoning model collapses really failures of reasoning?. The rationale is real knowledge. It just isn't attached to the output.

So why does the rationale persuade? One answer is that the *form* of good reasoning carries much of the weight, for models and readers alike. Chain-of-thought examples that were logically invalid improved model performance almost as much as valid ones: the model was picking up the shape of reasoning, not the inference Does logical validity actually drive chain-of-thought gains?. Design language works the same way. A tidy rationale citing hierarchy, affordances and accessibility looks like the output of a careful process whether or not one took place. A separate audit found that LLMs reach for logical appeals and numbers in almost every conversation. That makes their claims sound objective and gives them authority they haven't earned Do LLMs persuade users more often than humans do?. A confident design writeup is that same habit applied to a mockup.

The reader's position adds to the effect. Users often can't fully say what they wanted until they see options Why can't users articulate what they want from AI?. When the AI hands back a fluent rationale, it can quietly stand in for the user's own unformed intent: 'yes, that's what I meant.' The user then checks the explanation against their hopes instead of checking the artifact against the spec.

The practical lesson is to judge the artifact, not the account of it. Work done in code is useful here because code can be run and inspected, so claims can be verified Can code serve as the operational substrate for agent reasoning?. Simple awareness also helps. In a study of political persuasion, a short warning that LLMs can be prompted to persuade cut belief change roughly in half without lowering trust in AI Can a simple warning reduce how much LLMs persuade people?. Design review would plausibly benefit from the same kind of reminder: a well-written rationale is something the model produces easily, and it is not evidence that the design was built.