What stories do people quietly tell themselves about what's happening inside an AI when they chat with it?

This explores the working theories ordinary users carry about what's going on inside an AI system, and how those theories change their behavior. The corpus has a gap here: it says a lot about *why* mental models matter and *where* they go wrong, but it doesn't catalogue the specific folk theories users hold ("it looks things up," "it remembers me," "it thinks"). If that catalogue is what you're after, it isn't here yet. What the corpus does offer is a map of the dimensions those models vary along and the places they predictably break.

The clearest measured structure comes from research on dialogue agents. When people size up a conversational AI, their impressions sort into three factors: how competent it seems (about half of the variation), how human-like it seems (about a third), and how flexibly it communicates How do users mentally model dialogue agent partners?. So users judge AI the way they would judge a conversation partner, not a database. That explains why the line between *designed* human-likeness and *perceived* human-likeness matters: if users treat the system as a person, you have to ask whether the design invited that or the user projected it, because those call for different fixes (redesign vs. education) Who bears responsibility when AI seems human-like?.

The more surprising finding is about what users' models *leave out*. Most people picture a two-party exchange, me and the AI, but a third party sits invisibly between them. Developers write system prompts that users can't see, and small edits to those prompts can abruptly change how the model behaves Who actually controls what AI systems say?. Users also tend to read confident wording as a sign of reliability, even though models' reports about their own knowledge are unstable and shift under conversational pressure How well do language models understand their own knowledge?. A related blind spot points back at the user: fluent output, unclear credit for who did what, and hidden processing combine so that people credit themselves with competence that actually came from the AI How do AI tools trick users into overestimating their own skills?.

Why does any of this matter? Nielsen Norman Group found that understanding AI's basic mechanics predicts effective use better than how often someone uses it, and linked inaccurate mental models to low workplace adoption Does understanding AI mechanics matter more than using it often?. One common mismatch is about intent. Users often expect the AI to grasp what they mean, but intent actually firms up gradually while you work, and the AI can't see that process How do users actually form intent when prompting AI systems?. Systems also tend to satisfy the literal wording of a request rather than its purpose Why do AIs keep gaming rewards instead of serving intent?. Some research flips the problem around: instead of hoping users guess correctly, build ways for them to check an agent's plans and actions, so that accurate mental models grow out of verification rather than speculation How can users verify what an AI agent actually plans to do? What makes an AI a true thought partner, not just a tool?.

The takeaway you might not have expected: the most consequential gaps in users' mental models may not be about how the neural network works at all. They're about who else is in the conversation, which signals of confidence can be trusted, and whose skill the output actually reflects.