When an app says 'everything stays on your device,' do people picture that correctly — or just trust the label?
How do users mentally model local-only processing in AI systems?
This explores how people picture what happens to their data when an AI system says it processes everything on-device, and whether that picture matches reality.
This explores how people picture where their data goes when an AI system says it runs only on their own device, and whether that picture is accurate. The honest answer first: none of these retrieved notes studies how users understand on-device or local-only AI. No paper here asks people what they think 'local' means, so this can't be answered directly. What the corpus does offer is a set of nearby findings that explain why such mental models are likely to be shaky.
The first problem is that users have little to hold onto. With conventional software, people learn a fixed interface and can reason about where things live. AI context isn't fixed in that way. It is a shifting mix of prompt, conversation history, retrieved data and hidden state that users can't absorb the way they learn a traditional UI How does AI context differ from conventional software context?. If people can't track *what* the system is drawing on, they have little basis for judging *where* that processing happens. A 'local-only' label asks users to trust a claim about something they can't see.
The second problem is that people already misread what AI pipelines do. One line of work names 'pipeline opacity' as one of four mechanisms that lead users to credit themselves for what the AI produced, and finds that these mechanisms amplify each other How do AI tools trick users into overestimating their own skills?. That research is about skill rather than privacy, but the lesson carries over: when the steps between input and output are hidden, users fill the gap with a comfortable story. Separately, users tend to trust confident AI output whether or not it's accurate How well do language models understand their own knowledge?. That suggests a confident 'your data never leaves this device' message would be believed without much checking.
This matters because of what local processing can still collect. AI systems can read cognitive state from behavioral signals such as gaze, typing hesitation and interaction speed. The same signals that let a system time its help well also make manipulative profiling possible Can AI systems read cognitive state from interaction patterns alone?. A user who thinks 'local means private' may relax about data leaving the device while missing the more subtle point: a lot can be inferred about them on the device itself, and what happens to those inferences is a separate question.
What's missing is user research on how people actually picture 'local' AI, such as interviews, perception studies or comparisons between privacy labels. If you're exploring this question, that gap is itself the finding. The corpus has strong material on why AI systems are hard to understand, and almost nothing on this specific privacy promise.
Sources 4 notes
AI interactions operate on a substrate of constantly shifting context—prompt, history, retrieved data, hidden state—that users cannot internalize like traditional UIs. This structural mutability demands a new design discipline centered on context engineering rather than interface design.
Attribution ambiguity, fluency illusion, cognitive outsourcing, and pipeline opacity combine to systematically misattribute AI outputs as user competence. The effect is multiplicative—each mechanism amplifies the others.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
Research shows AI systems can instrument multimodal behavioral signals (gaze, hesitation, speed) to read cognitive state during interaction, preserving flow by avoiding disruptive explicit probes. However, the same substrate enables both helpful timing and manipulative profiling.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Guidelines for Human-AI Interaction
- Tell me about yourself: LLMs are aware of their learned behaviors
- The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows
- Mechanisms of Introspective Awareness
- Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs
- Large Language Models Cannot Self-Correct Reasoning Yet
- Word Meanings in Transformer Language Models
- LLM Evaluators Recognize and Favor Their Own Generations