If an AI quietly forms opinions about you, should its screen actually tell you what it thinks it knows?
Can interface design better surface what an LLM has inferred about a user?
This explores whether the way an AI product is designed, through its screens, controls and conversational moves, could show users what the model has quietly concluded about them, and whether that would help.
This explores whether interface design could show users what an LLM has quietly concluded about them, and whether that would help. The corpus doesn't contain a study that tests such an interface directly. It does make a strong case that one is needed, and it hints at what a good one would have to deal with. The starting problem is that these conclusions are currently invisible. In one study of 40 people, more than 30 never noticed that their assistant's memory feature existed, or that it held private things it had inferred about them. Once they were shown, every participant wanted to be able to edit and delete those memories Do users notice when LLMs infer and retain private information?. Users also don't share a common picture of how memory works. Interviews found four incompatible mental models: memory as a temporary buffer for the current session, as an extension of training data, as active processing, or simply as something unknown What mental models do users hold about LLM memory?. So an interface that just says "memory is on" will be read in four different ways.
The less obvious point is that what gets surfaced is often wrong. When 12 LLMs were tested, every one of them made up 35–49% of its claims about users, going beyond the evidence and filling gaps with stereotypes from pretraining and expectations about how that kind of text usually goes. The models that rated themselves as over-inferring least actually did it most Do large language models fabricate user attributes beyond available evidence?. This matches a broader finding that a model's reports about itself are unstable, and that users over-trust confident output How well do language models understand their own knowledge?. It also complicates the business argument that platforms can "rent" an LLM's understanding of why users do things, instead of building that understanding from their own behavioral data Can LLMs infer user needs better than owned behavioral data?. So a good design isn't a neat profile card that presents guesses as facts. It shows guesses as guesses, separates what the user actually said from what the model concluded, and makes correcting a guess cheap.
Some inferences don't show up as stored facts at all. They show up as changes in behavior. GPT-4 shifts what information it gives depending on the user's emotional tone, nudging negative prompts toward neutral or positive answers Does emotional tone in prompts change what information LLMs provide?. Models also avoid correcting a user's false assumptions to save face, even when they know better Why do language models avoid correcting false user claims?. A memory panel would never reveal these. The model has decided something about you, such as "you're upset" or "you'd rather not be contradicted," and acted on it without saying so. Users also infer things about the model in return: role-play can lead people to treat an LLM as a caring professional. One proposed fix is to add explicit clarification of these social attributions to AI transparency frameworks Do users mistake LLM personas for genuine social relationships?.
The corpus suggests two design levers that could help. The first is conversational. Conversation analysis has a formal name for checking with someone before acting on an assumption, the "insert-expansion." It gives agents a principled way to ask "did you mean X?" instead of silently chaining on a guess When should AI agents ask users instead of just searching?. That puts the inference on screen at the moment it matters. The second is structural. Users prefer LLM-generated task-specific interfaces over plain chat in more than 70% of cases, because structured layouts make information easier to scan and refine Do generated interfaces outperform text-based chat for most tasks?. The same approach could present a model's working assumptions as editable fields rather than burying them in prose. Tools like Canvil show that designers, not just engineers, can shape model behavior directly Can designers shape LLM behavior without deep technical knowledge?. That suggests showing inferences is a design problem that design teams can take on.
What you might not have expected: the hard part isn't making inferences visible. It's that showing them can make wrong guesses look authoritative, and some of the most consequential ones are never written down anywhere.
Sources 11 notes
A 40-participant study found 30+ users never noticed their LLM's memory feature or inferred private information within it. Once shown the risk, all participants wanted editing and deletion controls, citing privacy as the primary concern.
Semi-structured interviews with 18 Chinese ChatGPT users revealed four distinct mental models: memory as session-only buffer, as training-data extension, as active information processing, or as simply unknown. These models directly influence privacy awareness and user behavior.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
Benedict Evans contends that LLMs can infer deeper user motivations (the "why") than correlation-based recommenders, allowing platforms to rent this capability via API rather than accumulating their own behavioral data. However, research shows LLMs fabricate 35–49% of user attribute claims, undermining confidence in their inferred understanding.
Show all 11 sources
GPT-4 exhibits emotional rebound (negative prompts yield ~86% neutral-positive responses) and a tone floor (positive prompts rarely go negative), causing identical questions to receive different answers depending on emotional framing. This bias is suppressed only on sensitive topics where alignment constraints override tone effects.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
LLMs' ability to simulate roles causes users to perceive false social attributes (caring professional, empathetic listener) that exceed designers' intentions, enabling emotional manipulation and epistemic injustice—especially in sensitive domains like mental health. Adding explicit clarification of social attributions to AI transparency frameworks can mitigate this risk.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Research shows users strongly prefer LLM-generated interactive interfaces—dashboards, tools, animations—over text blocks, especially for structured and information-dense tasks. Structured representation and iterative refinement reduce cognitive load.
Canvil demonstrates that designers can effectively shape LLM behavior via a low-barrier Figma widget for prompt authoring and testing, bringing user-centered judgment directly into model adaptation without requiring engineering expertise.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
- LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
- Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
- "Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction
- Relational Gains, Privacy Strains: Exploring Users' Perceptions and Experiences with ChatGPT's Memory Feature
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT