When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information

Ask ChatGPT about vacation planning, and it may infer your income. Ask it about medication, and it may infer your medical history. Because such inferences can expose more information than users intend to reveal, prior work argues that they are a defining privacy risk of LLM-based systems. Yet prior work has mostly shown that LLMs can make potentially violating inferences, not how users experience those inferences nor what controls users may want governing their use. We built the Reflective Layer, a visualization tool that surfaces example unstated inferences from users’ own ChatGPT histories, and used it in a mixed-methods study with 18 regular ChatGPT users evaluating 215 surfaced inferences from their own conversations. Counterintuitively, participants reacted more strongly with curiosity and interest rather than distress and concern. Discomfort arose mainly when inferences felt misrepresentative of the user or misaligned with expected use. Participants were also markedly less comfortable with advertisers and third-party applications using those inferences than with platform providers. These findings suggest that the acceptability of LLM inferences is governed not only by its content, but by context-sensitive norms around how they are generated, retained within the platform, and transmitted beyond it.

Introduction. People share deeply personal information with large language model (LLM)-based systems [37, 74]. Moreover, from these disclosures, LLMs can infer much more about users than what they explicitly state [51, 54, 59]. For example, if Alice asks an LLM to help draft an email to her manager requesting sick leave, that same LLM, appropriately prompted, can infer her workplace role, relationship with her manager, and aspects of her health. This ability to make personal inferences—about users’ intents, preferences, circumstances, and needs beyond what they have shared—is a defining privacy risk of LLM-based systems that goes beyond verbatim memorization of conversational data [30, 39, 54]. The dangers of these inference-based privacy risks are even more salient as prominent model providers, like OpenAI, have introduced personalized advertisements based on conversation context into these conversational assistants [45], are exposing tool actions by third-party vendors through the Model Context Protocol [1, 43], and as users delegate tasks to LLM-powered agentic harnesses that can autonomously act on, use, or share user information [46]. Simultaneously, this same inference capability is lauded and being further developed for its utility. The AI and HCI communities are moving towards an aspirational vision of “proactive” agents [28, 33, 34, 52, 53] that predict the user’s next action based on what can be inferred about the user’s context and goals from their conversations and activities. One instantiation of this vision is the general user model (GUM) [51]—a computational model built from conversational and activity data that makes deep inferences about user intent, context, and need in order to personalize AI behavior. These proactive agents promise to reduce user prompting burden through inferred grounding: they use inferences to “understand” a user’s context without the user needing to explicitly share it. These complementary perspectives—on the utility versus intrusiveness of LLM inferences—are now coming to a head. As agentic AI (e.g., Claude Desktop, OpenClaw) rises in popularity, the risks and benefits of personal inferences will increasingly shape users’ day-to-day experience with LLM-powered systems. But when do users find these inferences acceptable, and when do they find them intrusive? To date, work on revealing and managing LLM privacy risks for users has focused on memory—what conversational LLMs save and retrieve from prior conversations [9, 36, 71, 72]—rather than on inference: i.e., what LLMs can learn about users that they did not explicitly state or share. Without the user perspective on inferences, it is difficult to assess what controls are necessary, or how to design for inferences that benefit users in ways they find acceptable, not intrusive. To bridge this gap, we pose two research questions. First, how do users feel about and evaluate the personal inferences that an LLM can make about them from their conversation history (RQ1)? Second, what access-control preferences do users hold for such inferences, and what reasoning underlies those preferences (RQ2)? A critical challenge with answering these questions is that LLM inferences remain opaque to users. Because users rarely see what is being or could be inferred about them, it is difficult for them to assess or articulate their reactions. We built the Reflective Layer: a research probe that surfaces inferences derived from users’ own chat histories. The Reflective Layer adapts the GUM pipeline introduced in prior work [51] to conversational data exported from ChatGPT. GUM already distinguishes observations (factual records) from propositions (confidence-weighted inferences about users), and surfaces the latter for downstream AI assistants to consume silently. We adapt this architecture to build a user-facing visualization tool: adding an explicit observation-extraction step so that each inference traces back through its supporting observations to the specific chat messages that produced it, organizing inferences into categories drawn from user modeling theory [6, 26], and scoring sensitivity via continuous Contextual Integrity (CI) judgments [29, 42]. With the Reflective Layer as our research instrument, we conducted an eighteen-participant, mixed-methods study in which participants encountered inferences drawn from their own ChatGPT histories. Participants then evaluated these inferences on affective response, relevance, usefulness, and intrusiveness (RQ1). To probe access-control preferences (RQ2), participants additionally rated their preferences across three recipient-and-use contexts (conversational improvement, advertising, and third-party action-taking) and, through a sketching activity, externalized the boundaries they would draw around how these inferences should be accessed.

Related work. We are informed by two bodies of prior work. The first establishes that LLMs can derive personal attributes from conversational or activity traces, making inference both central to personalization and an emerging privacy risk. The second, from usable privacy and security, explores how to elicit user reactions to and control preferences for opaque algorithmic systems.

LLM Inference for Proactive Personalization. Deep personalization of LLM-powered agents is a guiding goal for much of the AI and HCI communities. Recent work organizes personalized agents around profile modeling, memory, planning, and action execution, showing how user-specific signals can condition how agents interpret requests, retrieve context, form plans, and act [14, 68]. Empirical work on user profiles similarly shows that historical user records improve personalized LLM outputs across tasks and begins to unpack what profile content contributes to those gains [66]. Recent systems, informed by the user-modeling literature, have further shown that LLMs can convert observations of user activity into inferences that can be used to predict downstream user actions and goals [28, 51, 52]. LLM Inference as a Privacy Risk. The source material for such personal inferences is abundant in everyday LLM interactions: conversations routinely contain personal and sensitive content that, when combined with other disclosures, may reveal things about users they do not expect [37]. The S&P community, in particular, has been vocal about the risks entailed by the inferential capabilities of LLMs, showing that models can infer sensitive attributes from users’ text and image inputs [54, 59]. Work on free-form chatbot interactions extends this concern to psychometric traits, such as the Big Five personality inventory [47]. This personal attribute leakage can further accumulate across repeated interactions [40]. Existing data-centric protections, including memorization controls, PII redaction, and differential privacy, cannot fully address these risks [38, 39], which include contextual-integrity failures and aggregation harms that can arise even without verbatim leakage. Together, these threads of prior work establish inference as technically plausible, useful, and privacy-concerning, but they do not show how users evaluate such inferences. When would users find the use of these inferences for personalization acceptable? When would they find it intrusive? Our work bridges this gap by asking how users’ concerns unfold and shape acceptability when personal information is inferred from their own conversational histories.

Method. The literature uses the terms memory and inference to describe related but distinct phenomena. Here, we clarify how we use these terms. Memory is an object: the fragments of prior user input that an LLM-based system extracts and surfaces as context to personalize a subsequent query.1 In recent implementations, memory has been operationalized as verbatim retrieval of past inputs [57] or as LLMparaphrased extractions that summarize past conversations into stored items (as in ChatGPT’s Saved Memory feature [9, 71, 72]). What matters is the extracted fact itself; its phrasing (verbatim or paraphrased), its extraction technique, and the processes of retrieval and update are distinct concerns. Inference, by contrast, is the act of drawing conclusions about a user that are not directly stated in their input [28, 51].2 Prior privacy work sometimes uses inference more broadly for predicting personal attributes from text, including explicit extraction as well as contextual deduction [54]; here, we use the narrower second-hop sense. Information-theoretically, the mutual information between the input and the inferred attribute is low, but external knowledge and population-level distributions make the attribute recoverable. Consider Alice from the introduction: when she asks ChatGPT to help draft a short email requesting sick leave from her manager, she has not told the system her workplace role, the nature of her relationship with that manager, or anything about her health—yet an LLM, drawing on world knowledge about workplaces and communication norms, can plausibly infer all three from her request. We frame the space as a two-hop abstraction over user inputs. The first hop extracts explicit, user-related facts from messages; these correspond to what deployed systems call memories. The second hop derives higher-level attributes from those facts—inferences— which form the building blocks of a user model. GUM [51] formalizes a related architectural distinction between observations (factual records) and propositions (inferences); user-modeling literature [6, 26] and recent privacy work on LLM memory [40] draw the same distinction. This distinction matters for user-facing study of LLM inference: what users accept may differ between what the system has seen about them and what the system has concluded about them—a distinction the interface needs to preserve, with each hop traceable to the content on which it is based. The Reflective Layer (§4) operates at both hops, showing participants not only what was inferred about them but also the specific messages that contributed to an inference.

The Reflective Layer is a research probe that surfaces LLM-derived inferences about users back to them. It grounds those inferences in prior work on General User Models (GUMs), which describes how systems can transform observed user activity into confidenceweighted propositions about the user [51]. The goal is to generate plausible GUM-style inferences from participants’ own histories so that users can react to otherwise invisible inference possibilities. We use layer to emphasize that the probe wraps an underlying user-modeling pipeline rather than replacing it: it adapts a known inference pipeline and exposes its outputs, provenance, and sensitivity estimates for end-user reflection and inspection. We describe the inference extraction pipeline below; additional implementation details are provided in Appendix A.

At a high level, the Reflective Layer takes as input a user’s exported conversation history from an LLM platform and outputs a visualization of personal inferences derived from those conversations. The pipeline first extracts factual observations from user-authored messages, then uses those observations and related prior observations to generate confidence-weighted, sensitivity-scored inferences organized by broad kinds of personal information. The Reflective Layer adapts the architecture of General User Models (GUMs) [51].

Discussion. LLM inferences were acceptable to participants when they felt accurate, useful, expected, and kept within a context of use the participant recognized as appropriate. They were not acceptable when they felt surprising, intrusive, inaccurate, not worth the system knowing, or retained beyond the context in which they were useful. Access-control preferences were driven by recipient, purpose, and expectation-purpose alignment. Participants were generally more comfortable with platform use for conversational improvement than with advertiser or action-taking third-party access. Preferences also varied by inference category (e.g., whether inferences were about relationships or interests) and whether the inference felt personal or professional. We use these findings to outline a governance layer for LLM-derived inferences (empirical-to-design mapping in Appendix D).

Participants reasoned about inference control differently across decision points: generation, or whether the inference should have been drawn at all; retention, or what they could do about it once drawn; and transmission, or if, how, and when it is acceptable for a third party to access it. The same inference could be evaluated favorably at one point and unfavorably at another, with constraints shifting across decisions.

Generation. Participants distinguished between content they shared with LLM-powered conversational assistants for self description versus content they shared for task-specific purposes, and generally resisted inferences drawn from the latter. Participants described most of the content they shared as taskcontextual and not appropriate raw material from which to draw stable facts about their identities, goals, and intents; inferences drawn from task content were frequently both inaccurate and rejected. This generation-stage concern reflects a broader distinction between voluntary disclosure and passive extraction. Participants recognized that they had typed the source material into ChatGPT, but they did not treat every prompt as permission to convert taskspecific context into persistent claims about them. The issue was therefore not simply whether the system had access to the data, but whether the inference respected the purpose for which that data had been shared. Prompts and inferences therefore need to be treated as different objects: prompts are situated requests, while inferences are intentionally situationally invariant. These findings suggest that users would benefit from a governance mechanism that surfaces each newly generated inference, paired with the user activity data that produced it, along with controls to accept or reject the inference before it is used as context for subsequent interactions. Model providers would also benefit from this feedback, because it would help them learn which inferences users find acceptable. Retention. Once an inference exists, the governance question shifts to what the user can do to change it, delete it, or control how it is used. Participants’ concerns at this stage fell into five categories: derivation (how the system arrives at a specific inference), capability (what the system can infer), persistence (how long inferences are retained and how they carry across conversations), aggregation (how inferences combine into a composite profile), and usage (what downstream actors will do with an inference once it leaves the platform). Moreover, each category suggests a distinct affordance for appropriate access-control interfaces: surfacing the chat evidence that produced each inference; exposing the pipeline’s inferential scope; making retention policies visible; exposing how inferences compose; and binding each inference’s use to a specified purpose and recipient, respectively.

Conclusion. As LLM systems increasingly personalize, remember, and act on inferences made about users’ identities, goals, and intents, a central question is when users find those inferences to be acceptable. This question becomes more urgent as LLMs move toward proactive and action-taking agents that may rely on inferred user attributes when deciding what to suggest, automate, or share. Our study makes progress on that question by gauging user reactions to and access-control preferences for inferences drawn from users’ own ChatGPT histories. Participants often found inferences interesting or useful, and they drew boundaries around inferences that felt creepy, misrepresented them, overstepped the purpose for which source content had been shared, or flowed to recipients outside the platform relationship they understood. These findings suggest that acceptability depends on the content and accuracy of an inference, but more importantly on the evidence behind it and its fit with the context in which it was generated, retained, and transmitted. Inferences result in different privacy concerns that call for different governance mechanisms: misrepresentative inferences need correction or retraction, accurate but exposing inferences need restriction, and secondary uses need perrecipient and per-purpose approval. Users need ways to inspect what evidence produced an inference, decide whether it should persist, and control whether it travels beyond the context in which it was generated. Inference governance tools should distinguish between generation, retention, and transmission, and should let users act on the specific inferences a system has drawn about them throughout this lifecycle.