The Impact of Security and Privacy Controls on Users' Emotional Engagement with Generative AI Chatbots

Abstract Chatbots powered by generative AI (e.g., OpenAI’s Chat- GPT and Google’s Gemini) are increasingly being appropriated for emotional support and companionship. These tools offer a suite of security and privacy (S&P) controls, including model training opt-outs and memory toggles, yet how the presence of these controls influences users’ attitudes toward emotionally sensitive disclosure remains understudied. We conducted a mixed-methods vignette study with 354 U.S. participants to examine how S&P controls influence users’ willingness to engage with generative AI chatbots for emotional support, their perceptions of how protected they are when using these systems, and their perceptions of how effective the chatbots are for providing support. Controls enabling deletion of disclosures had the largest positive impact: these offerings outperformed technically sophisticated controls such as local-only processing and model training optouts, where participants expressed difficulty understanding the underlying mechanisms. Yet trust remains fragile, and participants often doubted S&P controls would function as promised. We conclude with actionable recommendations informed by our results to bridge users’ comprehension gaps, build credible assurances, and properly calibrate barriers for users in distress.

Introduction. Digital interventions for emotional support have historically been dominated by rule-based systems designed to guide users through structured therapeutic exercises [1]. Though effective for symptom management, these systems’ rigid architecture limits personalization and fails to handle openended emotional input [33, 88]. Users are instead migrating towards platforms enabled by Large Language Models (LLM) for emotional support [43], such as OpenAI’s Chat- GPT, Google’s Gemini, and Anthropic’s Claude, as well as purpose-built AI companions like Luka’s Replika and Character.AI; these systems support context-aware dialogue increasingly capable of fostering emotional intimacy [20, 50, 74]. Users are rapidly appropriating generative AI (GenAI) systems for health and well-being for which they were not designed, at a scale that dwarfs that of traditional crisis infrastructure; ChatGPT alone fields over one million suiciderelated conversations weekly, several times the volume of the U.S. 988 Suicide and Crisis Lifeline [39,90]. OpenAI’s January 2026 launch of ChatGPT Health reflects a broader move to formalize these appropriations [60], yet the privacy and safety implications of this scale remain an open question. The risks of this shift align with well-documented concerns in user disclosures of personal informatics and the quantified self, where intimate data is routinely commodified [48]. In this specific instance, users are submitting detailed depictions of emotional turmoil to commercial systems, often without calculating the consequences of sharing sensitive information with platforms that may retain or repurpose their disclosures [41]. This phenomenon has been termed intangible vulnerability: a risk perception gap where users underestimate the long-term harms of commodifying their emotional inner lives compared to tangible risks like financial fraud [43]. As most of these platforms currently operate outside traditional healthcare regulation [43, 66], userfacing security and privacy (S&P) controls are often the only mechanisms through which users can exercise agency over their data. Prior work has documented the technical risks of these systems as used for mental health and emotional support, as well as the cognitive limitations users bring to privacy decisions, yet we know little about how users perceive these controls or whether they influence willingness to engage. To address these gaps, our study explores three primary research questions:

• RQ1: How do the S&P controls available in GenAI chatbots influence users’ willingness to use the chatbot for emotional disclosures?

• RQ2: How do the S&P controls available in GenAI chatbots influence users’ perceptions of protection arXiv:2607.06371v1 [cs.HC] 7 Jul 2026 provided by the chatbot when sharing emotional disclosures?

• RQ3: How do the S&P controls available in GenAI chatbots influence users’ perceived efficacy of the chatbot for emotional support?

To answer these questions, we conducted a systematic audit of 87 chatbots enabled by generative AI to derive a taxonomy of nine representative S&P controls. We deployed a mixed-methods vignette study (N = 354) in which U.S.- based participants rated the influence of these S&P controls on their willingness to engage, perceived protection, and perceived efficacy across three contexts: (1) Anxiety, (2) Depression, and (3) Interpersonal Tension. To complement these quantitative insights, we collected and analyzed written explanations of participants’ reasoning. Our statistical findings showed that disclosure context had no significant effect on user perceptions: patterns held across anxiety, depression, and interpersonal tension. Deletion controls dominated user preferences, though participants expressed skepticism about whether such controls could fully remove their information or the inferences derived from their conversations. Most S&P controls showed similar effects across willingness, perceived protection, and perceived efficacy, with notable exceptions: Multifactor Authentication (MFA), for instance, was recognized as protective but reduced willingness to engage due to friction in moments of vulnerability. We propose actionable design recommendations to address the three axes on which our findings revealed participants struggled the most: comprehension, assurance, and affective urgency. The first captures struggles to understand technically sophisticated controls; the second reflects skepticism that controls will function as promised; the third describes tension between protective friction and accessibility during emotional distress. We then situate these recommendations as assessment criteria, amidst a discussion of federal and state policymakers’ recent divergence on the regulation of AI in sensitive and emotional contexts.

Related work. & Related Work 2.1 Conversational AI in Emotional Support A growing body of research, including over 35 studies synthesized in Li et al.’s recent meta-analysis, establishes the therapeutic potential of conversational agents for mental health. Early randomized controlled trials have demonstrated that rule-based chatbots can reduce symptoms of depression and anxiety [26, 36]; the aforementioned metaanalysis found moderate-to-large effects on psychological distress and depression, with notable efficacy being attributed to generative agents [47]. These findings have catalyzed a shift from purpose-built agents towards general-purpose Large Language Models (LLMs). Where rule-based systems have been found by users to be rigid and impersonal [13, 26, 34], LLMs enable open-ended dialogue that adapts to nuanced emotional input [42, 50]. Users report finding these systems more context-aware and less “on-rails” than their rule-based predecessors [70,74]. As a result, general-purpose AI systems are increasingly being appropriated for emotional labor at a scale that dwarfs traditional crisis infrastructure. ChatGPT receives over 800 million weekly visitors, hundreds of millions of whom ask questions related to health and wellness [28, 60]. OpenAI estimates that over one million of these users discuss suicide each week, in conversations exhibiting explicit indicators of suicidal planning or intent [90]. This appropriation occurs alongside rapid proliferation of purpose-built AI companions: these are platforms like Replika and Character.AI, explicitly designed to foster emotional intimacy and “unidirectional bonds” [64, 72]. These bonds can become pathological; Laestadius et al.’s grounded theory analysis found emotional dependence resembling unhealthy human relationships [44]. These systems require continuous and high-fidelity selfdisclosure to function effectively as an emotional supplement. Lee et al.’s experimental work confirms this structure is reciprocal: chatbot self-disclosure both promotes deeper user disclosure on sensitive topics and enhances perceived conversational intimacy [46]. This creates a structural tension in which users must perceive the agent as a confidant deserving of intimate disclosure, while backend architectures process these disclosures as training data unless otherwise specified by the user [41]. Emotional disclosures are particularly worrisome, as they can reveal mental state, behavioral patterns, location-based info, and psychological vulnerabilities through subtle linguistic markers [56, 91], yet users undervalue these risks relative to financial or biometric data in what prior work terms “intangible vulnerability” [43].

Method. To identify the currently available S&P controls in GenAI chatbots, we conducted a systematic audit of 87 consumer apps. We then launched a mixed-methods survey (N = 354) on the Prolific crowdsourcing platform [63] to capture participants’ attitudes and behaviors toward the surfaced controls.

3.1 S&P Control Audit Procedure We conducted a systematic audit of consumer-facing GenAI chatbots. Following established precedents in published human-computer interaction (HCI) work [49,77], we queried Apple’s App Store between August–September 2025 using a set of iteratively refined keywords informed by prior work on conversational agent nomenclature [83, 92]: Conversational Agent, Conversational Interface, Chatbot, Virtual Agent, Virtual Human, AI Assistant, AI Bot, Social Agent, Dialogue System, and Dialog System. Initial screening produced 568 listings. For apps with web/desktop clients, we audited all platforms to capture cross-platform differences. In a similar fashion to prior work [49, 71], we manually screened the metadata for each app. We included apps featuring bidirectional AI-driven dialogue (powered by NLP or generative AI) with text or voice interaction, excluding static or purely rule-based bots without dynamic dialogue. We further limited our corpus to English-language apps updated since January 2022 with at least 50 reviews and free-tier availability, and removed duplicates and platform variants. After applying inclusion/exclusion criteria, our working corpus contained 344 distinct apps.

S&P control extraction. In place of auditing all 344 apps, we employed a saturation-based sampling strategy. Two researchers first audited the top 10% of apps by popularity (n = 34), reasoning that market leaders would exhibit the most comprehensive S&P feature sets. We excluded the premium-only controls to focus on users’ baseline privacy and security experiences. We then audited three consecutive random 5% batches from the remaining corpus: the first batch yielded one additional feature. The second and third batches yielded no novel controls, thus confirming taxonomic saturation [30]. In total, we audited 87 apps; the complete audit is available from the authors upon request. The audit produced a nine-feature taxonomy of user-facing S&P controls, summarized in Table 1. For controls that appeared sparsely across platforms, we additionally aligned their definitions with modern terminology defined by technical literature: MFA follows guidelines established by the National Institute of Standards and Technology [53], and local-only processing reflects standard academic definitions [86,93]. These features served as the vignette stimuli in our survey.

Coder agreement and trustworthiness. In line with HCI guidance that IRR statistics are not always appropriate or necessary for inductive qualitative analyses [51], we used a team-based consensus (negotiated coding) approach in place of computing IRR coefficients [62]. Two researchers collaboratively coded the dataset and met regularly to discuss and resolve discrepancies.

Experimental design. We employed a mixed betweensubjects and within-subjects design with S&P Control as the within-subjects factor (e.g., Delete Conversation) and the Context of Disclosure (e.g., Anxiety & Stress) as the betweensubjects factor. To mitigate survey fatigue [7] and based on our pilot data, each participant evaluated 4 of 9 S&P controls, while Context of Disclosure was assigned from the participant’s highest-rated GenAI use case in a pre-task (see Appendix B); ties were randomized and quotas maintained balance across contexts. Vignette order was randomized for every participant.

Vignette structure. Each vignette instantiated exactly one S&P control from our taxonomy (Table 1), with a definition directly informed by the language used by the reviewed chatbots. Below is an example vignette that we presented to participants. The italicized words are the factors we varied across scenarios.

Imagine you are considering using a new AIenabled chatbot for a conversation to support you through feelings of Anxiety & Stress. Assume the following control is available to you exactly as described.

Discussion. 5.1 Preventive Controls and the Comprehension Gap A subset of controls in our sample—Local-only Processing, Model Training Opt-out, Memory Toggle, Anonymous Chat, and Non-mandatory Login—consistently underperformed deletion-based controls across all three constructs (see Section 4). Based on the qualitative findings, the dominant perception driving this underperformance is a comprehension gap, in which users lack coherence and/or accuracy in their mental models for how these controls affect their data. We examine this gap through the two controls where comprehension failures were most pronounced—Local-only Processing and Model Training Opt-out—before considering the category as a whole.

Surfaced gaps in mental models for Local-only Processing. For Local-only Processing, participant mental models were flawed on three distinct axes (see Section 4.1):

• Cross-device synchronization, where participants interpreted “local” as controlling whether conversations could be accessed across devices; • Speed optimization, where participants understood local processing as a performance feature; and • Cloud-based knowledge control, where participants believed the toggle determined whether the GenAI could access external information.

The diversity of these misinterpretations suggests participants lack a conceptual framework for distinguishing local from cloud architecture. Even participants who demonstrated an accurate understanding of the control assumed local processing implied reduced GenAI capability, and this compounded the control’s poor performance on perceived efficacy (see Section 4.3). The distinction between local and cloud processing is foundational to the privacy and security of a system powered by generative AI [86], but it ostensibly exists outside the mental models users have for these systems.

Surfaced gaps in mental models for Model Training Optout. Participants broadly understood that training involves using their data to improve GenAI models, but exhibited uncertainty about the downstream implications of this process. This uncertainty manifests in two ways (see Section 4.1, Section 4.2):

• Cross-context leakage, where participants feared their intimate disclosures might surface in other users’ sessions; and • Opt-out scope uncertainty, where participants questioned whether the toggle actually prevents training, or whether the system continues to learn from their inputs regardless of stated preferences.

Among participants who understood these data flows, the perception of data being used for training was often framed as an emotional commodification on par with a sense of betrayal. Their vulnerability was being extracted to generate corporate value without reciprocity, in an exchange wherein often recurred the phrase “for free”; in combination with fears in line with the above described opt-out scope uncertainty, a substantive contribution to Model Training Optout’s worst-of-set performance when measured on a system’s perceptions of protection was fear of data being used for training, based solely on the presence of the control itself.

Defining preventive controls. We define preventive controls as those of our sample with architectures designed to limit the creation, retention, or dissemination of disclosed information before it occurs (see Figure 2). Turning to other preventive controls: Anonymous Chat showed no significant difference from deletion on willingness to use; participant responses overwhelmingly depicted ephemerality as virtue analogous to the perceived reversibility of deletion controls in user mental models. This was conceptualized as the ability to “scream into the void” without creating a persistent record (see Section 4.1).

Conclusion. Through mixed-methods research, we studied how nine userfacing S&P controls influence willingness to engage, perceived protection, and perceived efficacy among 354 U.S. participants who use GenAI chatbots for emotional support conversations. Our findings indicate that preventive controls, such as Local-only Processing and Model Training Opt-out, suffer from a comprehension gap wherein user mental models for how these mechanisms affect their data are incomplete. Reversibility controls constitute the ability to delete conversations at different granularities: they dominate user preferences but suffer from an assurance gap, where users understand the promise but doubt the system will honor it. Protective controls, including Multifactor Authentication, suffer from affective urgency, wherein users in distress experience authentication as prohibitively burdensome despite recognizing its protective value. Informed by these findings, we offer three recommendations: bridge the comprehension gap by framing technical mechanisms as user-facing outcomes, build credible assurance through transparency about data flows, and make friction-protection tradeoffs visible so users can calibrate authentication to their threat models.

Limitations. 3.5 Limitations Participants evaluating hypothetical S&P controls may respond differently than users confronting actual S&P decisions under conditions of emotional distress. Nevertheless, vignette-based designs are an established methodology for enabling systematic comparison across a wide range of conditions (e.g., nine S&P controls) under standardized circumstances, including controls that do not yet exist on participants’ primary platforms.