Disclose with Care: Designing Privacy Controls in Interview Chatbots

Collecting data on sensitive topics remains challenging in HCI, as participants often withhold information due to privacy concerns and social desirability bias. While chatbots’ perceived anonymity may reduce these barriers, research paradoxically suggests people tend to over-share personal or sensitive information with chatbots. In this work, we explore privacy controls in chatbot interviews to address this problem. The privacy control allows participants to revise their transcripts at the end of the interview, featuring two design variants: free editing and AI-aided editing. In a betweensubjects study (N= 188), we compared no-editing, free-editing, and AI-aided editing conditions in a chatbot-based interview on a sensitive topic. Our results confirm the prevalent issue of oversharing in chatbot-based interviews and show that AI-aided editing serves as an effective privacy-control mechanism, reducing PII disclosure while maintaining data quality and user engagement, thereby offering a promising approach to balancing ethical practice and data quality in such interviews.

Introduction. In human-computer interaction (HCI) and social science research, data quality on sensitive topics is often compromised by participants’ tendency to withhold or misrepresent information [50, 51]. This phenomenon, often driven by concerns about privacy [20], social desirability bias [8, 30], or potential repercussions [64], directly compromises the validity and reliability of self-report data. When individuals obscure their true opinions, behaviors, or experiences on sensitive topics, such as health conditions, financial practices, or personal beliefs, researchers are left with an incomplete or inaccurate understanding of the phenomena under investigation [7, 16]. Consequently, such studies yield biased insights that produce flawed theoretical models, ineffective design recommendations, and misguided interventions, failing to address societal challenges and cause harm. To address these data quality challenges, chatbots demonstrate considerable potential as scalable interviewers, offering interactive and conversational experiences conducive to collecting data from participants compared to paper-based surveys and face-to-face interviews, particularly concerning sensitive topics [34–36, 54, 72]. This is further supported by findings indicating that individuals often disclose more sensitive information to computer-assisted interfaces due to perceived non-judgment, reduced social inhibition, and confidentiality assurance [10, 53]. However, this affordance also carries the risk of oversharing [74], as interviewees may disclose more sensitive or identifiable information than is necessary for the questions, or more than they would naturally share without using a chatbot. This problem is further exacerbated as LLM-powered chatbots exhibit more human-like behavior, which can induce even greater levels of disclosure [75, 76]. A trending industrial example is Anthropic Interviewer, an AI-powered tool that can scale up interviews with high quality [22]. However, the rich details elicited by the chatbot also result in high vulnerability of privacy leakage, due to the lowered barrier and ease of re-identifying interviewees with web-search LLM agents [33]. Such unnecessary disclosures may place users at risk of privacy violations or identity theft, particularly if a data breach occurs or the data is released without sufficient privacy-preserving measures. Common practices to address oversharing and the resulting privacy risks include obtaining informed consent before data collection, ensuring and communicating secure storage procedures to reduce breach risks and build trust, and enforcing strict access controls and data minimization once the data has been uploaded [74]. These methods offer essential safeguards by fostering transparency and accountability in the research process. However, they remain insufficient to fully protect users from unwanted or impulsive disclosures, given the inherent power asymmetry between researchers and study participants. The efficacy of informed consent has long been questioned [23], complicated by participants’ differing levels of literacy and understanding of research information [26]. Furthermore, the question of how much detail is sufficient in an informed consent remains under debate [63]. Accordingly, endowing users with more controls, such as controlling what information to share [4] and how the shared information is used [74], has been proposed as a way to repair this power asymmetry. Nevertheless, this also raises researchers’ concerns about the authenticity of participants’ responses [74]. Currently, there is limited research on this topic, which remains an essential gap in achieving a balance between ethical research practices and data validity. In our research, we aim to bridge this gap by presenting the first study to examine privacy controls in chatbot-based interviews, guided by two research questions: (1) How would participants use privacy controls in chatbot-based interviews, and (2) How would privacy controls impact participant privacy, data quality, and user engagement in chatbot-based interviews? To this end, we developed an interview chatbot system, featuring two modes of post-interview privacy controls: free editing and AI-aided editing. In both modes, participants are provided with an opportunity to freely revise their responses after the interview and are informed of this opportunity beforehand. The AI-aided editing mode is motivated by the recurring theme in usable privacy literature of limited privacy awareness and need for scaffolds to support rational privacy decisions [26, 76, 78]. Our AI-aided mode scans the entire interview log for personally identifiable information (PII), and generates suggestions to either redact the PII to a type placeholder (e.g., [NAME1]) or abstract them to a more general level (e.g., replacing “18 years old” with “young adult”), which participants can optionally accept and apply.

Related work. 2 Related Works 2.1 Challenges in Data Collection for Research on Sensitive Topics Prior work has demonstrated that obtaining accurate self-reports on sensitive topics, including health, finance, and ethical dilemmas, is challenging. Participants often underreport or distort their responses due to social desirability bias [7, 8, 52, 67], and privacy concerns further deter disclosure [2]. Similar patterns have been observed in online communities and digital platforms, where disclosure is influenced by contextual norms and perceived surveillance [39, 40, 65]. Additionally, research on social media has highlighted the fragility of privacy decision-making, often stemming from regrets over disclosure [71] and underestimation of audience size [38]. Recent studies have further explored this through the lens of cognitive styles and automated systems, showing that risk-benefit evaluations drive information sharing, while user-friendly interfaces can support better privacy behaviors [5]. Despite voicing strong privacy concerns, users frequently disclose personal information when they trust the recipient, revealing a disconnect between attitudes and behaviors [5]. This phenomenon, known as the privacy paradox, describes how individuals’ actions do not align with their stated concerns, often justified by default settings or perceived invulnerability [46]. Research has unraveled this paradox by examining factors like cognitive and psychological influences, technological aspects, and sociocultural elements, emphasizing the role of perceived benefits outweighing risks in digital marketing and e-commerce contexts [42]. These issues underscore the entanglement of self-report data collection with impression management and privacy concerns, prompting the development of alternatives that reduce judgment and build trust. Accordingly, HCI research increasingly focuses on designing data collection methods that address disclosure hesitation and balance impression management with privacy protection, especially for stigmatized topics [3, 31]. Our study advances this research by emphasizing post-hoc privacy mechanisms that act after disclosure to address the over-sharing of privacy information. Notably, we observe a discrepancy between the subjective perception in privacy disclosure and the objective detected PII which echos the classic privacy paradox issues, and demonstrate the efficacy of AI-aided editing in addressing the gaps in privacy awareness and promoting more responsible sensitive data sharing behaviors.

Method. 3 EDIT: Enhanced Disclosure Intervention Tool We developed a web-based application, hereafter referred to as the Enhanced Disclosure Intervention Tool (EDIT), to conduct semistructured, multi-turn interviews with online participants, and then implemented the two proposed privacy control features based on it. Figure 2 depicts the proposed extension, illustrating example actions and outcomes using EDIT.

3.1 Base Chatbot System We implemented a chatbot with multi-component orchestration engine that manages the conversational flow as the base system for interviews. The orchestration is shown in Figure 3. Recent works on dialog agent suggests that prompting a frozen LLM to generate orchestration signals can facilitates the process of deciding what actions the dialogue agent should take to effectively to achieve specific goals during the dynamic interactions with users [14, 17, 32]. Inspired by those work, we similarly adopt an orchestrator-executor design to ensure smooth, consistent interview experiences throughout the chatbot session. This engine operates in a loop, processing each user message through a series of specialized LLM-powered modules to generate a contextually appropriate and protocol-adherent response and a smooth transition between follow-ups and the main questions in the designed semistructured interview (see the detailed prompts in Appendix B). We implemented an Executor-Auditor state-machine loop because in early pilot prototyping, a single-pass generator occasionally drifted and skipped required follow-ups, barely finishing a single interview. Adding lightweight quality and completion audits provided more reliable adherence, reaching 100% completion rate and topic coverage in our pilot testing.

3.1.1 The Orchestration Engine. The orchestration engine is responsible to translate the audit results from last step to the executable prompt that can be directly fed into the executor for the next generation. It controls the smooth transition of the state machine. For different tasks such as asking another follow-up or proceeding to the next main question, the orchestration engine would fetch the context needed accordingly.

3.1.2 The Executor Module. The primary conversational turn is generated by the Executor module. After receiving a user’s message, the orchestrator constructs a prompt for the Executor that includes the current main question, the conversation history, and a dynamically generated list of ALLOWED_ACTIONS (e.g., ASK_FOLLOWUP, NEXT_QUESTION). The Executor is tasked with generating a response in a structured JSON format that specifies which action to take and the corresponding utterance. The system prompt for this module explicitly instructs it to “elicit a concrete personal story” while maintaining a concise and conversational tone. This is because high-quality interview data requires interviewees to describe their actual experiences and feelings, rather than speaking in highlevel or abstract terms. Participants could still choose to say they do not have related experiences and skip any question.

3.1.3 The Auditing Layer. To ensure adherence to the research protocol, the Executor’s output and the user’s input are evaluated by an Auditing Layer composed of two distinct LLM-driven auditors. Completion Audit: The auditQuestionPSS function serves as the primary mechanism for advancing the interview. After a user provides a response, this auditor is invoked to determine if the conversation has sufficiently covered the predefined follow-ups associated with the current main question. It is prompted with the conversation history and a JSON list of required follow-up topics. Its verdict, either ALLOW_NEXT_QUESTION or REQUIRE_MORE, is used by the orchestrator to decide whether to transition to the next main question or to ask another follow-up. Quality Audit: To maintain conversational quality, the auditQuestionPresence function acts as a quality control check on the Executor’s generated utterance. It validates that the bot’s message is on-topic, well-formed (e.g., contains exactly one question), and does not prematurely drift to other topics.

Discussion. Our findings show promising evidence that the AI-aided postinterview editing feature improved privacy protection, as indicated by a significant reduction in PII, while not significantly affecting data quality or user engagement. In this section, we discuss the implications of our findings for designing ethical interview chatbots while balancing the need for high-quality data to ensure research validity.

6.1 Am I Disclosing? Discrepancy Between Perceived Safety and Objective PII Edits Participants’ perceived comfort and confidence did not necessarily track their objective privacy edits. Although free-editing and AI-aided users reported similar comfort (per post-task reflections), their final transcripts diverged: AI-aided responses contained far fewer residual PII tokens than both free-editing and control (Section 5.1). The fact that participants reduced more PII when explicitly reminded suggests that the additional sharing of PII in the control and free-editing groups may stem from a lack of awareness. The fact that participants reduced more PII in the AI-aided condition suggests that additional disclosure in the control and free-editing groups may stem less from active risk-taking and more from the power of defaults. Before any intervention, the implicit default is to leave the transcript unchanged, and prior work on privacy apathy [11] suggests that many people will simply stick with such a status quo even when given a manual editing option. AI suggestions effectively lower the effort required to deviate from this default: participants can accept targeted, privacy-preserving edits with minimal cognitive and interaction cost, which “unlocks” more PII reductions without requiring a corresponding rise in privacy concern. In other words, users unconsciously shared more personal information than they would have intended if fully informed, highlighting the power asymmetry between participants and researchers. We attribute the effective privacy protection in the AI-aided mode to two mutually reinforcing mechanisms. The first involves heightened awareness: AI suggestions bring to the fore specific spans that might seem harmless when disclosed individually but appear risky when considered together during review (e.g., names + locations + dates), prompting removals or abstractions that manual passes often miss. The second concerns convenience: by reducing the cognitive overhead of privacy work, AI suggestions make sanitization as simple as a single click (replace/abstract), enabling participants to perform more targeted, high-impact edits rather than broad, imprecise deletions. In contrast, we observed that although participants in the freeediting condition reported a similarly positive sense of privacy control and confidence in sharing as those in the AI-aided condition (Table 4), their edits often only superficially addressed privacy protection and at times even introduced additional PII by supplying more detailed answers (Section 5.3). Overall, we consider the free-editing mode to foster a potential false sense of safety. This perceptual-behavioral gap underscores that privacy risk is not a reliable intuitive perception; usable controls must translate vague safety intuitions into concrete, actionable edits at the span level.

6.2 Balancing Privacy, Engagement, and Data Quality A prevailing myth assumes that giving users meaningful privacy controls risks reducing data quality, an assumption reflected in research practices [74] and in companies’ use of dark patterns to make “both-edit” pathway. These patterns suggest that participants accepted the abstraction suggestions and then provide even more details via manual edits.

Conclusion. We conducted the first study on privacy controls for chatbot-based interviews, focused on two post-interview editing features: (1) free editing and (2) AI-aided editing. Our AI-aided editing workflow significantly reduced residual PII without degrading response quality or engagement. Subjective perceptions did not fully track the behavior: comfort and willingness to discuss sensitive topics were similar across conditions, yet AI-aided suggestions converted vague concerns into concrete, high-impact edits while preserving informativeness. The design implications are straightforward: make risk perceptible and actionable with in-line, span-level highlighting; favor reversible replace and abstract operations; keep suggestions conservative and stable; and employ domain-tuned taxonomies that prioritize high-risk entities. These choices support autonomy, enabling users to remain in control while the system guides their effort toward edits that matter for their privacy. We also observe heterogeneity in benefit, suggesting adaptive policies (e.g., stronger defaults or earlier prompts when early responses are dense with PII). Future work should test additional domains and populations, improve recall for high-risk spans, and explore real-time variants that preserve user agency. Our results, together with emerging evidence that user-facing, perceptible controls can minimize unnecessary disclosure, motivate a practical privacy-by-design layer for research and product chatbots. Taken together, our study positions post-hoc manual user edits with AI-aided hints as a practical complement to the method of privacy control in traditional interview procedures, showing a viable path for HCI reseachers to embed privacy-by-design into chatbot workflows in a less invasive way without giving up rich, analyzable data.

Limitations. 7 Limitation While our study provides initial evidence that AI-aided post-interview editing can meaningfully reduce privacy risks without compromising data quality, several limitations should be noted.