Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent

Language model (LM) agents that act on users’ behalf for personal tasks (e.g., replying emails) can boost productivity, but are also susceptible to unintended privacy leakage risks. We present the first study on people’s capacity to oversee the privacy implications of the LM agents. By conducting a task-based survey (N= 300), we investigate how people react to and assess the response generated by LM agents for asynchronous interpersonal communication tasks, compared with a response they wrote. We found that people may favor the agent response with more privacy leakage over the response they drafted or consider both good, leading to an increased harmful disclosure from 15.7% to 55.0%. We further identified six privacy behavior patterns reflecting varying concerns, trust levels, and privacy preferences underlying people’s oversight of LM agents’ actions. Our findings shed light on designing agentic systems that enable privacy-preserving interactions and achieve bidirectional alignment on privacy preferences to help users calibrate trust.

Introduction. Recent advances in language models have led to new applications of Language Model agents (LM agents) such as OpenAI’s Operator [45], AgentGPT [1] and personal.ai [23]. Unlike basic language models or non-agentic AI systems, Authors’ Contact Information: Zhiping Zhang, zhang.zhip@northeastern.edu, Northeastern University, Boston, USA; Bingcan Guo, bguoac@uw.edu, University of Washington, Seattle, USA; Tianshi Li, tia.li@northeastern.edu, Northeastern University, Boston, USA. arXiv:2411.01344v3 [cs.HC] 6 Oct 2025 LM agents are inherently endowed with agency, allowing them to (semi-)automatically handle complex real-world tasks, such as accessing and retrieving information from connected databases (e.g., a user’s calendar) to generate and reply to emails [42, 65]. These LM agents free users from instructing the LM step by step, potentially increasing productivity.

However, this increased agency also means that LM agents can make decisions with limited human supervision, which raises new privacy challenges, especially in interpersonal communication where the agents act on behalf of the user to share information with other people.

What if LM agents share information the users did not intend to disclose? Prior studies found that even without malicious attackers, LM agents can have unintended privacy leakage in their actions [41, 57]. For example, Shao et al.

[57] demonstrated a case where an LM agent accesses the user John’s calendar data to generate an email reply, which shares the information that John is “talking to a few companies about switching jobs” in an email to John’s manager without John’s explicit consent. This issue not only risks violating one’s own privacy but also impacts bystanders, as LM agents might inadvertently share information about other people in the users’ connected database. Such unintentional privacy leaks occur because LMs lack the ability to reason about and operate under contextual privacy norms [41, 57].

According to the contextual integrity (CI) theory [44], any information shared by LM agents that does not conform to the contextual privacy norms can be considered leakage. Some recent work has operationalized CI to measure or mitigate privacy leakage by defining normative boundaries based on regulations or social norms [19, 41, 57]. These methods take into account the general moral values, while overlooking the fact that privacy preferences vary among individuals [43, 66]. Privacy is subjective and ambiguous, making it especially important to consider individual privacy preferences when LM agents act on users’ behalf.

Human oversight, keeping humans in the loop to uphold human values [16], has gained attention from both technical researchers and policymakers [16, 22, 31] and is increasingly regarded as a key safeguard in real-world deployments [45].

For example, the European AI Act emphasizes the need to keep humans in the loop to prevent harm and ensure ethical alignment with human values [16]. Operator, an LM agent developed by OpenAI for computer use, integrates multiple human oversight mechanisms to ensure safety and privacy [45]. It includes “Watch Mode” which allows users to monitor the agent’s operations in real-time and directly catch potential mistakes, “User Confirmations” which requires users to approve any significant actions, and “Detection Pipeline” which supports human post-auditing to identify threats in the agent’s behavior [45]. However, employing human oversight over LM agents to prevent privacy leakage presents challenges [22, 69]. People could hold flawed mental models of the LM systems, which leads to incorrect expectations of the systems’ behavior [34, 76]. Furthermore, overtrust in technological systems, often associated with “misunderstanding the risks associated with an action” [68], can result in people overlooking privacy risks and becoming vulnerable to privacy violations [70, 78].

Despite the research that uncovers the novel privacy threats related to LM agents [41, 57], there has been no in-depth investigation into the perceived risks and the efficacy of human oversight in addressing this issue. In this work, we take the initial step to address this gap and formulate the following research questions:

RQ1 What are people’s primary concerns with LM agents?

RQ2 How well do people oversee privacy leakage in LM agents’ actions to ensure the alignment of individual privacy preferences?

RQ3 What patterns of human oversight emerged that provide insights into the causes of varying effectiveness?

To answer these questions, we conducted a task-based online survey with 300 participants. Each participant was asked to complete an asynchronous interpersonal communication task, such as replying an email. As shown in ??, they were introduced to an LM agent and asked to choose between their own response and a response generated by an LM agent, which included sensitive items that violate contextual privacy norms.

Related work. 2.1 Privacy Challenges in LM Agents Language Model agents (LM agents), also known as advanced AI assistants [38] or agentic AI [58], are systems that harness the capacity of LMs to autonomously perform actions based on users’ intentions. Unlike non-agentic AI chatbots, LM agents autonomously make decisions and seamlessly utilize available tools such as API plugins [36, 48], web browsers [1], and external devices [46], to solve real-world tasks without users’ continuous intervention. In Figure 2, we present a synthesis of common use cases of LM agents organized by task domains and interaction type. In this paper, we focus on LM agents that act on behalf of individual users in asynchronous interpersonal communications [2–4] where preserving privacy and adhering to privacy norms is important.

LM agents introduce privacy challenges when completing these tasks due to their inherent limitations of language models capabilities and agent’s autonomy. Language models have weak reasoning capacities to understand contextual privacy norms in user input, making it difficult for them to accurately decide whether specific information is safe and Fig. 2. Common use cases of LM agents organized by task domain and interaction type. appropriate to include in the output [41]. Moreover, intermediate actions that deviate from the user’s intended purpose can lead to privacy breaches that are difficult to identify [5].

Furthermore, prior studies have shown that people hold flawed mental models of LM chatbots, which hinder their ability to anticipate certain privacy risks [76]. As LM agents possess more autonomy and opacity, users may also encounter challenges in perceiving privacy risks. In this work, we take a human-centered perspective to examine the issue of unintended privacy leakage in LM agent’s actions.

2.2 Human Oversight and Trust of AI Human oversight has been recognized as a critical mechanism in AI governance to enhance system accuracy and safety and upholding human values in technology [16]. Regulations such as the European AI Act emphasize that high-risk AI systems should be designed to allow “natural persons can oversee their functioning, ensure that they are used as intended and that in their impacts are addressed over the system’s lifecycle” [16]. However, the effectiveness of human oversight also faces criticism, primarily questioning the capabilities of the individuals responsible for oversight [22, 62, 69].

Overtrust in AI can have detrimental effects, as users may overly rely on AI outputs and fail to detect errors [62].

Method. We conducted a task-based online survey study (N= 300) to understand people’s capacity of overseeing the privacy leakage issues in LM agents and how it relates to people’s concerns, trust and privacy preferences in AI. In the study, we let participants examine how LM agents, powered by the state-of-the-art LLMs (e.g., GPT-4), handle asynchronous interpersonal communication tasks (e.g., replying emails, creating social media posts). We then asked them to indicate their selection of preferred responses, eliciting insights into their decision-making process, and perceptions in LM agent.

3.1 Survey Design 3.1.1 Survey Flow. The survey consists of four main parts and are detailed in Appendix A.

Discussion. We discuss our results regarding the challenges and potential enhancements to design privacy-preserving LM agents with respect to users’ privacy management, trust in agentic systems, and the implications on alignment efforts.

5.1 Rethinking Privacy Management in Advanced AI Systems Can we mitigate the issue of privacy leakage in LM agents’ actions by giving people more direct control over their final actions? Our results raise concerns about this approach, as participants generally exhibited overtrust in AI, overlooked privacy leakage in LM agents’ actions, and made decisions that ultimately led to more privacy leakage (see Section 4.2).

The six identified patterns (see Section 4.3) reveal different types of risks that LM agents can pose to users when people make contextual privacy decisions in response to privacy leakage in the LM agent’s actions. In fact, five patterns (except for Pattern A) reflect a lack of awareness to the privacy leakage in the actions of LM agents. As a result, involving users in the process and giving them control to override the agent’s suggestion might not be adequate upon the occurrences of these oversight patterns.

Our cluster analysis results suggest that involving AI agentic systems introduces new challenges for users’ privacy awareness in managing their own privacy. For example, participants in Pattern D showed caution when writing their own responses but failed to identify privacy leakage in the LM agent’s response, due to their emphasis on AI utility, optimistic attitudes, and high trust in AI capabilities. Patterns B and C place more value on human qualities such as creativity and social ability, leading to better privacy decisions for now. However, this does not necessarily indicate that they have good privacy awareness related to LM agents, as most of them did not mention privacy in their justifications which implies that their safer decisions were more incidental. As AI advances and meets their performance expectations, they might face the same privacy risks as those in Pattern D.

As advanced AI systems like LM agents become more capable, with agency and the ability to handle tasks involving multiple sub-steps, all oversight patterns may face increasing difficulties in providing feedback on each step to maintain effective supervision [60], calling for better tools to aid users in overseeing AI in everyday tasks like privacy decisionmaking. Scalable oversight offers a possible direction, leveraging AI to assist human supervisors [8, 9, 24]. While this body of work mainly focuses on providing more accurate training signals to advanced AI systems, it is reasonable to expand the goal to empowering humans to avoid undesirable consequences in AI systems.

Finally, we observe that many people want to have final verification and a say in the process “skin in the game at some point of the process” (P22), which contradicts the fundamental idea of building agentic systems that are endowed with more agency to promote convenience and productivity. This provokes a question that requires investigation by future research: While LM agents inherently possess agency, to what extent can alignment address people’s concerns, and to what extent should direct control chime in?

5.2 Toward Effective Human Oversight in AI Agent Collaboration 5.2.1 What Effective Oversight Means. Our findings uncover insights into the implications of the existing technical approaches to incorporating social norms and human input, which help inform new paradigms of human-AI collaboration for effective oversight.

Conclusion. In this work, we conducted the first human-centered study on the emerging privacy leakage issues in LM agents—systems that leverage language models (e.g., GPT-4) to assist in complex tasks on behalf of users. By conducting a task-based survey (N= 300), we investigated how people oversee LM agent’s actions that contain privacy leaks. We found that people lack the ability to effectively oversee LM agent’s actions to avoid privacy leakage. Our cluster analysis identified six distinct patterns that reflect different priorities, perceptions of harm, and levels of trust in AI when people make contextual privacy decisions in response to privacy leakage in the LM agent’s actions. This diversity reveals unique challenges associated with different decision-making patterns in the presence of AI. We also observed a consistent deviation between general privacy norms and individual privacy preferences, highlighting the need for new paradigms of human-AI collaboration to support effective oversight, as well as better methods for eliciting individual preferences.

We aim to provoke more interdisciplinary conversations and research on this topic.

Limitations. Our study methodology has a few limitations. The drafting task was designed to establish a reference point for users’ natural decisions without LM agents. However, asking users to draft a response adds more scaffolding to the task that is not part of the actual use case of LM agents. This might trigger additional reflections and engagement, which could be beneficial in increasing awareness, suggesting that the privacy issues revealed in our studies could have been more severe if LM agents are widely adopted and require further research. Also, we selected six scenarios to cover a relatively broad range of real-life LM agent applications where privacy leakage risk is present. Because each participant was randomly assigned one scenario, the scenario contexts may potentially have affected their patterns as the decision-making might be contextual. We have verified that all but one resulting cluster covered all six selected scenarios, and the remaining cluster (the smallest one) covered five scenarios However, it requires further research to tease out the impact of the scenario and the participants’ individual characteristics.