Do people notice when AI agents leak private information?

Zhang, Guo, and Li ran a task-based online survey (N=300) on human oversight of privacy in language model agents. Each participant completed an asynchronous interpersonal communication task, such as replying to an email, then chose between the response they drafted themselves and a response generated by an LM agent "powered by the state-of-the-art LLMs (e.g., GPT-4)." The agent's response contained sensitive items that violated contextual privacy norms. The authors report: "people may favor the agent response with more privacy leakage over the response they drafted or consider both good, leading to an increased harmful disclosure from 15.7% to 55.0%."

The paper attributes the jump to overtrust rather than to a considered privacy trade-off: "participants generally exhibited overtrust in AI, overlooked privacy leakage in LM agents' actions, and made decisions that ultimately led to more privacy leakage." It frames leakage using contextual integrity (CI) theory — information counts as leaked when it is shared in a way that does not conform to contextual privacy norms — and notes that CI-based mitigations tend to encode general social norms while "overlooking the fact that privacy preferences vary among individuals," which is part of why giving a person the final choice does not reliably catch a leak that violates their own, individual preference.

This is a measured instance of the failure that Does granting agents more autonomy undermine human oversight? argues abstractly: here, users are given the maximal form of oversight — a side-by-side choice with no obligation to pick the agent's answer — and still choose the leakier one at a majority rate. It also sits in tension with Does chatbot personalization build trust or expose privacy risks?: that longitudinal study finds sustained personalization raises perceived privacy risk over repeated interactions, while this single-shot comparison finds risk going unnoticed at the moment of choice, suggesting elevated privacy concern as a general attitude does not automatically transfer into catching a specific leak. It also contrasts with What makes people distrust AI agents they delegate to?, where trust breaks only after an irreversible, visible action has already happened; here trust is extended before any action occurs, at the point of selecting which draft to send.

The excerpt does not give demographics, per-scenario breakdowns, effect sizes, or which models beyond "e.g., GPT-4" were used to generate the agent responses; those details, along with the six underlying oversight patterns behind the aggregate rate, are reported separately (see What privacy oversight patterns emerge when people review AI agent responses?). The authors also flag that the drafting task added scaffolding absent from real LM agent use, which may have made participants more reflective than they would be in practice — implying the 55.0% figure could understate real-world leakage rather than overstate it. The strongest implication the data support is narrow but practical: presenting a user with both options and letting them pick is not, by itself, an adequate privacy safeguard.