People picked AI replies that leaked their private info over their own drafts more than half the time — without noticing.

This explores whether an AI that infers personal things about you gets a pass when its guesses are right and useful for what you're doing. The corpus doesn't test that trade-off head-on. What it does show is that the trade-off people seem to make usually isn't a deliberate one. In a 300-person survey, participants picked AI-written replies that leaked private information over their own drafts 55% of the time Do people notice when AI agents leak private information?. That looks like usefulness beating privacy. The researchers trace it to overtrust and to people not noticing the leak, though, not to a conscious choice. Usefulness wins by default because the privacy cost never shows up.

That pattern repeats. In a 40-person study, most users never noticed that their LLM's memory held inferred private information. Once they were shown it, every participant wanted controls to edit and delete it Do users notice when LLMs infer and retain private information?. A related study found that people's sense of privacy depends on the editing interface they used, not on how much identifying information they actually removed Why do people feel safe sharing when they haven't actually removed sensitive information?. So privacy concerns don't seem to be outweighed by accuracy. They seem to be asleep, and they wake up when the inference becomes visible. Over time, personalization raises trust and privacy worry together rather than trading one for the other Does chatbot personalization build trust or expose privacy risks?.

The accuracy side is also weaker than the question assumes. A benchmark of 12 LLMs found that every one over-infers: 35–49% of their claims about users go beyond the available evidence. Models that rate themselves as more careful are actually worse Do large language models fabricate user attributes beyond available evidence?. 'It's fine because it's accurate' is a shaky defence when a large share of the inferences are confident guesses drawn from stereotypes in the training data rather than from you.

The usefulness side is real, though, and that's the uncomfortable part. When models reason through a task, they recall private user data inside their reasoning traces. Removing that data afterward makes the models worse at the task, which suggests the private details act as working material for the reasoning Do reasoning traces actually expose private user data?. In agents acting for several parties, prompting, distillation and RL only move a model along a fixed trade-off between leaking and refusing too much. None of them escapes it Can training methods escape the leak versus over-refusal tradeoff?. So usefulness and privacy pull against each other inside the model too, not just in users' heads.

The practical answer the corpus points to is visibility and control rather than one side overriding the other. Users worry most about what gets collected in the first place, ahead of how it's used What privacy concerns matter most to ChatGPT users?. Designs that let users state preferences while the AI does the editing aim to make that control manageable Does splitting privacy work between users and AI reduce cognitive burden?. The surprise is that the 'privacy paradox' here may be less about people knowingly trading privacy for convenience and more about the inference never being shown to them.