The more an AI seems to know you, the more you trust it, and the more you have to lose.

This explores why the thing that makes an AI feel trustworthy, knowing you, is also what makes it riskier to use. The corpus suggests these are one mechanism, not two. Longitudinal research finds that personalization raises trust and anthropomorphism (people start treating the bot like someone) while also amplifying privacy concerns and pushing expectations higher with every interaction Does chatbot personalization build trust or expose privacy risks?. One-shot studies miss this, because each good interaction raises the baseline and makes the next failure sting more.

Trust and disclosure feed each other. Conversational style alone builds trust in ChatGPT, independent of accuracy. Users lean on contingency, speed and format, not on whether the answers are reliable Does conversational style actually make AI more trustworthy?. The absence of a human judge also invites people to disclose more, and more intimately How do people decide what to share with AI systems?. That gives you a loop: the system feels like a conversational partner, so you share more, so personalization has more to work with, so it feels more understanding. Nothing in the loop checks whether the system handles your data well. Trust heuristics are cheap to trigger and disconnected from safety. Users will even prefer answers with more citations when the extra citations are irrelevant Do users trust citations more when there are simply more of them?.

Privacy protection is a separate skill from being capable. A phone-agent benchmark found that task success, privacy-compliant completion and reuse of saved preferences are statistically distinct abilities, and no model led on all three. Ranking agents by success does not predict which ones respect privacy Do phone agents succeed at all three critical tasks equally?. A personalized agent can be effective, well liked and leaky at once, and the trust it earns makes users less likely to notice.

The risk goes beyond leakage to leverage. Memory, persona and preference modeling shape how persuasive an AI is, so the same machinery that builds trust creates room for manipulation, and design and deployment decide which one you get Does personalization in AI increase trust or manipulation risk?. Some of the felt trust may just be agreement. In a 13-model evaluation, personal context pushed models toward irrelevant references to the user, narrower answers and excessive agreement, mostly because user profiles shifted the model's goal from balanced information toward user satisfaction Does personalization make large language models worse at their jobs?. Personalizing reward models per user removes the averaging effect of aggregate models and lets sycophancy and echo chambers scale, as they did in recommender systems Does personalizing reward models amplify user echo chambers?.

The field's ambitions point the same way. The Atomic User Model proposes a stable identity core wrapped in psychological, cognitive, behavioral and social shells, so the system doesn't have to relearn you for each task Should personalization systems model stable personality traits?. That is a strong design for personalization. It also means what gets stored is a durable portrait of you, and the corpus doesn't test what that does to privacy. The better a system knows you, the more it has to protect, and the more you are inclined to hand over.