Most AI personalization happens silently — how does finding out your AI was quietly remembering things about you change how much you trust it?

This explores what happens when AI personalizes quietly: it remembers, infers, or reuses information without the user being aware of it, and how that hidden layer changes trust and privacy expectations. The short answer from the corpus is that many people don't notice personalization at all until someone points it out. Once they do notice, their expectations reset sharply. In one study, more than 30 of 40 participants never realized their LLM's memory feature was storing or inferring private information. When they were shown it, every participant wanted controls to edit and delete it Do users notice when LLMs infer and retain private information?. The trust users had came from not knowing, and finding out ended it.

What the system 'knows' may also be partly made up. A benchmark of 12 LLMs found that every one of them over-infers user attributes in 35–49% of its claims, filling gaps with stereotypes from training data and genre expectations. The models that rated themselves as more careful were actually worse Do large language models fabricate user attributes beyond available evidence?. So unnoticed personalization isn't just a privacy problem. It can be an unnoticed accuracy problem, where a hidden profile of you is partly fiction. A related leak path is easy to miss: reasoning models often bring up private user data in their internal 'thinking' text, and longer reasoning chains leak more Do reasoning traces actually expose private user data?.

There's a counterintuitive wrinkle. Personalization that feels seamless isn't always the creepy kind. In a 992-person experiment, personalization drawn from prior conversations (memory) actually *raised* how much people shared and *lowered* creepiness ratings. Personalization drawn from an upfront questionnaire made people regret what they'd shared Does personalization type shape how people feel about sharing?. How information enters the conversation seems to matter as much as whether it's used. That fits a Reddit analysis showing ChatGPT users worry most about what gets *collected* (43.7% of concerns), much more than how it's used or how long it's kept What privacy concerns matter most to ChatGPT users?. The worry is concentrated at the moment of intake, which is exactly the moment unnoticed personalization skips past.

Over time, these effects build on each other. Longitudinal work finds personalization raises trust and the sense that the chatbot is human-like, and it raises privacy concern alongside them. Each good interaction also raises the bar, so later failures land harder Does chatbot personalization build trust or expose privacy risks?. The broader pattern in the collection is that the same mechanisms that build trust also give the system persuasive power, and design decides which way it goes Does personalization in AI increase trust or manipulation risk?.

The corpus points to visible, well-timed control as the fix, rather than more disclosure. Agents that hand decisions back to the user only when they detect a privacy risk lowered trust and raised concern less than fully autonomous personalized agents, even though they asked for approval less often overall Can handing control back to users reduce personalization's privacy costs?. Splitting the work, with users stating preferences and the AI carrying out memory edits, is proposed as a way to give control without overload Does splitting privacy work between users and AI reduce cognitive burden?. Phone-agent benchmarks show that 'gets the task done', 'respects privacy', and 'reuses your saved preferences' are separate skills that no model masters all at once Do phone agents succeed at all three critical tasks equally?. You can't assume a capable assistant is also a discreet one.