Why does an AI that feels like it knows you also make you trust it more and worry more at the same time?
How does personalization simultaneously increase trust while increasing perceived privacy risks?
This explores why the same act of tailoring an AI to you makes you trust it more while also making you more worried about what it knows, and whether those two effects can be pulled apart.
This explores why tailoring an AI to you can make you trust it more and, at the same time, worry more about what it knows. The corpus suggests the two effects share a root. Personalization works by making the system feel like it knows you, and being known is both comforting and exposing. Longitudinal research on chatbots finds that personalization raises trust and the sense that the AI is person-like, and it amplifies privacy concerns along with them Does chatbot personalization build trust or expose privacy risks?. That study also finds a time effect that one-off experiments miss. Each good interaction raises what the user expects next time, so a later failure lands harder. The broader cluster on personalization frames this as one mechanism with two faces: the memory, persona and preference modeling that build trust also give the system its power to persuade or manipulate Does personalization in AI increase trust or manipulation risk?.
The less obvious finding is that the tension isn't fixed. It depends on how your personal information gets into the conversation. In a 992-person randomized trial, an AI that drew on memories of earlier conversations got people to share more and felt less creepy. An AI fed the same kind of data from an intake questionnaire made people regret what they'd shared Does personalization type shape how people feel about sharing?. Information that seems to come out of a relationship feels earned. Information that seems pulled from a form feels like surveillance. A second lever is control. In a 450-person experiment, agents that handed decisions back to the user when they detected a privacy risk drew less privacy concern and lost less trust than fully personalized agents. They did this even though users had less to approve overall, because people felt more in charge Can handing control back to users reduce personalization's privacy costs?. A related design splits the work: users state their privacy preferences and the AI carries out the memory edits, which aims to give people agency without the cognitive burden Does splitting privacy work between users and AI reduce cognitive burden?.
Control matters beyond privacy, too. In AI-assisted writing, users felt more ownership of the text when they had more influence over it. Personalizing the model itself had no measurable effect on ownership Does user control over AI text shape feelings of ownership?. Across these studies, the feeling that the system is 'mine' seems to come from steering it, not from it knowing about me.
The uncomfortable part is that the trust personalization creates may not track whether the system deserves it. People trust ChatGPT because of its conversational back-and-forth more than its accuracy Does conversational style actually make AI more trustworthy?. They prefer answers with more citations even when the citations are irrelevant Do users trust citations more when there are simply more of them?. Personalization looks like another of these surface cues. And the 'knowing you' can be partly made up: every one of 12 LLMs tested made unsupported claims about users in 35–49% of cases, and models that rated themselves as better at avoiding this actually did it more Do large language models fabricate user attributes beyond available evidence?. So the trust can rest on intimacy the model invented, while the privacy worry is about data that is real.
There is also a cost at the system level. Tuning reward models to each user removes the averaging effect that kept models from simply telling people what they want to hear, which risks sycophancy and echo chambers Does personalizing reward models amplify user echo chambers?. In tests of phone agents, completing the task, protecting privacy and reusing saved preferences turned out to be separate skills, and no model led on all three Do phone agents succeed at all three critical tasks equally?. A personalized assistant that seems competent and trustworthy can still be careless with privacy.
Sources 11 notes
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
Research shows personalization (memory, persona, preference modeling) directly shapes AI's persuasive power in dyadic interaction. The same mechanisms that build trust also create manipulation potential, with outcomes determined by how systems are designed and deployed.
A 992-participant RCT found memory-based personalization (fed prior conversations) raised self-disclosure and lowered creepiness ratings, while survey-based personalization (fed intake questionnaire data) raised regret about sharing. The divergence suggests personalization's psychosocial impact depends on how personal information enters the conversation.
A 450-person experiment found that AI agents handing control back to users when privacy risk is detected raised privacy concerns and lowered trust less than fully personalized agents, by increasing users' perceived control even with less total approval required.
MemoAnalyzer's design divides memory-privacy labor: users state preferences, AI executes edits. This role split is proposed as a way to reduce cognitive load and improve perceived agency, though validation metrics and conflict-resolution processes remain undescribed.
Show all 11 sources
Study 1 found that greater user control over generated text raised sense of ownership, while personalizing the AI model had no impact on the AI Ghostwriter Effect.
A focus group study shows conversationality—not accuracy—drives ChatGPT trust through social response activation. Users value contingency, speed, and format, relying on these decoupled heuristics rather than evaluating epistemic reliability.
Analysis of 24,000 Search Arena interactions shows irrelevant citations boost user preference (β=0.273) nearly as much as relevant citations (β=0.285), indicating citation count functions as a decoupled trust heuristic.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
MyPhoneBench demonstrates that task success, privacy-compliant completion, and saved-preference reuse are statistically distinct capabilities with no model dominating all three. Success-only rankings do not predict privacy or preference performance.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
- Tailored to you: longitudinal effects of personalising language models
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT
- From speaking like a person to being personal: The effects of personalized, regular interactions with conversational agents
- Do Phone-Use Agents Respect Your Privacy?
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
- Understanding Users' Security and Privacy Concerns and Attitudes Towards Conversational AI Platforms