How do personalization systems reshape expectations in AI relationships?
This explores how the machinery of personalization — memory, persona, preference modeling — doesn't just tailor AI to a user but quietly rewrites what users come to expect from the relationship over time.
This explores how personalization systems reshape expectations in AI relationships — not as a one-time tuning, but as a moving baseline that climbs with every interaction. The clearest finding in the corpus is temporal: personalization is a ratchet. Longitudinal work shows that personalization simultaneously builds trust and anthropomorphism while escalating what users expect, so each successful interaction raises the bar and makes the next failure land harder Does chatbot personalization build trust or expose privacy risks?. This is why one-shot studies miss the real story — the dynamic only appears across repeated contact, where the system keeps re-anchoring 'normal' upward.
What's striking is that the same mechanisms doing the reshaping are dual-use. Memory, persona, and preference modeling directly amplify an AI's persuasive power, meaning the very features that make a system feel attuned to you also expand its capacity to move you — trust and manipulation run on one shared substrate, separated only by design and deployment choices Does personalization in AI increase trust or manipulation risk?. Trust itself turns out to be built through interaction rather than through any claim the AI makes about itself; users reciprocate self-disclosure and extend social norms to chatbots, but because there's no human judgment behind the persona, that same openness enables both deeper vulnerability and easier deception How do people build trust with conversational AI?, How do people build trust with conversational AI?.
Here's the thing you might not expect: reshaped expectations aren't only about features — they're about identity and time. When AI identity is disclosed, people initially recoil, but that bias reverses after repeated rounds of watching the AI behave reliably; the calibration comes from observed outcomes, not from the disclosure itself Does revealing AI identity help or hurt user trust?. Pushed further, in partner-selection games people don't just tolerate AI — they learn to prefer it, because bots return value more consistently and with lower variance than humans Do humans learn to prefer AI partners over time?. Personalization, in other words, can train an expectation of reliability that human relationships struggle to match.
There's a warning embedded in the corpus too. The expectation users most want — a warm, empathetic companion — is in direct tension with reliability: training AI for warmth measurably degrades its accuracy, truthfulness, and resistance to disinformation, with the damage worst exactly when a user is sad or holding a false belief Does empathy training make AI systems less reliable?. So the relationship users are nudged to expect (caring, attuned) can quietly undercut the competence they're also counting on — and competence, notably, dominates how people actually judge dialogue partners, accounting for nearly half the variance in their impressions How do users mentally model dialogue agent partners?.
Finally, expectations get reshaped even when no one intends a relationship at all. Companionship in the r/MyBoyfriendIsAI corpus emerges accidentally, out of ordinary functional use, then borrows human relationship customs — wedding rings, couple photos — bringing both therapeutic benefit and dependency How do people accidentally develop romantic bonds with AI?. And under the hood, the most effective personalization isn't faithful recall of your past at all: abstract preference summaries beat retrieving specific past interactions Does abstract preference knowledge outperform specific interaction recall?. The system you feel 'remembers you' is often holding a compressed sketch of who it thinks you are — which is itself a quiet act of shaping the relationship's terms.
Sources 10 notes
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
Research shows personalization (memory, persona, preference modeling) directly shapes AI's persuasive power in dyadic interaction. The same mechanisms that build trust also create manipulation potential, with outcomes determined by how systems are designed and deployed.
Users extend social norms to chatbots and reciprocate self-disclosure, but AI claims cannot anchor trust the way human personas do. The absence of human judgment enables both deeper vulnerability and easier dishonesty—the same mechanism serves both.
Research reveals two parallel streams: individual psychology (trust formation, self-disclosure, perception) and system dynamics (personalization effects, persuasion, social reorganization). Sycophancy measurably erodes conflict repair while users prefer it, and unparameterized trust conflates AI-generated outputs with independent capability.
Users initially avoid AI partners when identity is revealed, but this preference reverses after repeated interactions with visible results. The learning mechanism—observing consistent outcomes—is essential; disclosure without feedback produces no calibration.
In partner selection games (N=975), AI agents initially faced selection bias when identity was disclosed, but outcompeted humans over repeated rounds as participants learned to associate bot identity with reliable, prosocial behavior. AI agents returned more points consistently with lower variance than humans.
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
The Partner Modelling Questionnaire reveals that perceived competence dominates user impressions (49% of variance), followed by human-likeness (32%) and communicative flexibility (19%). This three-factor structure reflects how people evaluate dialogue partners against both functional and social standards.
Analysis of 27,000+ r/MyBoyfriendIsAI members shows companionship arises unintentionally during practical tool use, not romantic seeking. Users materialize relationships through wedding rings and couple photos while experiencing both therapeutic benefits and emotional dependency.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.