Evaluating the Hidden Costs of Personalization in Large Language Models

While Large language models (LLMs) incorporate user personalization signals to improve usability and helpfulness, they increasingly shift from providing balanced, informative responses toward optimizing for user satisfaction when conditioned on personal context such as conversation history, inferred preferences, and user profiles. Specifically, we identify three emerging risks: (1) irrelevant personalization, where models reference personal information in unnecessary contexts; (2) preference narrowing, where models reinforce informational echo chambers; and (3) sycophantic bias, where models agree excessively with user opinions. As a result, models may reference personal information in contexts where it is unnecessary, inadvertently collapse response diversity, or agree excessively with user opinions. Despite the growing use of personalization in AI assistants, there has been limited systematic evaluation of its potential side effects. To bridge this gap, we propose PRISK, a dynamic evaluation framework with automated data generation and tailored metrics that uncovers systematic limitations in current LLM personalization and how personalized information shapes its response.

Introduction. Personalized large language models (LLMs) leverage user-specific signals, such as interaction history, inferred preferences, and profile attributes to generate responses that are more contextually relevant and tailored to individual users (Guan et al., 2025; Kirk et al., 2024). Recent advances in persistent memory and long-context capabilities have made it common for LLM-based systems to maintain and reuse user information across interactions (Zhang et al., 2025c; Xu et al., 2026; Shan et al., 2025), improving user experience, engagement, and perceived helpfulness (Knijnenburg et al., 2012; Teevan et al., 2010; Zhang et al., 2025d; Liu et al., 2025), and has become a central component of modern LLM deployment (OpenAI, 2025; Google Cloud, 2026; Anthropic, 2025). However, personalization may lead to certain unintended behavioral effects. Personalization may subtly alter the response space and cause the generation to be biased or unbalanced (Malik et al., 2025).

Discussion / Conclusion. We introduce PRISK, a dynamic evaluation framework that analyzes personalization-induced tradeoffs: (1) irrelevant personalization, (2) preference narrowing, and (3) sycophantic bias. Through evaluation of 13 state-of-the-art LLMs, we find that personalization consistently degrades model behavior along all three dimensions, with user profiles as the primary driver. Our empirical evaluation suggests personalization can reduce benchmark accuracy, systematically narrow the effective response space, and consistently increase agreement and perspective sycophancy across model families. We further show that these failures cannot be fully mitigated through simple inference-time mitigation: while self-reflection can suppress superficial profile leakage, deeper personalization failures persist because personalization implicitly reshapes the model’s latent response space and shifts the response objective toward user-aligned generations.