Large Language Models as Conversational Movie Recommenders: A User Study
This paper explores the effectiveness of using large language models (LLMs) for personalized movie recommendations from users’ perspectives in an online field experiment. Our study involves a combination of between-subject prompt and historic consumption assessments, along with within-subject recommendation scenario evaluations. By examining conversation and survey response data from 160 active users, we find that LLMs offer strong recommendation explainability but lack overall personalization, diversity, and user trust. Our results also indicate that different personalized prompting techniques do not significantly affect user-perceived recommendation quality, but the number of movies a user has watched plays a more significant role. Furthermore, LLMs show a greater ability to recommend lesser-known or niche movies. Through qualitative analysis, we identify key conversational patterns linked to positive and negative user interaction experiences and conclude that providing personal context and examples is crucial for obtaining high-quality recommendations from LLMs.
Discussion / Conclusion. Our findings showed that few-shot prompts didn’t significantly improve recommendation quality compared to zero-shot or one-shot approaches. This aligns with Dai et al., who noted that recommendation quality doesn’t always improve with more examples [13]. To address the lack of personalization and novelty in our findings, retrieval-augmented generation (RAG) has been proposed, allowing The success of LLM recommenders cannot be achieved without well-informed users with proper goals and interaction flows. The first learning we summarize from the study is to remind users to limitations could be valuable for future research to enhance understanding in this domain. 6 CONCLUSION
Lines of inquiry this paper opens 24
Research framings built by reading the notes related to this paper — the questions it feeds into.
What tradeoffs between accuracy, cost, and customization optimize recommendation systems?- Why do LLM recommenders drop 60 percent recall when missing collaborative signals?
- How does collaborative filtering integrate into LLM-based recommendation systems?
- Why is popularity bias harder to fix in LLM recommenders than in collaborative filtering?
- Do pretraining biases and traditional selection bias compound in production recommenders?
- Why do LLM recommenders underperform item-only collaborative filtering baselines?
- How does pretraining corpus popularity bias affect LLM recommendation behavior?
- Which deployment domains favor LLM recommenders over traditional collaborative approaches?
- Why does inductive bias outweigh model capacity in recommender systems?
- Does input augmentation outperform direct language-based recommendation systems?
- What efficiency costs does unified language modeling impose versus specialized recommenders?
- How do large pretrained language models scale the unified recommendation paradigm?
- Can dataset-level debiasing methods fix popularity bias inherited from pretraining?
- Can post-hoc reranking actually fix popularity bias created during model training?
- Which LLM recommender paradigm actually performs best empirically?
- How do different LLM integration paradigms affect inheritance of pretraining biases?
- How do LLM biases manifest differently across the three paradigms?
- How do LLM biases reflect social classification schemas rather than random errors?
- Why do review corpora contain biases that affect generated comparisons?
- Why do LLMs show gender bias but humans evaluators do not?
- Why do LLMs inherit causal biases from their training data?
- Can implicit association tests reveal LLM biases beneath trained responses?