Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning

Conversational recommender systems (CRS) enable the traditional recommender systems to explicitly acquire user preferences towards items and attributes through interactive conversations. Reinforcement learning (RL) is widely adopted to learn conversational recommendation policies to decide what attributes to ask, which items to recommend, and when to ask or recommend, at each conversation turn. However, existing methods mainly target at solving one or two of these three decision-making problems in CRS with separated conversation and recommendation components, which restrict the scalability and generality of CRS and fall short of preserving a stable training procedure. In the light of these challenges, we propose to formulate these three decision-making problems in CRS as a unified policy learning task. In order to systematically integrate conversation and recommendation components, we develop a dynamic weighted graph based RL method to learn a policy to select the action at each conversation turn, either asking an attribute or recommending items. Further, to deal with the sample efficiency issue, we propose two action selection strategies for reducing the candidate action space according to the preference and entropy information.

Introduction. Conversational recommender systems (CRS) aim to learn user’s preferences and make recommendations through interactive conversations [12, 16, 41]. Since it has the natural advantage of explicitly acquiring user’s preferences and revealing the reasons behind recommendation, CRS has become one of the trending research topics for recommender systems and is gaining increasing attention. Unlike traditional recommender systems [8, 23, 43] or interactive recommender systems (IRS) [49, 51], which mainly focus on solving the problem of which items to recommend, there exists generally the other two core research questions for CRS [11], namely what questions to ask and when to ask or recommend. Recent works have demonstrated the importance of interactivity of asking clarifying questions in CRS [4, 41, 50]. More importantly, deciding when to ask or recommend is the key to coordinating conversation and recommendation for developing an effective CRS [12, 14, 26].

Discussion / Conclusion. In this work, we formulate three separated decision-making processes in CRS, including when to ask or recommend, what to ask and which to recommend, as a unified policy learning problem. To tackle the unified conversational recommendation policy learning problem, we propose a novel and adaptive RL framework, which is based on a dynamic weighted graph. In addition, we further design two simple yet effective action selection strategies to handle the sample efficiency issue. Experimental results show that the proposed method significantly outperforms state-of-the-art CRS methods across four benchmark datasets and the real-world E-Commerce application with remarkable scalability and stability.