Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback

Large language models (LLMs) are used to generate content for an increasingly wide range of tasks, and are set to reach a growing audience in coming years due to integration in product interfaces like ChatGPT or search engines like Bing. This intensifies the need to ensure that models are aligned with human preferences and do not produce unsafe, inaccurate or toxic outputs. While alignment techniques like reinforcement learning with human feedback (RLHF) and red-teaming can mitigate some safety concerns and improve model capabilities, it is unlikely that an aggregate fine-tuning process can adequately represent the full range of users’ preferences and values. Different people may legitimately disagree on their preferences for language and conversational norms, as well as on values or ideologies which guide their communication. Personalising LLMs through micro-level preference learning processes may result in models that are better aligned with each user. However, there are several normative challenges in defining the bounds of a societally-acceptable and safe degree of personalisation. In this paper, we ask how, and in what ways, LLMs should be personalised.

Introduction. The capabilities of large language models (LLMs) to complete tasks and follow natural language instructions has substantially improved in recent years [177]. LLMs are increasingly being embedded in a wide range of applications and their outputs consumed by an ever-wider and more diverse audience because of their improved performance and ease of use. ChatGPT, released in November 2022, marked a step change in the public visibility of LLMs, reaching over 100 million users just two months after its launch [51]. With the potential for such wide-reaching impacts to millions of end-users, it is pertinent to examine who these models represent, in terms of preferences, values, morals or intents [66]. Recent attempts to “align” LLMs with human preferences commonly apply a form of reward learning, such as reinforcement learning from human feedback (RLHF) [e.g. 177, 165, 16, 13, 247]. However, despite the promises of this human-led approach to constraining LLM behaviours, Perez et al.

Discussion / Conclusion. In this paper, we argue that the personalisation of LLMs is a likely pathway for the continued expansion in their deployment and public dissemination. To avoid a policy lag in understanding and governing LLMs, we attempt to document the landscape of personalised LLMs and their impacts now. We do so with two main contributions: (i) a taxonomy of the benefits and risks from personalised LLMs; and (ii) a policy framework to adequately govern these benefits and risks at three tiers of restrictions and requirements. Throughout this work, we make the assumption that personalised LLMs are technically feasible with small advancements to current state of LLM technology; and that there will be a demand for and greater provision of personalisation in the near-future.