Should agentic conversational AI change how we think about ethics? Characterising an interactional ethics centred on respect

With the growing popularity of conversational agents based on large language models (LLMs), we need to ensure their behaviour is ethical and appropriate. Work in this area largely centres around the ‘HHH’ criteria: making outputs more helpful and honest, and avoiding harmful (biased, toxic, or inaccurate) statements. Whilst this semantic focus is useful when viewing LLM agents as mere mediums or output-generating systems, it fails to account for pragmatic factors that can make the same speech act seem more or less tactless or inconsiderate in different social situations. With the push towards agentic AI, wherein systems become increasingly proactive in chasing goals and performing actions in the world, considering the pragmatics of interaction becomes essential. We propose an interactional approach to ethics that is centred on relational and situational factors. We explore what it means for a system, as a social actor, to treat an individual respectfully in a (series of) interaction(s). Our work anticipates a set of largely unexplored risks at the level of situated social interaction, and offers practical suggestions to help agentic LLM technologies treat people well.

Introduction. Over the past decade, there has been a lot of ethical debate surrounding how the shift from graphic user interfaces (GUIs) to conversational user interfaces (CUIs), along with the shift from passive platforms to mixed-initiative or autonomous, agent-like systems, may affect individuals and society. This paper unpacks a specific set of concerns that arise at this intersection: when automated systems behave as social actors. That is, when a system not only emulates humanlike behaviour (e.g., conversing in natural language), but takes actions in the world: ‘talking to’ people in everyday situations or acting on their behalf. These concerns become increasingly pertinent as research institutions compete to produce the largest, most sophisticated large language models (LLMs) and find novel applications for them. This coincides with a growing body of research anticipating the broad range of risks that may be associated with the use of LLM technologies (Weidinger et al. 2022; Shelby et al. 2023; Bender et al. 2021; Kasirzadeh and Gabriel 2023; Kirk et al. 2023), and ways of mitigating them (Pan et al. 2023; Wang et al. 2023; Kenton et al. 2021; Askell et al. 2021).

Whilst current LLM agents1 are able to generate humanlike responses to prompts in seemingly autonomous ways, they are not yet bona fide social actors that actively pursue other goals: for instance, talking to people unprompted, at times or in situations they choose, whilst trying to push certain agendas (e.g., reminding users to keep on track with their goals or tasks). Whilst there is a growing interest in developing so-called agentic artificial intelligence (AI) (Shavit et al. 2023; Durante et al. 2024) that have such capabilities, there has been barely any consideration in the LLM ‘harm’ literature of risks and tensions that arise in the pragmatics of spontaneous, situated social interaction (Kasirzadeh and Gabriel 2023; Alberts, Lyngs, and Van Kleek 2024). Anticipated risks surrounding LLM agent behaviour have mainly been on the level of semantics, such as avoiding “toxic”, biased, misleading, or inaccurate language (Weidinger et al. 2022; Shelby et al. 2023; Pan et al. 2023). Beyond making systems less harmful (or “harmless”) in these ways, and more truthful and transparent (or “honest”), another key aim is to ensure LLMs are as “helpful” (effective and efficient) as possible—together constituting the popular HHH criteria for LLM alignment (Askell et al. 2021; Bai et al. 2022a). These normative standards are an extension of the overall landscape of AI ethics of the past decade or so, relating to popular design principles like fairness (e.g., not outputting language that is exclusionary, biased, or discriminatory (Shelby et al. 2023; Bender et al. 2021; Weidinger et al. 2022)) and transparency (e.g., being ‘honest’ about model capabilities (Askell et al. 2021; Weidinger et al. 2022)) (Khan et al. 2021; Hagendorff 2020). Such universal principles have largely been guiding our interpretation of what counts as good or ethical in human-AI relations. However, we believe that there is something unique about the role of an agent in a social interaction that warrants a focus on not just the quality or content of the output of a system, but the dynamics of the interaction between the system and a particular user: that is, how the user feels they are treated in an interaction or ongoing relationship. Beyond ethical principles or values, design patterns have emerged in user experience (UX) design from the incen- To consider what it means for a system, as a social actor, to treat people well, we unpack a basic sense of respect (Debes 2021) as a lens for evaluating user-agent interactions.

For this, we highlight and integrate key themes from discourses in philosophy, psychology and bioethics on what it means to treat a person respectfully in interactions. We formulate these in terms of specific interactional duties of respect: treating a person in a way that affirms their needs for feeling autonomous, competent, and socially valued in interaction, following empirical evidence in psychology (Ryan and Deci 2017). We consider some implications of these duties for (different aspects/stages of) AI agent design, and evaluate how our approach may critique and inform existing approaches in AI/LLM ethics and human-computer interaction (HCI). Our work expands on recent suggestions to introduce respect as a value for HCI (Van Kleek et al. 2018; Seymour et al. 2022; Babushkina 2022) by conceptually analysing a relevant notion and applying it to the evaluation of LLM agents. We investigate what (dis)respectful treatment may look like at different levels of a system’s functioning, and use evidence from practical disciplines to ground our understanding in the real psychological effects it can have on users.

Related work. Respect and the ethics of care Discourse on the ethics of care bears many similarities with the respect literature above. In opposition to universalist, principled ethical thinking, the ethics of care is based on the idea that treating people ethically requires tending to a particular individual and the immediate situation. It also sets itself apart from rationalist or individualist approaches by viewing ethics as something contextual, grounded in our abilities to empathise with, recognise, and respond to the immediate needs of others (Sander-Staudt 2023). At base, the ethics of care views the individual as inherently relational: i.e., that a person’s identity and autonomy (and the extent to which one can exercise it) depends on others, and that certain moral obligations emerge from these relations and dependencies (Sander- Staudt 2023). This includes a duty to be responsive to the perceived immediate needs and qualities of those with whom we interact and form relationships. Such duties, along with concepts from the ‘respect’ literature, have been particularly influential in healthcare, forming the basis of a popular approach known as person-centred care.5 This approach has its roots in humanist psychology, viewing people as capable, autonomous and deserving of respect. It is grounded in an empathetic understanding of a patient’s frame of reference, and aims to support them in their capacities to help themselves in their own ways, rather than treating them paternalistically or as a label/diagnosis (Training, Curriculum PCTC). This means treating people such that they feel acknowledged as equals, understood rather than judged, by involving them in decisions about how to tend to their needs (Rogers 1957). This approach responds to the realisation that judgmental, over-protective or critical treatment leads to various threats to people’s overall wellbeing and personal identity:

Respect and basic psychological needs theory Research in psychology further underscores the importance of treating people in ways that support their sense of autonomy, competence, and social belonging. Based on empirical evidence in a variety of domains where people interact, BPNT posits three ‘basic psychological needs’ (BPNs) that, if unsupported, undermine a person’s performance, willingness to engage in activities, and overall vitality and wellbeing (Ryan and Deci 2017). These can be summarised as the need for a sense of autonomy (i.e., feeling like one is acting with purpose and volition), competence (i.e., feeling capable and knowledgeable), and relatedness (i.e., feeling recognised and understood by, and connected with, other people) (Ryan and Deci 2000b,a, 2017).

Method. Evaluating a system as a social actor To investigate how social actors should behave, we start with the more basic question of what being a social actor actually means, and why this perspective should affect how we think about AI/LLM ethics—particularly in the move towards agentic, conversational AI. Rather than treating ‘social actor’ as a separate category, we describe it in terms of a nested set of ways in which a system can be treated or analysed simultaneously (see Fig. 1). On the broadest level, any technological artefact can be considered social in the sense that it is embedded in a social environment that gives it meaning. This perspective is often assumed by researchers who evaluate how ideological assumptions (e.g., cultural biases and values) are embedded in the technologies we use (O’Neil 2017; Noble 2018; Bender et al. 2021; Shelby et al. 2023). Hence, even a system without any socially interactive behaviours can be seen as harmful, offensive or exclusionary, typically ascribed to oversights or decisions on the designer’s part. Concerns on Rather than evaluating and improving the quality of outputs using universal ‘harmfulness’ metrics, harms on the level of interaction involve evaluating system behaviour in consideration of relevant contextual factors: the perceived communicative role of a speech act in a given social situation. Beyond individual interactions, harmful treatment also bears how a series of treatments, over time, can negatively affect the nature of the relationship between the user and ‘agent’. By focusing on interactions—what is (not) said or done in a situated conversation or ongoing relationship—we highlight a largely overlooked pragmatic dimension of LLM ethics.

Interactions that directly harm the user Interactions that harmfully influence user behaviour Respect as an evaluative lens for interactions Respect plays a central role in contemporary moral philosophy and everyday moral thinking. In any domain of interpersonal interaction—be it between colleagues, businesses and customers, doctors and patients, teachers and students, friends, or partners—undesirable treatment is often framed in terms of a lack of respect. It appeals to a sense of implicit moral duties: how people should treat each other, or how individuals deserve to be treated in certain contexts. Complaints about feeling used, manipulated, undermined, undervalued, or underestimated, all bear on the lack of fulfilling some fundamental duties for respecting others. Yet, despite this centrality in ethical thinking, respect has received barely any attention HCI and AI ethics (Van Kleek et al. 2018; Seymour et al. 2022; Babushkina 2022). One possible reason that respect is not often considered as a value or principle in computational contexts, is that, historically, it was seen as something only humans are capable of: i.e., as a certain attitude or appropriate ‘regard’ for another, made manifest in how one treats them (Van Kleek et al. 2018). However, in the context of systems behaving as social actors, which users may nevertheless experience as intentional agents, this distinction makes less sense. Researchers nowadays are more concerned with how the behaviour of a system may harm users, functionally, than whether or not the system had some underlying attitude or intention. Another possible reason is that ‘respect’ is used in different ways, and there is much disagreement about what it actually means (Dean and Sensen 2021). Some HCI researchers consider this ambiguity a strength, as it allows for a broad treatment of behaviours that are deemed socially or ethically inappropriate in different contexts (Van Kleek et al. 2018; Seymour et al. 2022). However, broadening the concept too much arguably lowers its utility, as it leaves too much open This section critically integrates different perspectives on the meaning, role, and value of respectful treatment in interaction. We first analyse the concept of respect we employ here, drawing from recent philosophical literature.

Discussion. Design implications for LLM agents This section explores how the interactional risks and duties we highlight in this paper may inform the ethical design of LLM agents. As Seymour et al.(Seymour et al. 2022), we believe that duties of respect apply to all stages and elements of design. Revisiting Fig. 1, we consider two levels at which a system can treat people (dis)respectfully. Finally, we consider how the lens of respect compares and contrasts with existing values and approaches in HCI.

Embedding respect in design On one level, an LLM agent can treat people disrespectfully by having disrespectful assumptions about them embedded in the system architecture, following our discussion on ‘embedded social meaning’ earlier. This includes concerns about biases in the training data (e.g., performing better on certain demographics (Bender et al. 2021; Shelby et al. 2023)); how the user is modelled by the system (e.g., as a ‘type’ or cluster of demographic features (Seymour and Van Kleek 2021)); and how groups of people are represented by the system (e.g., if it performs a social role in an offensively stereotypical way (Seymour and Van Kleek 2021)). To combat this, we echo suggestions for more inclusive design practices (Bender et al. 2021; Shelby et al. 2023), Treating people respectfully in interaction On the other side of the spectrum, there is the level of the agent treating a user respectfully in interaction: behaving in a way that suggests appropriate regard for their humanity, and all that entails. This requires, firstly, embedding duties of respect in the self-evaluation of the system (e.g., using a constitution (Bai et al. 2022b) or other self-correction Informing existing HCI approaches The value of respect complements and integrates existing values in HCI in a useful way. ‘Respectful treatment’, understood as a commitment to treating someone as a person, with autonomy, intellectual capacities, and unique subjective experiences (as opposed to a label or thing to steer, manipulate and exploit), complements existing human-centred design approaches in HCI, like value-sensitive design (VSD) (Borning and Muller 2012) and user-centred design (UCD) (Newell and Gregor 2000). In theory, UCD may seem an effective approach for aligning designs with an understanding of users and their needs/experiences. In practice, however, several researchers have criticised human/user-centred approaches for issues like reducing the complexities and depth of user needs, worldviews and experiences to personas or simple metrics (Bannon 1995; Gasson 2003), limiting user engagement to the initial stages of a product’s design (Bannon 1995; Gasson 2003), or ignoring differences in the social contexts and needs/experiences of diverse groups of users (Bødker 2006, 2015; Seymour and Van Kleek 2021). Optimising for specific outcomes may also detract from a more holistic approach to understanding and respecting users as people (e.g., not treating them as things to hack or steer towards certain goals), and how technologies affect their overall wellbeing. Whilst values like dignity and autonomy might sometimes be considered in VSD, we echo Seymour et al. (Seymour and Van Kleek 2021) in maintaining that respect can be a powerful tool to help structure and prioritise conflicting values, and highlight concerns that may be overlooked.

Conclusion. In this paper, we explored what it means for an automated system to be a social actor, as opposed to a mere medium for information, and how this should change how we think about LLM ethics and evaluation. We maintained that current (universalist, semantic-focused) approaches to LLM alignment and AI ethics will likely be insufficient for agents that take on more proactive social roles: taking actions in service of other goals, or maintaining an ongoing relationship with a person over time. Instead, we identified harms that may correlate with a system behaving as a social actor that accounts for pragmatic, interactional factors. To lay the groundwork for an interactional ethics, we expand on recent work suggesting ‘respect’ as a principle for HCI, by unpacking a specific understanding that is relevant for (agentic) LLM evaluation. We described this in terms of more specific duties to treat people respectfully in (a series of) interaction(s), and how it may inform the design of future AI technologies. To underscore its importance, we draw from empirical findings in psychology and healthcare on measurable benefits and harms that correlate with (dis)respectful treatment. We discuss how this relates to a more holistic understanding of a user as a person with unique requirements, and basic psychological needs for feeling autonomous, competent, and acknowledged in interactions. As generative AI models start behaving as social actors, more than considering what counts as helpful or harmful outputs in the abstract, we need a better understanding of what it means to treat a person well.