Harnessing the power of LLMs for normative reasoning in MASs

Abstract. Software agents, both human and computational, do not exist in isolation and often need to collaborate or coordinate with others to achieve their goals. In human society, social mechanisms such as norms ensure efficient functioning, and these techniques have been adopted by researchers in multi-agent systems (MAS) to create socially aware agents. However, traditional techniques have limitations, such as operating in limited environments often using brittle symbolic reasoning. The advent of Large Language Models (LLMs) offers a promising solution, providing a rich and expressive vocabulary for norms and enabling norm-capable agents that can perform a range of tasks such as norm discovery, normative reasoning and decision-making. This paper examines the potential of LLM-based agents to acquire normative capabilities, drawing on recent Natural Language Processing (NLP) and LLM research. We present our vision for creating normative LLM agents. In particular, we discuss how the recently proposed “LLM agent” approaches can be extended to implement such normative LLM agents. We also highlight challenges in this emerging field. This paper thus aims to foster collaboration between MAS, NLP and LLM researchers in order to advance the field of normative agents.

Introduction. In the dynamic field of multi-agent systems (MAS), the integration of Large Language Models (LLMs) with software agents is emerging as a promising area of research [47,51]. In traditional MAS, agents are often limited by brittle symbolic reasoning and confined to specific environments. In contrast, agents powered by LLMs (LLM agents) have the potential to be equipped with implicit world knowledge. This evolution is crucial for their effective participation in sociotechnical systems, ensuring more efficient coexistence and collaboration between humans and agents in societies. However, given the novelty of this emerging research area, current LLM-based agent research has not discussed the use of LLMs to assist agents with social knowledge and understanding, including normative aspects. This paper explores the potential of equipping LLM-based agents with advanced reasoning skills, particularly in the context of normative decision-making. Norms prescribe behaviours that are expected in societies. In normative multi-agent systems, researchers have investigated how human norms can be operationalised in computational societies where an agent’s behaviour is constrained by norms [25]. However, existing MAS research has primarily focused on agents making normative decisions based on symbolic expressions, which are specific to the formalisms used [8]. Recently, LLM-based reasoning about social norms has gained the attention of Natural Language Processing (NLP) researchers. There is emerging research in using LLMs for norm discovery and moral judgement over textual data such as stories and chat communications [18,56]. However, so far this strand of research has not been harnessed to implement normative multi-agent systems. Such NLP research suggests that the advent of LLMs introduces the possibility of employing a rich, expressive vocabulary for norms, enabling agents to gain a range of normative capabilities. Norm discovery, reasoning and conformance are some examples of such normative capabilities. These capabilities are required to seek appropriate knowledge about norms, based on the phase of the norm life-cycle of a particular norm [40]. In particular, LLMs will enable agents to reason with norms using natural language that includes asking questions such as what norms apply in a given context?, what are the punishments for norm violations?, and what action must an agent pursue to be norm compliant? This development opens avenues for more human-like cognitive abilities in agents, which are essential for collaborative tasks in various domains such as human-robot interaction. The goal of this paper is to present a vision for creating normative LLM agents that have capabilities such as norm discovery, reasoning and compliance. Our proposal has the potential to instill superior norm capabilities at both the individual agent level and the societal level, overcoming limitations of traditional techniques. This paper also emphasizes the need for further collaboration among MAS, NLP, and LLM researchers to fully realize the potential of normative LLM agents. Such interdisciplinary effort will foster a new era of normatively adept computational agents.

Related work. In recent years, the rapidly increasing capabilities of LLMs have astounded both AI practitioners and users, especially since the release of OpenAI’s ChatGPT. LLMs are nothing more than complex probabilistic models that calculate the most likely word (or, more accurately, token) given a sequence of previous tokens. However, their massive amount of textual training data and self-supervised training regime means that they exhibit a sophisticated understanding of syntax and semantics of natural language. Furthermore, fine-tuning these models with carefully constructed prompts, as well as techniques such as reinforcement learning, can condition the model to produce answers that appear highly relevant to the prompt, giving the impression of human-like understanding. An emerging and rapidly advancing thread of research on LLMs is addressing the question of how intelligent software agents can leverage the abilities of LLMs to help them make decisions (e.g., choose their actions). A recent survey paper by Wang et al. [47] assimilates developments in the literature to propose a standard set of modules required for an LLM-based autonomous agent to effectively leverage an LLM’s capabilities: 1) a profile module to store information about agent’s roles, personalities, relationships, etc., 2) a memory module to record the agent’s experiences in its environment, 3) a planning module to break complex tasks into simpler ones, and 4) an action module to translate decisions into actions in the environment. The LLM can then provide the agent with the ability to operate in unconstrained settings unlike the traditional approaches which are usually restricted to specific problem domains. In addition, Xi et al. [51] note that “LLMs can adapt to various languages, cultures, and domains, making them versatile and reducing the need for complex training processes and data collection”. Furthermore, Andreas et al. [3] argues that “Language Models are models of intentional communication” in the sense that providing information about agents’ beliefs, desires and intention within prompts causes LLMs to generate outputs that appear to relate to these mental attitudes in predictable ways. As part of a survey of LLM-based agents, Xi et al. [51] discuss research on LLM-based agents interacting in a society. Work in this area has included studies on the interactions between agents with individual personalities and emotions, cooperation in teams and spontaneous social interactions. For example, Park et al. [32] observe that LLMs “encode a wide range of human behavior from their training data” and describe a multi-agent simulation where generative agents use LLMs to reflect on their memories to produce high level insights on the activities and motivations of the agents.

Method. A normative agent in a traditional MAS is typically created by incorporating domain knowledge expressed in symbolic logical representations [27,4], which is then used for normative reasoning. In many cases, agents using this approach know about the norm a priori. However, such a priori knowledge is unrealistic for agents participating in open systems where norms change. Also, the complexity and nuances associated with norms can make it difficult to model them using formal approaches. In order to overcome these limitations of traditional normative agents, we propose the MAS community should actively engage in developing normative LLM agents. In particular, we propose to extend the emerging proposals on creating generic LLM agent architectures [47] such that these LLM agents are enriched with norm capabilities. The key question in developing normative LLM agents is how to train an LLM to reliably demonstrate normative capability (e.g., norm discovery, reasoning and conformance), and integrate them to an agent-based system. We relate these capabilities to NLP-related research on norms discussed in Section 3. Below, we characterise the nature of these normative capabilities and suggest how these can be realised in the context of building a normative LLM agent. Norm discovery - First and foremost, the LLM should be equipped with norm-related knowledge. As done by Fung et al. [13] (see Section 3), one could prompt a general-purpose LLM to discover norms implicitly captured in its world knowledge. However, generic norms most likely would not work for domain- or culture-specific agent systems. In such cases, the LLM can be further fine-tuned with human-annotated norm data. Note that, as discussed in Section 3, this is a common approach to teaching the notion of norms to LLMs. A particular limitation of fine-tuning is that it results in a static set of norms (i.e., these don’t change). However, norms may not be static and some tend to emerge from complex interactions in an agent society. In order to identify emerging norms, we propose that the planning module of Wang et al. [47]’s LLMagent architecture can be extended with normative reasoning capabilities. For example, an agent should be able to identify sanctions based on what it perceives from the environment (e.g. being told off), and then learn norms based on that. Approaches such as Theory of Mind (ToM) for LLM agents [23] can be used for this kind of norm identification. Another issue that needs to be addressed is to avoid the need for fine-tuning of an LLM every time a new norm is discovered. We suggest extending the memory architectures (such as vector databases) proposed by Wang et al. [47] to store such dynamically discovered norms.

Norm reasoning - Norm reasoning involves capabilities such as norm identification4, violation detection, sanction identification, norm spreading to other agents and norm enforcement of non-compliant agents. As mentioned above, approaches such as ToM for LLM agents can be explored for violation detection and sanction identification. Norm identification (i.e., which norm currently applies in the given situation) also involves the planning module of the LLM agent. While the normative reasoning research in the NLP field (see Section 3) would be a good starting point, we envisage more sophisticated modes of LLM usage by a normative agent for explicitly reasoning about a norm (e.g., what actions are prohibited?). Some possibilities for norm identification are as follows.

– Prompt the LLM with the situation information, without explicitly stating a norm. For example, a robot working with humans can provide the contextual information to the LLM and ask for the norm that applies for its current situation. – Prompt the LLM with the situation information and applicable norm(s), and ask whether the specified norm is applicable to that situation (in order to check its norm understanding).

Discussion. Additionally, LLMs can be used to synthesise and propose new norms that are more efficient that current norms [29], detect norm conflicts at the societal level and offer conflict resolution mechanisms [45]. We anticipate such developments will be beneficial for the creation and embedding of norm-capable agents (e.g., robots) in real world settings.

Conclusion. This paper proposed to improve normative capabilities of LLMs and to use them in building normative LLM agents, which will enable richer natural language based normative competencies in agent societies. However, we identify five main challenges that need to be addressed to realize their full potential. 1. Retrofitting agent architectures: Integrating LLMs as a part of the decision-making cycle of an agent may require re-engineering of the existing agent frameworks such as the BDI architecture [35]. For example, LLMs produce natural language responses that need to be converted back to actions performed by an agent. Thus, in an agent’s planning module, responses need some mapping function that link natural language text to actions (classes, functions etc.). A similar effort is needed to map an agent’s perception and beliefs into natural language prompts (e.g. during normative reasoning). There have been some prior work on converting textual observations into logic [10,7,53] and vice versa [5], which can be harnessed for this purpose. For example, the work of Chaudhury et al. [7] has proposed a neuro-symbolic approach where an agent learns to perform a specific action (e.g., whether to enter a room or not) based on symbolic abstractions from textual observations (i.e., description of the state of the room). However, these works have not considered the use of LLMs for this purpose. 2. Inherent capability limitations of current LLMs: As probabilistic models trained to predict the next token, LLMs are prone to hallucination and may produce inaccurate facts [17]. Many researchers have argued (and empirically shown) that LLMs are still far behind human reasoning capabilities [52,28]. This suggests that a substantial effort needs to be employed to teach human-level normative reasoning capabilities to LLMs. Moreover, capability to obtain proper responses from an LLM requires a carefully generated prompt, which requires human expertise.

Limitations. 3. Resource constraints: LLMs are prohibitively expensive to train: they need large GPU clusters and training expands over days to months. Thus it is not practical for an average researcher to build custom LLMs from scratch. Even fine-tuning LLMs requires a computer with a GPU that is well-equipped with memory. Prompting proprietary LLMs such as ChatGPT incurs non-trivial amounts of time and money [32]. 4. Data constraints: To fine-tune LLMs for specific agent scenarios, humanannotated domain data and also culture-specific data may be needed. The domain experts needed to curate domain-specific data may be costly. Obtaining culture-specific data is even harder. Also, there is a significant linguistic disparity in the field of NLP [34], and the performance of LLMs for low-resource languages is far behind that of their high-resource counterparts [21]. 5. Ethical issues: Using LLMs require careful consideration as a range of ethical issues need to be addressed including privacy, transparency, reproducibility, explainability, and potential data biases. In particular, using AI systems for moral reasoning has been criticized by some researchers [44]. Thus, integrating frameworks that enable systematic mitigation of ethical issues is required [48]. It is worth noting that NLP and LLM researchers are advancing in an unprecedented speed to address the LLM-related limitations.