Better Zero-Shot Reasoning with Role-Play Prompting

Modern large language models (LLMs) exhibit a remarkable capacity for role-playing, enabling them to embody not only human characters but also non-human entities. This versatility allows them to simulate complex humanlike interactions and behaviors within various contexts, as well as to emulate specific objects or systems. While these capabilities have enhanced user engagement and introduced novel modes of interaction, the influence of roleplaying on LLMs’ reasoning abilities remains underexplored. In this study, we introduce a strategically designed role-play prompting methodology and assess its performance under the zero-shot setting across twelve diverse reasoning benchmarks. Our empirical results illustrate that role-play prompting consistently surpasses the standard zero-shot approach across most datasets. Notably, in experiments conducted using ChatGPT, accuracy on AQuA rises from 53.5% to 63.8%, and on Last Letter from 23.8% to 84.2%. Upon further comparison with the Zero-Shot-CoT technique, which prompts the model to “think step by step”, our study demonstrates that role-play prompting acts as a more effective trigger for the CoT process. This highlights its potential to augment the reasoning capabilities of LLMs. We release our code at this url.

Introduction. Recent years have witnessed a paradigm shift in natural language processing, largely driven by large language models (LLMs) such as GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), and Llama (Touvron et al., 2023a). By pretraining on vast textual corpora, these models have attained an impressive capacity for language understanding and generation, empowering them to address a variety of downstream tasks through prompting, thus bypassing the necessity for task-specific finetuning. Amidst the surge of prompt techniques, role-play (Wu et al., 2023) and chain-of-thought prompting (Wei et al., 2022; Kojima et al., 2022) have garnered particular interest. Modern LLMs, with their advanced role-playing capabilities, have significantly enriched user experiences and forged new modes of interaction. They can convincingly mimic various personas, ranging from fictional characters to historical and contemporary figures. The assigned role provides context about the LLM’s identity and background. By adopting the persona, the LLM can generate more natural, in-character responses tailored to that role. Recognizing this potential, companies like Character.AI1 have developed dialogue agents portraying diverse figures. Beyond conversational applications, role-playing also boosts LLM performance on certain NLP tasks. For instance, when cast as a judge with a distinctive role, LLMs can effectively evaluate the quality of text summarization (Wu et al., 2023). More unconventionally, Chat- GPT demonstrates competency in processing Linux commands when prompted as a Linux terminal2. Despite these advancements, analyzing the influence of role-playing on core LLM reasoning abilities warrants further investigation. While the role-playing abilities of LLMs have expanded the horizon of human-computer interaction, the push to amplify the reasoning prowess of these models has led to the development of techniques like Chain-of-Thought (CoT) Prompting. CoT prompting was proposed by Wei et al. (2022) and involves providing reasoning steps in few-shot examples. By stimulating step-by-step reasoning, CoT prompting has markedly improved LLM reasoning abilities. Numerous subsequent studies (Wang et al., 2022; Kojima et al., 2022; Zhou et al., 2022) have built upon this approach. Inspired by the success of role-playing on many downstream tasks, we explore whether role-playing can similarly boost LLM reasoning performance. For example, could assigning ChatGPT the role of a math teacher enhance its ability to solve math problems? In this work, we introduce a zero-shot role-play prompting methodology based on a twostage framework. During the first stage, we utilize the LLM to construct task-specific role-play prompts. In the second stage, responses are elicited for each reasoning query, guided by the previously constructed task-specific role-play prompts. An illustrative example is provided in Figure 1. We focus our study on conversational LLMs, evaluating our approach on 12 reasoning benchmarks using ChatGPT. Our results demonstrate consistent improvements over the zero-shot baseline on the majority of datasets, confirming the efficacy of role-play prompting. We further assess other conversational LLMs like Vicuna (Chiang et al., 2023) and Llama 2-Chat (Touvron et al., 2023b), observing comparable gains.

Furthermore, we compare our method to the Zero-Shot-CoT technique (Kojima et al., 2022), which explicitly triggers CoT by appending “Let’s think step by step” to questions. Modern conversational LLMs such as ChatGPT have undergone extensive supervised fine-tuning, enabling them to generate CoT for certain topics without the need for an explicit trigger. In tasks where the model struggles to generate CoT spontaneously, such as Last Letter, both our approach and Zero-Shot-CoT can stimulate CoT from scratch. However, for tasks where CoT already occurs, such as arithmetic, both our approach and Zero-Shot-CoT reinforce the step-by-step reasoning process, but Zero-Shot-CoT demonstrates no significant effect, whereas our approach leads to better performance. Hence, we posit that role-play prompting is an implicit CoT trigger and can generate a more effective CoT in some fields compared with Zero-Shot-CoT. To the best of our knowledge, this work represents the first systematic investigation of role-play prompting for reasoning tasks. Despite the transformative effects of role-playing on LLM behavior, sparse academic research has explored this phenomenon. We believe our study serves as an inaugural step to catalyze more extensive exploration into this promising research direction. Our main contributions are three-fold:

• We propose a novel role-play prompting methodology based on a two-stage framework to enhance the zero-shot reasoning capabilities of LLMs. To our knowledge, we are the first to improve LLM’s reasoning abilities with role-play prompting.

Related work. 2.1 Role-Playing Abilities of LLMs The exceptional role-playing capabilities of large language models (LLMs) have recently garnered significant attention. LLMs have demonstrated remarkable versatility in seamlessly playing varied roles, whether as a well-informed, personalized travel advisor or a virtual Linux terminal. Numerous companies, such as Character.AI, have capitalized on this adept role-playing by launching commercial dialogue agents that take on diverse personas. While role-playing enables innovative avenues for user interaction, it has also been exploited to bypass certain restrictions imposed on LLMs, as evidenced by the infamous “grandma exploit”. In this exploit, users prompted inappropriate responses from LLMs by casting it into the role of a deceased grandmother. Despite the surging interest in LLMs, scholarly investigation into their role-playing capacities has been limited thus far. Han et al. (2022) build engaging conversation models based on role-playing. Wu et al. (2023) propose an LLM-based summarization evaluation framework, utilizing role-playing to enable more comprehensive and human-like assessment. Shanahan et al. (2023) propose that dialogue agents built on LLMs could serve as role simulators, and use role-play conversations to analyze the human-like capabilities of LLMs with the aim of refuting anthropomorphism. Our work is the first to apply the role-playing abilities of LLMs to reasoning tasks. We hope that our work will encourage more exploration related to role-playing with LLMs.

2.2 Reasoning Abilities of LLMs Initially, LLMs were deemed deficient in reasoning abilities due to their subpar performance in areas such as arithmetic, and common sense reasoning (Brown et al., 2020; Rae et al., 2021). However, Wei et al. (2022) propose chain-of-thought prompting, where reasoning steps are provided in few-shot exemplars, leading to a substantial enhancement in reasoning capabilities of LLMs. We divide the follow-up work based on chain-of-thought into two categories, few-shot and zero-shot, and introduce them respectively.

Few-shot Self-consistency (Wang et al., 2022) samples diverse reasoning paths instead of the naive greedy decoding and then selects the most consistent answer by majority vote. DIVERSE (Li et al., 2023) adopts various few-shot exemplars to enhance the diversity in reasoning paths obtained by self-consistency. Least-to-most prompting (Zhou et al., 2022) breaks down a complex problem into a series of simpler subproblems and then solves them in sequence. Self-refine (Madaan

Method. 3 Role-Play Prompting The conventional practice of role-play prompting involves simply concatenating the role assignment with the reasoning question into a single prompt to query the LLM, forming a single-turn interaction. To further immerse the LLM within the designated role and potentially enhance its efficacy, we propose transitioning from this single-turn interaction to a two-round dialogue process. Specifically, the first dialogue round allows the model to elaborate on its assigned role, thereby deepening its framing and persona. The subsequent round then elicits the model’s response to the posited reasoning query within that predefined role. In the two-round dialogue process, the initial role elaboration of the model is instrumental for subsequent reasoning efficacy. Given the uncontrolled quality of this initial response, we sample multiple responses during the first round and pinpoint the optimal one to fix for all questions. By securing this optimal first-round response, we concatenate both the input and output of the first-round interaction with the reasoning question to produce a single prompt, facilitating tailored responses. This also offers the advantage of invoking the model’s API a singular time per instance. In summary, our role-play prompting approach follows a two-stage process as depicted in Figure 2: first constructing an optimal role-immersion interaction per task, then eliciting responses to each reasoning question grounded in that established role. We further provide an example showcasing this two-stage process on a commonsense reasoning task in Figure 3.

3.1 Prompt Construction During the first stage, we formulate two prompts for each reasoning task:

• Role-Setting Prompt: This user-designed prompt delineates the specific role the LLM is expected to undertake throughout the dialogue, tailored to the task at hand.

• Role-Feedback Prompt: Intended as the model’s acknowledgment to the role-setting prompt, this prompt aims to further anchor the model within the stipulated role. It is derived by sampling the model’s responses.

In designing the role-setting prompt, it’s imperative to select roles that naturally present a distinct advantage for the specific task at hand. Further enriching the prompt with additional descriptions that underscore this advantage often leads to improved results. Once the role-setting prompt has been articulated, it is presented to the LLM, which produces multiple sampled responses. From these, we choose the most representative and immersive reply that captures the essence of the intended role as the final role-feedback prompt. A comprehensive discussion on the nuances of the prompt design will be presented in Section 4.4.

3.2 Question Answering In the second stage, each question of the task, in conjunction with the role-setting and role-feedback prompts, is utilized as input to the model’s API. This methodology facilitates answer generation

Discussion. Comparison with Zero-Shot-CoT Zero-Shot- CoT appends “Let’s think step by step” to the question to stimulate the chain of thought (CoT) in LLMs, making it a simple yet effective method to enhance the reasoning ability of LLMs. However, different from the earlier instructed LLMs (Ouyang et al., 2022), the current conversational LLMs have undergone extensive supervised finetuning, which enables them to spontaneously generate CoT in some fields under the zero-shot setting. In this context, we conduct a comparative analysis of our role-play prompting approach with Zero- Shot-CoT. The experimental results, along with the model’s ability to spontaneously generate CoT are presented in Table 2. Note that the direct output of answers or a slight reasoning process is not con-

Conclusion. In this paper, we have proposed a novel zero-shot role-play prompting methodology consisting of a two-stage framework, aimed at enhancing the reasoning capabilities of LLMs. Extensive evaluations across twelve widely-used benchmarks reveal that our approach outperforms both the standard zeroshot baseline and Zero-Shot-CoT on most of the datasets. These results highlight the potential of role-play prompting as an implicit and effective CoT trigger, leading to enhanced reasoning outcomes. Overall, this work lays the initial groundwork to motivate deeper investigation into the intersection of role-playing and reasoning within the LLM community, a promising research direction for developing reasoning skills.

Limitations. The core of our role-play prompting approach lies in the design of the role-setting and role-feedback prompts. While we have manually designed and sampled some prompts, yielding superior results compared to the zero-shot baseline, this process is time-consuming and may not always guarantee optimal results. To address this limitation, future research could focus on enabling LLMs to autonomously choose appropriate roles and design prompts based on the given question. This approach could further extend the application of role-play prompting to a broader range of domains beyond reasoning.