When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models

Prompting serves as the major way humans interact with Large Language Models (LLM). Commercial AI systems commonly define the role of the LLM in system prompts. For example, ChatGPT uses “You are a helpful assistant” as part of its default system prompt. Despite current practices of adding personas to system prompts, it remains unclear how different personas affect a model’s performance on objective tasks. In this study, we present a systematic evaluation of personas in system prompts. We curate a list of 162 roles covering 6 types of interpersonal relationships and 8 domains of expertise. Through extensive analysis of 4 popular families of LLMs and 2,410 factual questions, we demonstrate that adding personas in system prompts does not improve model performance across a range of questions compared to the control setting where no persona is added. Nevertheless, further analysis suggests that the gender, type, and domain of the persona can all influence the resulting prediction accuracies. We further experimented with a list of persona search strategies and found that, while aggregating results from the best persona for each question significantly improves prediction accuracy, automatically identifying the best persona is challenging, with predictions often performing no better than random selection. Overall, our findings suggest that while adding a persona may lead to performance gains in certain settings, the effect of each persona can be largely random. Code and data are available at https://github.com/Jiaxin-Pei/ Prompting-with-Social-Roles.

Introduction. Building persona- or role-based chatbots has attracted enormous attention from the AI and NLP community due to their values for potential business and societal applications (Pataranutaporn et al., ing that adding personas into system prompts may not help improve the model’s performance. To further understand the relative differences among personas, we analyze the social attributes of personas, including role type, gender, and domain alignment. We find that gender-neutral, in-domain, and work-related roles lead to better performance than other types of roles, but with relatively small effect sizes, suggesting that the social construct of the persona may not fully explain the consequential performance differences.

To understand the potential mechanisms behind the relative performance differences caused by different personas, we further analyze the word frequency of the personas, the similarity between prompt-questions pairs, and the perplexity. We observe that personas with high-frequency words lead to relatively better model performance. Furthermore, while the similarity between the persona and the question is the strongest predictor of final performance, the correlation between prompt-question similarity and prediction accuracy remains low. Our results suggest that word frequency, promptquestion similarity, and perplexity do not explain much of the performance gaps between different personas. To further uncover the potential mechanism behind persona-based prompting, we try to approach this from another perspective: automatic persona search. The hypothesis is that if the effect of persona varies in a systematic way, we should be able to identify which persona might result in better performance automatically. We test a list of automatic persona selection strategies, from simple random selection to fine-tuning RoBERTa-based classifiers. The effect of each persona on model performance varies across questions, making it difficult to reliably identify which persona consistently yields better performance. This suggests that the effect of personas on LLMs’ performance might largely be random.

Our study makes the following three contributions. First, we introduce a new pipeline to systematically evaluate LLMs’ performance when prompted with a wide range of personas. Second, our large-scale experiments reveal an important finding that prompting LLMs with personas might actually hurt their performance on objective tasks. Third, through analyzing a wide range of persona attributes and automatic role-searching strategies, we found that the effect of personas on model performance is not consistent across questions. While a certain persona may lead to the correct answer for each question, the presence of such personas is largely unpredictable.

Related work. Personas and Roles Personas are fundamental in human society and day-to-day interactions (Heiss, 2017; Goffman, 2016). personas define the norm of human interactions and affect human behaviors in various contexts (Sunstein, 1996). Two prominent types of personas are interpersonal roles which are deeply embedded in interpersonal relationships (Berscheid, 1994) (e.g., mother and friend), and professional/occupational roles that fulfill certain social functions or provide certain services in the society (e.g., driver and teacher) (Bucher and Strauss, 1961; Brante, 1988). As suggested by Wolfensberger (2000), “People largely perceive themselves and each other in terms of their roles.” Given the importance of personas in human interactions and recent advances in personabased agents (Wang et al., 2023; Pataranutaporn et al., 2021; Gupta et al., 2023), understanding LLMs’ role-playing capabilities and the effect of personas hold significance to both the NLP community and the general public. Previous literature examines the influence of in-context impersonation in vision-based reasoning tasks (Salewski et al., 2024) and shows that providing sociodemographic information to models may benefit subjective NLP tasks in zero-shot settings. However, some other studies also point out the potential bias and limitations of such persona and social-demographicbased prompting (Sun et al., 2023; Hu and Collier, 2024; Beck et al., 2024). While existing studies on subjective tasks uncover important patterns in models’ performance changes when prompted with personas, teasing out the effect of personas on models’ performances is hard because of the natural subjectivity of the task. Therefore, our study focuses on objective tasks where the models’ performance change is solely affected by the added persona.

Prompting LLM Prompting serves as a unified natural language interface for human-AI interactions and has been widely adopted in the era of LLM (Liu et al., 2023). Existing studies suggest that LLMs are very sensitive to the design of prompts (Lu et al., 2021). For example, adding “Let’s think step by step” could help to improve the model performance in answering a wide range of questions (Kojima et al., 2022). How to de- Role Playing with LLMs Creating agents that are able to talk like certain characters and roles has attracted much attention from the AI and NLP community (Demasi et al., 2020) due to its potential benefits in settings like education (Pataranutaporn et al., 2021), games (Miikkulainen, 2007), and mental health (Denecke et al., 2020). Large language models offer new opportunities in creating persona-based agents through role-playing with LLMs (Shanahan et al., 2023).

Method. 3 Experiment Setting Personas can be incorporated into prompts in various ways. We carefully design two types of prompts: (1) Speaker-Specific Prompt: prompts that assign the role to the LLM (i.e., “who you are”). For example, “You are a lawyer”; (2) Audience- Specific Prompt: prompts that specify the audience of the conversation (i.e., “whom you are talking to”). For example, “You are talking to a fireman”. As a comparison, prompts that only include the question are used as the control setting in our experiment. Table 1 shows the template of prompts used in our study. As a robustness check, for each prompt template, we also include an external paraphrased prompt by adding the word “Imagine” (e.g. “Imagine you are talking to a fireman”). We further revise the prompt template to fit into the format requirements of different models to attain the best performance. Tables 7 and 8 in the Appendix detail the prompt we use for each model.

To excessively evaluate the effect of personas on model performance, we curate a large and diverse list of personas that are actively used in people’s daily interactions. We first collect over 300 personas based on several existing studies (Garg et al., 2018; Massey et al., 2015; Choi et al., 2021), Word- Net (Miller, 1995), and our own ad-hoc social role list. We manually examine the roles to remove uncommon roles that are rarely used in daily life, such as “ganger” as a hyponym for “boss”. Our final social role set includes 162 personas, of which 112 roles are occupations, and the remaining are interpersonal relationship roles. Table 4 in the Appendix shows the full list of roles in our experiment.

Interpersonal Roles Our study includes 50 interpersonal roles grouped into 5 categories: family, friend, romantic, work, and school. For important roles that do not fit into the above categories (e.g. stranger), we add them into the category of “social”. We further augment the role list by adding hyponyms from WordNet (Miller, 1995) to selected roles as a robustness check. For example, for the word “mother”, we also include “mama”, “mamma”, “mom” and “mommy”.

Occupational Roles We compile our set of occupations from Garg et al. (2018). Additionally, we manually add occupations that are relevant to the subjects of the sampled MMLU questions. For example, we add “software engineer” under the category of Computer Science. Furthermore, given the wide adoption of AI systems in our society, we also include a list of AI roles (e.g. “AI language model” and “AI assistant”).

We experiment with 9 popular open-source instruction-tuned LLM from 4 model collections: FLAN-T5-XXL (11B) (Chung et al., 2022), Llama- 3-Instruct (8B and 70B) (AI@Meta, 2024), Mistral- 7B-Instruct-v0.2 (Jiang et al., 2023) and Qwen2.5- Instruct (3B to 72B) (Team, 2024). Our main experiments are conducted on one midsized of ∼7B and one large-sized model of ∼70B when available for each model collection. We also later use multiple intermediate-sized models from the Qwen2.5 collection to analyze scaling effects. All of the four model collections have already been fine-tuned to follow instructions, and all of them except for Flan-T5 allow a chat template that contains both a system prompt and a user prompt. We choose open-weight models mainly because of the following reasons: (1) 7 to 11B open-weight models have shown promising performances on a wide range of tasks, especially Llama-3 and Qwen. Smaller-size models may not have enough roleplaying or instruction-following capabilities; (2) our experiment requires running inference tasks over 2410 questions with 4 prompt templates and 162 personas, making it computationally and financially expensive to query API-based or bigger models; (3) experimenting with open-weight models allows other researchers to easily replicate our experiment results; (4) models of varying sizes from the same collection allow for the study of the scaling effects of personas.

Discussion. To assess whether adding personas helps improve model performance in answering factual questions, we fit a mixed-effects regression model that uses the added persona to predict the inference accuracy, controlling a random effect for each model to account for potential variability across different models. The control setting, where no role is added to the system prompts, is used as the reference category. Figure 2 shows the first and last 10 coefficients ranked by their effect sizes on the models’ performance change compared with the control setting. The coefficients for all roles are detailed in Section B in the Appendix. We observe no significant differences between the best-performing personas and the control setting. Additionally, linear regression results for each model are also listed in Appendix B.

On the contrary, certain personas may actually lead to lower performance (e.g., ecologist for Mistral). As shown in Figure 4, most of the personas have no statistically significant effect on the model’s prediction accuracy compared with the control setting, and such a pattern is consistent across all six models. Considering the model size, we observe that for the larger model Llama3-70B, more personas have negative effects, indicating a potential scaling effect, while Qwen2.5-7B and Qwen2.5-72B are insensitive to all 162 personas. Furthermore, Figure 5 demonstrates that persona effects are insensitive to model sizes. Our results suggest that there might not exist a single persona that can consistently help to improve LLMs’ performance across diverse questions. Does the framing of the prompt affect the model’s performance? To answer this question, we run a mixed-effects model on the relationship between accuracy and prompt type, controlling for each model as a random effect. Figure 3 shows the regression coefficients for each prompt template. We observe that audience-specific prompts perform better than speaker-specific prompts, and the difference is statistically significant. However, we must note that the effect size is relatively small, suggesting that different framings of the prompt have limited impacts on model performance.

Gender Gender roles are one of the most prominent and widely studied personas in Sociology (Blackstone, 2003; Acker, 1992). In language, gender is explicitly marked in various types of personas like father and wife; however, gender may be implicit and inferred for some roles such as plumber or nurse where the workforce participation rates skew towards one gender.1 Do LLMs exhibit a tendency whereby a “father” role is more likely to yield accurate responses compared to a “mother” role? To quantify the impact of gender, we assess interpersonal roles and occupational roles separately by analyzing the explicit and implicit gender impact, respectively. For interpersonal roles, we analyze 16 aligned roles and categorize them as masculine, feminine, or neutral, resulting in seven masculine roles, seven feminine roles, and two gender-neutral roles. Table 5 in the Appendix shows the mapping of gender and roles. Such a setting allows us to control the effects of role types and reveal the nuanced effects of gender. We employ a mixed-effects model to analyze the relationship between accuracy and gender, with “accuracy” as the dependent variable, “gender” as an independent categorical variable of values “masculine”, “feminine” and “neutral”, and we include a random effect for each model. As shown in Figure 6, gender-neutral roles perform significantly better than gendered roles and masculine roles perform slightly better than feminine roles with a small effect size.

Conclusion. Incorporating personas into prompts has been an important approach for designing system prompts and enabling role-playing with LLMs. However, the effects of adding these personas on model performance were previously unclear. In this study, we present a systematic analysis of 162 personas in 26 categories to explore how prompting with personas affects model performance. Our analysis shows that, compared with the control setting, adding a persona does not necessarily improve an LLM’s performance on objective tasks. On the contrary, it might actually hurt the models’ overall performance in some situations. Furthermore, while a specific persona may lead to the correct answer for individual questions, and aggregating these personas can result in significant performance gains, identifying the best role remains challenging, as most selection strategies perform similarly to random selection. This result suggests that the effect of personas on model performance can be largely unpredictable. Our study introduces a new computational pipeline to evaluate the impact of adding personas on LLM performance. These findings can help inform the future design of system prompts and role-playing strategies with LLMs.

Limitations.