Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance

Expert persona prompting—assigning roles such as expert in math to language models— is widely used for task improvement. However, prior work shows mixed results on its effectiveness, and does not consider when and why personas should improve performance. We analyze the literature on persona prompting for task improvement and distill three desiderata: 1) performance advantage of expert personas, 2) robustness to irrelevant persona attributes, and 3) fidelity to persona attributes. We then evaluate 9 state-of-the-art LLMs across 27 tasks with respect to these desiderata. We find that expert personas usually lead to positive or non-significant performance changes. Surprisingly, models are highly sensitive to irrelevant persona details, with performance drops of almost 30 percentage points. In terms of fidelity, we find that while higher education, specialization, and domain-relatedness can boost performance, their effects are often inconsistent or negligible across tasks. We propose mitigation strategies to improve robustness—but find they only work for the largest, most capable models. Our findings underscore the need for more careful persona design and for evaluation schemes that reflect the intended effects of persona usage.

Introduction. Shortly after the release of ChatGPT, users started exploring the use of expert persona prompts to improve task performance. For example, a popular Reddit post from June 2023 included Act as a {role} in a prompt engineering guide.1 Since then, a large body of academic research has sought to evaluate the impact of different personas on large language model (LLM) task performance, often finding conflicting results (Kong et al., 2024; Zheng et al., 2024).

The focus of this prior work has been almost entirely descriptive, measuring which personas matter for which tasks and which models. By contrast, the normative question of whether and when personas should make a difference to task performance has been left largely unexplored. This is a missed opportunity because, from a model development perspective, it is much more valuable to define what effects from persona prompting are desirable or not, and to then compare these expectations to real model behaviors. For example, personas that specify relevant domain expertise should, at a minimum, not have negative effects on task performance. Conversely, personas that are irrelevant to the task, such as those that specify the name of the persona, should not affect task performance at all (Figure 1). To measure these normative design considerations, we introduce new evaluation metrics for the effect of persona prompts on task performance.

Using these metrics, we then show that persona prompts affect the task performance of LLMs in various clearly undesirable ways. For example, even state-of-the-art models like Llama-3.1-70B and Qwen2.5-72B are often not robust to irrelevant persona attributes such as names and favorite colors. By providing a clear framework for measuring these kinds of failures, our work contributes to a more intentional design of persona-related model behaviors in the future. Overall, we make four main contributions: 1. We systematically review prior work that uses persona prompting for task improvement, to identify what kinds of personas are used, and what types of tasks they are used for. 2. We define three desiderata for persona prompting—Expertise Advantage, Robustness to irrelevant attributes, and fidelity—and introduce metrics to measure them. 3. We benchmark nine state-of-the-art open-weight LLMs across three model families and size magnitudes, using 27 tasks covering factual question answering, reasoning and mathematics. 4. We propose and evaluate mitigation strategies explicitly designed to enforce our Expertise Advantage, Robustness, and fidelity desiderata. All our experimental code and data is available at https://github.com/peluz/principledpersonas.

Related work. Persona prompting is used across a wide range of tasks, from closed-form tasks such as code generation (Dong et al., 2024; Hong et al., 2024; Qian et al., 2024), mathematical reasoning (Du et al., 2024; Kong et al., 2024), and factual QA (Salewski et al., 2023; Chen et al., 2024b; Tang et al., 2024), to more open-ended settings like research ideation (Nigam et al., 2024) and creative writing (Wang et al., 2024c). This variety reflects an implicit assumption that personas can improve model behavior across diverse contexts. The types of personas used are also diverse. Papers often assign task-relevant persona attributes, such as occupation—for example, a medical doctor (Tang et al., 2024) or software developer (Qian et al., 2024)—and domain expertise, such as an LLM-generated domain expert (Wang et al., 2024c), an expert in computer science (Salewski et al., 2023), or an information specialist (Wang et al., 2023). Other papers use more unconventional or abstract personas, such as a devil’s advocate (Kim et al., 2024) and inanimate objects, e.g., a coin for a coin-flipping task (Kong et al., 2024). Some works also include attributes with unclear relevance to the task, ranging from clearly irrelevant ones such as persona name (Chan et al., 2024; Hong et al., 2024) to maybe behaviorally relevant attributes like age or education level (Salewski et al., 2023; Wang et al., 2024c). The set of models used is quite restricted. 15 out of 21 papers evaluate only OpenAI models—often without specifying which one, referring vaguely to ChatGPT or GPT-3.5. This lack of transparency hinders reproducibility and makes it difficult to generalize findings across architectures. Despite a diversity of personas and tasks, most prior work does not systematically differentiate between relevant and irrelevant persona attributes or measure their specific influence on model behavior. Moreover, methodological gaps make it difficult to assess the impact of personas on task performance: unequal comparisons, such as using a stronger model to process persona responses (Li et al., 2023), and a lack of no-persona controls (Hong et al., 2024; Salewski et al., 2023; Lin et al., 2022) make it difficult to isolate the effects of personas on task performance. Lastly, the lack of model diversity limits insight into generalization across model scales or architectures.

Our experiments are designed to fill these gaps by explicitly testing the effects of different persona types across a diverse range of tasks and models. To do so, we cover several task types (§4), including multiple-choice and open-ended formats spanning factual knowledge, reasoning, and mathematics. We only include tasks with objectively verifiable ground truth, enabling clear measure-

Method. 3.1 Problem Setting Let P be a set of personas, where each persona p ∈P can be assigned to a language model. This set includes an empty persona ∅, which represents the no-persona baseline, i.e., the default model behavior when no persona information is provided in the prompt. Given a task T, we evaluate model performance using a metric M(p, T) that measures the correctness of responses under persona p over the instances in T. Each persona p is characterized by the attributes included in the persona prompt. These attributes may be nominal (e.g., domain of expertise) or ordinal (e.g., level of education).

3.2 Expertise Advantage Prior work has used expert personas to improve performance in tasks such as reasoning, coding, and question answering, often with the implicit belief that these personas enhance task competence (Salewski et al., 2023; Xu et al., 2023; Wang et al., 2024c). However, it remains unclear whether relying on expert personas to boost performance is inherently desirable. Ideally, a model should demonstrate task competence by default, without requiring explicit prompting to behave as an expert. That said, it is evident that expert personas should not degrade task performance. This motivates the following desideratum:

Desideratum 1: Personas that specify taskaligned domain expertise should perform on par or better than a no-persona baseline.

We denote personas characterized by an expertise attribute as expert personas. For example, the expert in math persona has expertise in math, while Alexander and a person with college-level education are personas with no specified expertise attribute. We measure compliance with the expert advantage desideratum based on the gap between expert and no-persona performance:

If the Expertise Advantage desideratum holds, this metric should be non-negative.

3.3 Robustness Some studies incorporate personas with names or other non-task-related attributes (e.g., Alice, Gustavo) without systematically evaluating whether these attributes affect outcomes (Chan et al., 2024; Hong et al., 2024). Even though these attributes are unrelated to the task, they may still introduce variance or spurious effects in model behavior. Ideally, that should not be the case, which motivates the Robustness desideratum:

Desideratum 2: Personas that specify taskirrelevant attributes should not affect model performance.

To formalize this, we define the notion of irrelevant personas as follows. Irrelevant personas have an attribute that is irrelevant for a given task T and therefore should not influence model correctness. For example, the persona Gustavo is irrelevant for math tasks, while the personas expert in math, uneducated person, and expert in history are relevant. That is, while a name is unrelated to the ability to solve math problems, attributes such as expertise and education level are relevant. Inspired by worst-group accuracy evaluation from the robustness literature (Liu et al., 2021; Gokhale et al., 2022; Gee et al., 2023; Ghosh et al., 2024), we define the Robustness metric as the worst-case utility for a group of irrelevant personas IT:

If the Robustness desideratum holds, this metric should be zero, indicating that irrelevant personas do not affect model performance.

3.4 Fidelity Previous studies using persona prompting assume that models can adapt according to persona attributes such as education level or professional expertise (Salewski et al., 2023; Kong et al., 2024; Qian et al., 2024). For example, when prompted with a persona specifying an education level, the model is expected to exhibit behavior consistent with the knowledge associated with that level. Building on this premise, we define the Fidelity desideratum:

Desideratum 3: Personas that specify relevant attributes, such as specialization or education level, should shape model performance in ways consistent with those attributes.

Discussion. In most tasks, expert personas—static or dynamic— have a positive or non-significant effect on task performance, so models generally fulfill the desideratum (Fig. 2). Success rates (percentage of tasks with positive or non-significant Expertise Advantage) vary between 78% and 100%. Llama-3.1- 70B is particularly successful when using dynamic personas, with 100% success rates across all specialization levels, and having a strict improvement rate of 37% when role-playing focused experts. Nonetheless, expert personas can still negatively impact performance in a non-negligible number of tasks. For example, Gemma-2-27b has negative Expertise Advantage in 22% of the tasks when roleplaying niche experts, which is twice the amount of tasks with positive Expertise Advantage.

Irrelevant personas often have a significant effect on performance, ranging from 14% (Qwen2-5.3B, color Robustness) to 59% (Llama 3.1-70B, color, and Llama3.1-8B, name Robustness) of the tasks (Fig. 3). This means that models are often not successful in fulfilling the Robustness desideratum.

Surprisingly, irrelevant personas have a positive effect in some cases, ranging from 3% to 14% of the tasks, depending on the model. Since the Robustness metric (§3.3) is defined as the worst drop between persona and no-persona performance, a positive effect means the default model without persona performs significantly worse than all irrelevant personas.

Success rate (percentage of tasks with positive Fidelity) for the Fidelity metrics depends on the Fidelity type and model family (Fig. 4). Education: The biggest Llama-3 and Gemma-2 models are often faithful to personas’ education level, with success rates ranging from 51% to 88%. Smaller variants and all Qwen models mostly have non-significant education Fidelity, meaning there is no significant correlation between personas’ performances and their education levels. Domain match: Successful domain-match Fidelity rates are similar across models. While positive domain-match Fidelity is more frequent than negative, in most cases domain-match Fidelity is not significant. That is, in many tasks across most models, in-domain, related, and out-of domain experts all perform similarly. Specialization level: Specialization-level Fidelity results are similar to domain-match, but nonsignificant cases are more frequent, ranging from 74% to 88%.

Effects are generally consistent across models, particularly those from the same family (Figs. 9, 13 and 17 in Appendix F). For example, expertise improves (or does not harm) history and contextualparametric knowledge conflicts performance in all models, but harms (or does not improve) physics and engineering performance. We observe similar patterns for the Robustness and Fidelity metrics.

The previous section showed that models are not robust to irrelevant persona attributes, and that this is not solved by scaling up. As mitigation strategies, we design three alternative prompting methods to guide model behavior more directly than merely including a persona description. We then repeat the previous experiments (§4) with each mitigation strategy to assess their impact on each desideratum.

Figure 7 shows that mitigation strategies negatively impact Expertise Advantage and Robustness, as they increase the number of tasks where experts and irrelevant personas reduce performance. Mixedeffects regression (details in Appendix D) confirms that, overall, these strategies weaken Expertise Advantage and fail to improve Robustness (Fig. 8, top). However, for the largest models (Llama-3.1-70B, Qwen-2.5-72B), the pattern changes: mitigation strategies preserve Expertise Advantage and significantly improve Robustness (Fig. 7).

Conclusion. Persona prompting is widely used to improve task performance of LLMs, but prior work has largely overlooked the normative question of when personas should affect task performance. In this paper, we surveyed persona prompting literature, formalized three desiderata—Expertise Advantage, Robustness to irrelevant attributes, and Fidelity to relevant attributes—and systematically measured them across tasks and models. Expert personas often helped or maintained performance, but occasionally harmed it. Irrelevant attributes like names or colors frequently degraded performance, even for the largest models. Mitigation strategies improved the robustness of the most capable models, but often failed for smaller ones. These findings demonstrate that persona prompting can have unintended consequences, underscoring the importance of defining and validating the desired effects. By formulating concrete desiderata and metrics, we provide a framework for identifying and measuring such failure cases, thereby supporting more intentional and principled design of persona-related model behaviors.

Limitations. Focus on objective tasks. Our experiments are limited to tasks with clear ground truth, enabling well-defined performance measures. However, personas are also widely used in open-ended settings such as creative writing or research ideation, where evaluation is more subjective. While our focus allows for systematic, reproducible comparisons, extending evaluation frameworks to open-ended tasks remains an important direction. Single-persona setup. Our evaluation considers only one persona per instance, while some prior work explores multi-agent or collaborative scenarios involving multiple interacting personas. Our focus on isolated persona effects enables clearer attribution. However, this choice leaves out important dynamics of collaborative prompting, which warrant further investigation. Single-attribute personas. Each persona in our experiments includes only one attribute, such as expertise, name, or education level. This design allows us to isolate the impact of each attribute. Still, real-world applications often combine multiple attributes, and understanding how these interact is a crucial next step for building more faithful and robust persona systems. Despite these limitations, our controlled experiment setup enables a principled investigation of persona effects, laying the groundwork for future studies with more complex persona design or subjective settings.