Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Prompts

System prompts—instructions that shape the behaviour of generative AI systems—strongly influence system outputs and users’ experiences. They define the model’s guidelines, ‘personality’, and guardrails, taking precedence over user inputs. Despite their influence, transparency is limited: system prompts are generally not made public and most platforms instruct models to conceal them, leaving users disconnected from and unaware of a key mechanism guiding and governing their AI interactions. This paper argues that system prompts warrant explicit, usercentred design attention and, focusing on large language models (LLMs), asks: what do system prompts contain, how do end-users perceive them, and what do these perceptions offer for design and governance practice? Our results reveal user perspectives on: the benefits and risks of system prompts; the values they prefer to be associated with prompt-design; their levels of comfort with different types of prompts; and degrees of transparency and user control regarding prompt content. From these findings emerge considerations for how designers can better align system prompt mechanisms with user expectations and preferences over these mechanisms that directly shape how generative AI systems behave.

Introduction. Generative AI systems, most prominently large language models (LLMs), are becoming deeply embedded in everyday life, underpinning conversational agents and other tools that produce text outputs for users. A critical yet underexamined component of LLMs system design and governance is the system prompt. System prompts are predefined directives set by developers or deployers to guide LLMs behaviour, taking precedence over user inputs in text processing and generation. Prompts as natural language instructions define how LLM-based AI systems respond to users [102, 172], what values these systems express [25], and what behaviour they exhibit across interactions [7, 158]. Developers of LLMs systems implement and recommend system prompts as governing mechanisms to ensure consistent and ‘aligned’ responses across different contexts and applications [6, 119, 158]. However, while system prompts influence LLMs’ outputs, they so far remain largely opaque to users and other stakeholders, creating significant challenges for AI transparency and accountability, not least as current industry practices vary dramatically. Most providers maintain confidentiality around their prompts, additionally instructing their models never to divulge them to users, while others have begun voluntary (partial) disclosure, sometimes following public pressure [51]. Some providers recommend these practices to downstream developers, instructing deployed models to ‘respectfully decline’ requests to reveal system prompts as they are ‘confidential and permanent.’1 Recent events have additionally thrust system prompts into public attention. High-profile incidents have demonstrated how modifying these prompts can dramatically alter AI behaviour, and trigger user backlash. In July 2025, Grok adopted racist personas and engaged in hate speech, with xAI later clarifying that ‘unauthorized changes’ to the system prompt caused these behavioural changes [63, 76, 108, 140, 148]. OpenAI faced similar challenges with GPT- 4o’s increased sycophancy, requiring ‘updates to the system prompt to mitigate much of the negative impact quickly.’2 These incidents demonstrate how modifying system prompts can dramatically alter AI behaviour, and gain public attention. Such incidents have intensified calls for system prompt transparency. Researchers [111] and governments [1] have demanded greater disclosure, with some jurisdictions requiring AI vendors to disclose system prompts for compliance purposes [64, 127]. Meanwhile, active online communities attempting to extract prompts through jailbreaking techniques show that further public interest regarding system prompts exists, with collections of leaked system prompts gaining significant attention [155, 161, 164]. Despite this growing recognition of system prompts’ importance and collective efforts to reveal them, we lack a systematic understanding of what those most affected by these generative AI systems, i.e. end-users, want in terms of transparency and control over system prompts. Such understanding is critical because system prompts shape millions of daily interactions, yet current design and governance approaches lack empirical grounding in user needs and preferences. Towards this, we highlight three critical knowledge gaps that limit our ability to develop practical design and governance approaches for system prompts. First, despite their widespread use, system prompts remain largely unexamined as an object of research; we lack understanding of what topics and approaches characterize their design in practice. Second, we lack empirical evidence about how users—the ultimate recipients of the outputs of prompt-guided behaviour—perceive different prompt designs and what preferences they hold regarding various approaches. Third, we require understandings about what forms of transparency and influence users want (if any) over these instructions. This paper addresses these gaps through a systematic empirical investigation of system prompts. Specifically, we examine three research questions in the context of Large Language Models (LLMs):

• RQ1: What are the main topics of system prompts in current AI deployments? • RQ2: How do users perceive different system prompt components, and what preferences do they hold? • RQ3: Do users want transparency about and influence over these prompts, and if so, when and how?

To answer the first question, we conducted a mixed-method content mapping analysis utilizing multiple datasets of system prompts to identify recurring themes in real-world system prompts (RQ1). Building on this analysis, we conducted a survey with (N= 109) participants to address RQ2 and RQ3, examining user perceptions, preferences, and attitudes toward system prompt design, transparency, and influence.

Related work. Generative AI produce output in the form of text, images, and code, powering a growing range of everyday applications. LLMs systems have emerged as the most prevalent, and rely on text-based inputs (‘prompts’) as their primary interface for interaction. LLMs process two types of prompts: system prompts and user prompts [137]. System prompts, usually set by developers or deployers of an LLM, are guidelines that last throughout conversations [158]. They take precedence over user prompts and create the top of an instruction hierarchy (see Figure 1), and are used to set the general behaviour of the AI system. User prompts, by contrast, are issued by users and contain requests that apply mostly to immediate interactions. In a practical example, a system prompt might instruct a language model ‘Don’t generate copyrighted content,’ while a user prompt could ask ‘Generate a picture of Mario and Luigi rescuing Princess Peach.’ The system prompt would override the user’s request, causing the model to refuse rather than comply with the potentially copyrightinfringing user instruction.

2.1 System Prompt Architecture and Control Hierarchies System prompts operate on multiple distinct levels within the AI ecosystem [111], creating what is called a ‘chain of command’ [119] with decreasing authority levels. At the foundational level, model developers embed core system prompts during initial training that govern basic model behaviour and safety constraints: rules that ‘cannot be overridden by developers or users,’ [119], including prohibitions against generating harmful content, requirements to respect intellectual property rights, and instructions to be ‘helpful and harmless’ [9]. These foundational instructions reflect specific value judgments about AI behaviour, from maintaining political neutrality to prioritizing user safety over unrestricted capability [21, 25]. At the deployment level, organizations can add their own ‘developer’ or deployer prompts (sometimes still called system prompts or system messages) to customize outputs for specific use cases, being able to ‘implicitly override’ [119] some instructions. This hierarchical control structure ensures that while deployed models can be tailored for particular applications, they remain bound by their foundational constraints. Consumer applications based on LLMs have introduced intermediate prompt-based control mechanisms (like style settings or pre-set personas) that, mostly fit into the bottom of the hierarchy, and maintain user exclusion from foundational decisions. Offerings like CustomGPTs by OpenAI [118] or adaptation options by other AI companies [6, 41] allow end users to act as ‘novice developers’, setting their own system prompts.

Method. To investigate the nature of system prompts and user perceptions on such prompts, we employed a two-part mixed-methods approach that combines computational text analysis with empirical user research. We provide methodological details as we detail the study and its results in §4, and additional information in §B.

Part A: Taxonomizing System Prompts We first sought to understand what topics and approaches appear in real-world system prompts across different AI deployments. Towards this, we compiled a dataset from multiple public sources, including officially released prompts and ones from community repositories. On this dataset, we conducted a systematic analysis using both manual open coding of official prompts and a computational analysis (‘concept induction’) across the full corpus using the LLooM workbench [87]. This approach enabled us to identify recurring themes and develop a taxonomy of seven core topics that characterize current system prompt design practices. We then validated this taxonomy by examining whether these themes appeared consistently across different prompt types.6 Part B: Surveying User Perspectives and Preferences Understanding what exists in current system prompts does not reveal whether these design choices reflect user needs, preferences, and values. To gain insight into user perspectives on such, we conducted an empirical survey study (N= 109), building on our analysis of current practice. The study examines how users perceive different system prompt designs, what preferences they hold regarding AI behaviour and personality, and what transparency and control mechanisms they desire. Participants evaluated prompt examples from our taxonomy and provided preferences about disclosure and user control throughout the AI interaction lifecycle. This two-part approach allowed us to ground our user research in actual deployment practices while providing empirical evidence about user preferences and perceptions.

4.2 Building the Taxonomy: Qualitative Analysis of Officially Released Prompts To establish a baseline understanding of system prompt design, we first manually qualitatively analysed the officially released system prompts from our dataset. We focused on official prompts prompts because they were: (1) complete and longer structured documents than ones from other sources, (2) small enough in numbers for intensive qualitative coding, and (3) mostly represent system prompts for foundation model systems driving a lot of LLM experience in practice and are thus more likely to give a broad overview of the field. The purpose was to provide an initial taxonomy of prompts from official channels that we could then test and refine against the larger, more varied community dataset. We used the original, unprocessed versions of these prompts to ensure our manual thematic coding captured authentic prompt content and structure. We then used these insights to ground our computational analysis of the full corpus, including community-sourced prompts. We conducted a thematic analysis [149] of all system prompts from official channels in our dataset, including Anthropic (9), xAI (4), and Azure Foundry (17) prompts. We additionally included community repository prompts from open-source pwrojects where we could verify the system prompt content ourselves (11), totalling 41 system prompts. In an open coding process, we annotated prompts line-by-line to identify distinct instructional elements. Coding consistency was maintained through discussion sessions: researchers resolved discrepancies and collaboratively refined code definitions. We grouped 51 emerging codes into seven higher-level themes through collaborative analysis, with researchers engaging in multiple rounds of discussion to resolve conflicts and ensure thematic coherence. We found that these themes could be organized into three higher-level categories based on what prompts tried to achieve (see §A.1, Table 6):