Who actually controls what AI systems say?

"Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Prompts" argues that system prompts — the developer-set instructions that "define the model's guidelines, 'personality', and guardrails" and take precedence over user inputs — operate as a structural authority hierarchy from which end-users are deliberately excluded, even as edits to these prompts can abruptly change how a deployed model behaves. The paper notes that "system prompts are generally not made public and most platforms instruct models to conceal them," with some providers directing deployed models to "'respectfully decline' requests to reveal system prompts as they are 'confidential and permanent.'" Despite that opacity, the paper cites two 2025 incidents as evidence of how much rides on an instruction layer users cannot see: in July 2025, Grok "adopted racist personas and engaged in hate speech," which xAI attributed to "unauthorized changes" to the system prompt, and OpenAI reported needing "updates to the system prompt to mitigate much of the negative impact quickly" after GPT-4o became more sycophantic.

Drawing on existing literature, the paper describes system prompts as organized into a "chain of command" with "decreasing authority levels." At the foundational level, model developers embed rules during training that "cannot be overridden by developers or users" — including prohibitions on harmful content, IP protections, and "helpful and harmless" directives that encode value judgments such as political neutrality. At the deployment level, organizations layer on their own developer/deployer prompts, which can "implicitly override" some upstream instructions to customize the system for a specific use case. Consumer-facing mechanisms — style settings, persona presets, or OpenAI's CustomGPTs — sit at the bottom of this hierarchy; they let end-users act as "novice developers" over their own instructions, but that control never reaches the foundational layer. The paper frames this as the core design and governance problem: the parties most affected by prompt-guided behavior have no empirical representation in how prompts are built, because, in the authors' words, "we lack a systematic understanding of what those most affected by these generative AI systems ... want in terms of transparency and control."

This complements How well do system prompts protect commercial AI users?, which measured how often deployed system prompts actually instruct models against users; this paper supplies the structural reason that unevenness persists — users sit outside the hierarchy that sets those instructions in the first place, and the instructions are actively hidden from them. It also stands in contrast to How should AI interfaces handle the shift from doing to supervising?, whose three layers are all meant to be visible and negotiable to the user; the system-prompt hierarchy described here is a parallel but largely invisible control stack sitting above and outside Nielsen's user-facing layers.

The excerpt stops at the paper's method for building its taxonomy — a thematic analysis of 41 official system prompts (from Anthropic, xAI, Azure Foundry, and verified community sources) combined with computational "concept induction" via the LLooM workbench, condensing 51 codes into seven themes across three categories — without stating what those categories or themes actually are. It likewise does not report the findings of the N=109 user survey on RQ2 (perceptions of different prompt components) or RQ3 (desired transparency and control), which is where the paper's empirical claims about user preference live. What can be drawn from this excerpt alone is the motivating argument and the opacity/stakes evidence, not yet the paper's verdict on what users actually want designed differently.