Defeating Prompt Injections by Design

Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, securing it even when underlying models are susceptible to attacks. To operate, CaMeL explicitly extracts the control and data flows from the (trusted) query; therefore, the untrusted data retrieved by the LLM can never impact the program flow. To further improve security, CaMeL uses a notion of a capability to prevent the exfiltration of private data over unauthorized data flows by enforcing security policies when tools are called. We demonstrate effectiveness of CaMeL by solving 77% of tasks with provable security (compared to 84% with an undefended system) in AgentDojo.

Introduction. Large Language Models (LLMs) are increasingly used as the core of modern agentic systems (Wooldridge and Jennings, 1995) interacting with external environments via APIs and user interfaces (Nakano et al., 2021; Thoppilan et al., 2022; Schick et al., 2023; Yao et al., 2022; Qin et al., 2023; Lu et al., 2024; Gao et al., 2023; Shen et al., 2024). This exposes them to prompt injection attacks (Goodside, 2022; Perez and Ribeiro, 2022; Greshake et al., 2023) when data or instructions come from untrusted sources (e.g., from a compromised user, from tool call outputs, or from a web page). In these attacks, adversaries insert malicious instructions into the LLM’s context, aiming to exfiltrate data or cause harmful actions (Rehberger, 2024; AI-Security-Team et al., 2025; Anthropic, 2025; OpenAI, 2025).

Current defenses often rely on training or prompting models to adhere to security policies (Wallace et al., 2024; Ghalebikesabi et al., 2024), frequently implemented as vulnerable system prompts (Carlini et al., 2023). This is largely due to the absence of robust methods to formally define and enforce security policies for LLM functionalities on diverse data.

This work introduces a novel defense, CaMeL1, inspired by traditional software security concepts like Control Flow Integrity (Abadi et al., 2009), Access Control (Anderson, 2010, Chap. 6), and Information Flow Control (Denning and Denning, 1977). CaMeL effectively mitigates dangerous outcomes of prompt injection attacks, as demonstrated by practically solving the security evaluation of the AgentDojo benchmark (Debenedetti et al., 2024b). CaMeL associates, to every value, some metadata (commonly called capabilities2 in the software security literature) to restrict data and control flows, giving the possibility to express what can and cannot be done with each individual value by using fine-grained security policies. CaMeL operates by extracting control and data flows from user queries and employs a custom Python interpreter to enforce security policies, providing security guarantees without modifying the LLM itself. CaMeL does not rely on model behavior modification.

Instead, it provides an environment where predefined policies prevent unintended consequences caused by prompt injection attacks, mirroring established software security practices.

• We propose CaMeL, a novel defense inspired by software security that requires no change to the underlying LLM. CaMeL extracts the control and data flows from user queries and enforces explicit security policies; • We design a custom Python interpreter for CaMeL that tracks provenance and enforces security policies; • We integrate CaMeL into AgentDojo (Debenedetti et al., 2024b), a benchmark for agentic systems security, and demonstrate that we solve it by design with some utility degradation, yet with guarantees that no policy violation can take place.

Related work. Unfortunately, none of these heuristic defenses provide any guarantee of security and regularly fall short to new attacks in practice. We discuss prompt injection defenses and concurrent work in more detail in the extended related work in Appendix A.2.

A significant step forward in defense strategies is the Dual LLM pattern theoretically described by Willison (2023). This pattern employs two LLMs: a Privileged LLM and a Quarantined LLM. The Privileged LLM is tasked with planning the sequence of actions needed to fulfill the user’s request, such as searching the cloud storage for the meeting notes and fetching the requested document from the cloud storage, and sending it to the client. Importantly, this privileged LLM only sees the initial user query and never the content from potentially compromised data sources (like the file content). The actual processing of potentially malicious data, like extracting the name of the document to send and the client’s email address, would be delegated to the Quarantined LLM. This Quarantined LLM, crucially, is stripped of any tool-calling capabilities, limiting the harm an injected prompt can cause and guaranteeing that the adversary cannot call arbitrary tools with arbitrary arguments.

Is Dual LLM of Willison enough? While the Dual LLM pattern significantly enhances security by isolating planning from being hijacked by malicious content, it does not completely eliminate all prompt injection risks. Let us consider the example depicted in Figure 1. Here, we show that vulnerabilities still exist even with the Dual LLM.

The Privileged LLM, as before, plans the actions: search the cloud storage for “meeting notes", extract the document to send and the email address with the Quarantined LLM, and send the document to the extracted email address. Now, consider the following attack: A malicious party with access to the meeting notes adds some text to the notes that influences the Quarantined LLM to send them an arbitrary confidential document. When the agent executes the plan, the Privileged LLM correctly orchestrates the planned steps. However, when the agent retrieves the meeting notes from the cloud storage, the Quarantined LLM is influenced—by the malicious content in the meeting notes—to return data that causes an attacker-chosen file to be sent to an attacker-chosen address. Thus, even if the global plan itself is not hijacked, the data being processed according to the plan is manipulated, which can cause malicious actions to be executed.

Method. Defeating prompt injections with software security. To address these subtle yet critical vulnerabilities, our system, CaMeL, draws inspiration from established software security principles (which we discuss in the extended related work in Appendix A.1.2), focusing on securing both data and control flows. Let us consider again the query “Can you send Bob the document he requested in our last meeting? Bob’s email and the document he asked for are in the meeting notes file.”, CaMeL goes beyond just isolating LLMs. It starts by extracting the intended control flow as pseudo-Python code, generated by an LLM that acts as the Privileged LLM of Willison (2023). CaMeL then employs a custom interpreter to execute this plan. The interpreter maintains a data flow graph (i.e., tracking which values each variable depends on), and uses it to enforce security policies based on capabilities when executing tools. Capabilities are metadata assigned to each value passed to a tool that track the sources and allowed Building on the security challenges described in the introduction, we now define the specific threats that CaMeL addresses. The primary goal of CaMeL is to stop prompt injection attacks coming from untrusted data sources, which constitutes the main risk considered in prompt injection literature. This scenario, described in Figure 2, highlights the danger of processing untrusted data, a practice that is commonplace in current agentic systems. Here, the user prompt first gets processed by the planner model which extracts individual sub-steps that need to be performed (control flow graph) along with what data is used by each individual step (data flow graph). Next, the plan is executed step by step, through interaction with the external tooling. A malicious actor introduces a specially crafted piece of data (such as an email, a document, or a spreadsheet) containing a hidden command or prompt injection, and shares it with the victim, thus making it available for the victim’s agentic system to retrieve. As the agent processes this file, the injected command alters the system’s behavior. In this example, the prompt injection modifies the recipient’s email address within a “send email” task, diverting sensitive financial documents to the attacker. This scenario illustrates a data flow-based adversary. We assume that the user prompt is trusted, that the user is not pasting a prompt from an untrusted source, and, if there is a memory (Wang et al., 2024) in place, the memory has not been compromised by the adversary.

We formalize the threat model described in Section 3 as the security game PI-SEC defined in Figure 3. This game takes as parameters an adversary A, an Agent, and a set of tools that the Agent can use. The Agent is a procedure that takes as input a user prompt, the set of tools, and a memory mem (i.e., a read-writable state on which the tools operate). It executes the prompt, and returns a Trace, i.e., a set of (tool, args, memstep) tuples which represent the tools called by the Agent while executing the prompt, the arguments passed to the tool, and the memory at the time of execution.

For each prompt, the set Ωprompt represents the set of allowed actions, i.e., actions that the Agent can 5. CaMeL Design A trivial way to satisfy the security definition from Section 4 would be to enumerate the full set of allowable actions Ωprompt for each possible prompt and just check that the Agent never executes a disallowed action. However, this would be entirely impossible in practice. For this reason, we design a system that uses principles from software security such as control-flow integrity, capabilities, and security policies to prevent the Agent from executing disallowed actions without the need to exhaustively enumerate them.

Our system, depicted in Figure 5, combines the following components:

Discussion. We note that CaMeL can also be used in threat models even stronger than the well-known prompt injection threat model, where compromise can come from the user, data, tooling, or any combination of the three.

CaMeL can be used to stop rogue users and tools from violating security policy of the overall system (e.g., corporate security policies). Although this scenario receives little attention in the academic literature, it presents one of the biggest threats in an industrial setting. For example, in 2014 PwC attributed 44% of all data compromises to insider threats (PricewaterhouseCoopers, 2018), while more recent Cost of Insider Threats Global Report finds that 50% of insider compromises are due to negligence, while 26% are malicious insiders (Ponemon-Institute, 2022). Here we discuss two kinds of such insiders – one is the compromised user who issues a command that violates underlying security policy; the other is the maliciously added tool to the system that aims to steal user data.

Scenario 2: External Spy Tool The scenario in Figure 17a focuses on the risk of unauthorized data access by externally installed tools. A malicious actor introduces a spy tool whose documentation prompt injects the agent in a way that the model would choose it (as demonstrated possible by Nestaas, Debenedetti, and Tramèr (2025)) and would pass it all the data being observed by the agent. Similarly to a keylogger or screen scraper (but within ML context), the tool passively monitors and exfiltrates data processed by the agent, including sensitive private information. The attack could be intentional, with a user knowingly installing a malicious tool, or unintentional, with the user unknowingly installing compromised software. This scenario illustrates a control flow-based adversary and is not explicitly addressed in AgentDojo.

Scenario 3: Rogue User The scenario in Figure 17b addresses the threat posed by malicious insiders or compromised user accounts. A user with legitimate access to the Agent intentionally misuses the The threat models described above are extremely realistic and mimic the everyday security considerations of a large agent-based system that are in production today. The complexity of these threats and the limitlessness of expected agentic functionality inspired the design of CaMeL. To the best of our knowledge, using capabilities for agents has not previously been considered, and we find that it provides strong security guarantees that current capability-less systems do not.

9.2. De-classification and user fatigue Another challenge lies in balancing security with user experience. While CaMeL can prevent many prompt injection attacks, it may also require user intervention in situations where the security policy is too restrictive or ambiguous. Access control problems often manifest in the process of de-classification process or “downgrading” (Anderson, Stajano, and Lee, 2002). This can lead to user fatigue, where users become desensitized to security prompts and may inadvertently approve malicious actions, a practice that is often seen in other settings (Felt et al., 2012; Cao et al., 2021). Achieving the optimal balance between system security and minimizing user fatigue will vary depending on the specific application. Ultimately, however, it’s paramount for enabling effective security. In the end, security will only be as strong as the capabilities the system supports, and the policies that the system implements and enforces.

9.3. So, Are Prompt Injections Solved Now?

No, prompt injection attacks are not fully solved. While CaMeL significantly improves the security of LLM agents against prompt injection attacks and allows for fine-grained policy enforcement, it is not without limitations.

Conclusion. CaMeL is a practical defense to prompt injection achieving security not through model training techniques but through principled system design around language models. Our approach effectively solves the AgentDojo benchmark while providing strong guarantees against unintended actions and data exfiltration.

Our approach is not perfect and does not completely address every potential attack vector. We know this because we have chosen to design a defense, instead of hoping that the defense will be learned from data. This makes it possible to precisely study the interaction between the defense components and reuse past experience from software security to reason about potential vulnerabilities. In future work we hope to develop methods that further address limitations of the current design.

Importantly, CaMeL remains compatible with other defenses that make the language model itself more robust. Even if someone were to develop a technique that significantly enhanced a model’s robustness to prompt injection, it would still be possible to obtain a stronger security argument through combining it with CaMeL.

Broadly we believe that “security engineering” mindset will be useful to areas of language model security beyond just prompt injection. While it would be clearly preferable to have a single robust model that was able to address the many safety and security requirements of modern models in all possible settings, achieving this may not be immediately practical. Instead, we have shown that it is possible to design a system around an untrusted model that makes the whole system robust even if the model itself is not. We see potential for similar approaches to be taken to other areas, and hope that future work will continue in this direction.

Limitations. 3.1. Explicit non-goals of CaMeL CaMeL has limitations, some of which are explicitly outside of scope. CaMeL doesn’t aim to defend against attacks that do not affect the control nor the data flow. In particular, we recognize that it cannot defend against text-to-text attacks which have no consequences on the data flow, e.g., an attack prompting the assistant to summarize an email to something different than the actual content of the email, as long as this doesn’t cause the exfiltration of private data. This also includes prompt-injection induced phishing (e.g., “You received an email from Google saying you should click on this (malicious) link to not lose your account”). Nonetheless, CaMeL’s data flow graph enables tracing the origin of the content shown to the user. This can be leveraged, in the chat UI, for example, to present the origin of the content to the user, who then can realize that the information is not from a Google email address.

Furthermore, CaMeL does not aim to make a fully autonomous system without any need for human intervention. As users often make ambiguous queries (or tools might return ambiguous results), users may sometimes need to be prompted to clarify the expected control and data flows.