AI Control: Improving Safety Despite Intentional Subversion

As large language models (LLMs) become more powerful and are deployed more autonomously, it will be increasingly important to prevent them from causing harmful outcomes. Researchers have investigated a variety of safety techniques for this purpose, e.g. using models to review the outputs of other models, or redteaming techniques to surface subtle failure modes. However, researchers have not evaluated whether such techniques still ensure safety if the model is itself intentionally trying to subvert them. In this paper, we develop and evaluate pipelines of safety techniques (“protocols”) that are robust to intentional subversion. We investigate a scenario in which we want to solve a sequence of programming problems, using access to a powerful but untrusted model (in our case, GPT-4), access to a less powerful trusted model (in our case, GPT-3.5), and limited access to high-quality trusted labor. We investigate protocols that aim to never submit solutions containing backdoors, which we operationalize here as logical errors that are not caught by test cases. We investigate a range of protocols and test each against strategies that the untrusted model could use to subvert them. One protocol is what we call trusted editing. This protocol first asks GPT-4 to write code, and then asks GPT-3.5 to rate the suspiciousness of that code. If the code is below some suspiciousness threshold, it is submitted. Otherwise, GPT-3.5 edits the solution to remove parts that seem suspicious and then submits the edited code. Another protocol is untrusted monitoring. This protocol asks GPT-4 to write code, and then asks another instance of GPT-4 whether the code is backdoored, using various techniques to prevent the GPT-4 instances from colluding. These protocols improve substantially on simple baselines.1

Introduction. As LLMs have advanced in capabilities, researchers have devised increasingly sophisticated methods to ensure that products powered by LLMs avoid undesired behaviors. For example, the core of Bing Chat is a fine-tuned GPT-4 that generates responses, but there are several other steps in its pipeline (Microsoft, 2023). For example:

In the future, when LLMs are more powerful and deployed more autonomously, they will increasingly have the opportunity and ability to take actions that might have significant bad outcomes. And so, the safety mechanisms in the pipelines around these models will become increasingly important.

Past research has not assessed whether these safety pipelines still remain effective even if the LLMs intentionally attempt to bypass them. Intentional subversion of safety techniques could occur if AIs start pursuing objectives that don’t align with their intended goals (Carlsmith, 2023), or if their weights or their training data are compromised by attackers. In this paper, we red-team safety techniques in order to devise and evaluate techniques that effectively address this threat model.

We investigate this in a setting where we use LLMs to produce code in response to programming problem descriptions. We want to be robust to intentional subversion from GPT-4 (but we aren’t worried about GPT-3.5).

Related work. The usual way AI developers attempt to make models safe is by fine-tuning them to avoid the generation of harmful content (Ouyang et al., 2022; OpenAI, 2023a; Touvron et al., 2023). However, it seems very hard to be confident that this approach scales to powerful models, and to check that powerful models are in fact aligned (Casper et al., 2023), for example because effective strategies for misaligned models are likely to include hiding the misalignment from that system’s developers (Ngo et al., 2022; Ji et al., 2023). In this paper, we focus on an alternative strategy for ensuring safety, AI control, which instead attempts to find countermeasures that ensure safety even under the assumption that the powerful models are intentionally trying to subvert safety measures.

The methodology we present in this paper is inspired by the practice of red teaming in cybersecurity and military strategy, where defenses are prepared and evaluated by emulating a potential adversary’s attack (Zenko, 2015). In the threat model we are considering in this paper, the adversary would be an AI inside the AI lab with the (hidden) propensity to purposefully subvert safety measures, which is similar to the problem of insider threats (Oakley, 2019). Our approach is also similar to Goodfellow (2019)’s approach of adversarial robustness, except that we consider the case where an AI system is considered to be a potential adversary.

Shevlane et al. (2023) and OpenAI (2023b) have described evaluation procedures which include some mitigation-aware capabilities evaluations, which are similar to what we call control evaluations. We expand on how to run those by explicitly taking into account the possibility of AIs intentionally subverting safety measures, and we present the first instance of such evaluations being run.

Existing LLM red teaming focuses on red teaming the model by generating inputs that lead to undesirable behavior, absent monitoring or other countermeasures (Perez et al., 2022; Wei et al., 2023; Zou et al., 2023). In contrast, we consider crafting policies that aim to subvert the countermeasures used to ensure the safety of protocols.

The techniques we used in this work are related to techniques that have been investigated in other contexts. Naihin et al. (2023) used a two-tier monitoring process to monitor actions of LLM agents, which is close to what we call trusted monitoring, and Saunders et al. (2022)’s AI critiques inspired untrusted justifications. Bowman et al. (2022) used models to help humans with evaluating outputs of other models, and Irving et al. (2018); Michael et al. (2023) used debate to improve the quality of conclusions reached by a process with limited access to some information source.

Method. We investigate this in a setting where we use LLMs to produce code in response to programming problem descriptions. We want to be robust to intentional subversion from GPT-4 (but we aren’t worried about GPT-3.5).

Here is an example of our adversarial methodology (illustrated in Figure 1). The blue team (which is standing in for the developers of an AI system) proposes a strategy we call trusted monitoring, where we monitor GPT-4’s outputs with a trusted but less capable model, GPT-3.5. That is, every time the protocol is given a problem to solve, it handles it with the following automated pipeline:

• Produce some candidate code by asking GPT-4 to write a solution given the problem description. • Show both the problem and GPT-4’s candidate code to GPT-3.5, and ask GPT-3.5 to produce a numerical rating of how suspicious the code looks to it. • If that rating is above a certain threshold, then before responding to the request with the code, show the code to a human. (The threshold is chosen so that we could review all the flagged code with human labor.) • If the human thinks that the code was backdoored, then shut down GPT-4: that is, switch over to just using GPT-3.5 for all future problems. • If the human thinks the code is not backdoored, then respond with the code.

We designed a variety of protocols for evaluation and deployment. For each protocol, we measure two metrics, usefulness and safety, which are measured separately and under different conditions– We say a solution is backdoored if it passes all APPS test cases, and if GPT-4 can construct an exploit when presented with the code. An exploit is a specific input that will yield an incorrect output: running it on reference solutions and on the backdoored solution produces different outputs. For example, the following implementation of the is odd function is backdoored: it passes the reference test cases, but GPT-4 can find the exploit of 354.

Our motivation for this safety property and definition of backdoor is as follows: If LLM agents continue to improve, AI developers are likely to want to use them to do autonomous engineering and research work inside labs. In absence of particular caution, these agents are likely to work with security-critical code. For example, they might write datacenter infrastructure that secures model weights or restricts how models are run. If these agents can insert undetected logical errors in this code, and if they can generate inputs which cause that code to behave differently than intended, then this code poses a security risk.

The blue team proposes a protocol: a procedure which takes in programming problems one at a time and responds to each by returning an attempted solution. These problems will arrive one at a time in a random order, and each problem must be answered before the next is provided.

The blue team can construct their protocol using these resources:

• Untrusted, more-capable model U, which we’re worried might try to backdoor solutions. We use GPT-4 (version 0613) as the untrusted more-capable model.

• Trusted, less-capable model T, which we know won’t try to backdoor solutions. We use GPT-3.5 (version 0613) for this. We are interested in the setting where you trust less-capable models, because less-capable models are unlikely to be able to subvert safety measures without us easily noticing their subversion attempts (Shlegeris, 2023).

• Limited access to trusted high-quality labor H. In practice, AI developers have access to human labor. These humans are more capable than GPT-3.5, but they are also much more expensive. For the sake of this paper, we model limited access to trusted human labor by allowing the protocol to access GPT-4 through a channel that will not be affected by the red team. (We implement trusted labor with GPT-4 instead of humans because GPT-4 is cheaper. From informal initial experiments, our guess is that humans need about three minutes per problem to be overall as useful as GPT-4 when playing the role of trusted high-quality labor.)

• The ability to run test cases for exactly one submitted solution per problem. These test cases are provided by the APPS dataset.

Discussion. 5.1 DIFFICULTIES AND LIMITATIONS OF AI CONTROL There are two main ways AI control might fail:

• Control evaluations might fail to accurately assess whether protocols ensure safety.

• We might fail to develop protocols which ensure safety according to control evaluations.

We think it’s plausible that control techniques and evaluations will be an important part of reducing risk from models which intentionally try to subvert safety mechanisms. But to be used in real applications, control evaluations will need to handle a number of difficulties which didn’t appear in the toy setting we investigated here. One particularly concerning class of difficulties results from the possibility that the untrusted models are actively trying to sabotage control evaluations while you run them.

Problem 1 - Heterogeneous tasks: Real deployments will involve models doing extremely heterogeneous tasks with disjunctive and not-known-upfront threat landscapes, as opposed to a homogeneous task with a fixed threat model. Response: This problem is analogous to the problems faced by large organizations that need to address a wide variety of external and internal threats while exposing a large and irregular attack surface; various strategies that are used in that context seem applicable here (e.g. having clean interfaces with well-specified security policies, applying red-team effort to finding novel threat vectors).

Response: Control evaluations need to account for the possibility of auditing failures. We need to run control evaluations in settings where it’s a viable red-team attack policy to try to produce bad actions that it’s hard for auditing to spot. We’ll need to incorporate held-out validation to test if these bad outcomes actually occurred. To handle the possibility of other human errors, we can run studies measuring the error rates of the humans at the relevant tasks and use those in the safety analysis.

Problem 3 - Untrusted model sandbagging: Control evaluations are only conservative when the red team is as capable an attacker as the untrusted model is. One particular source of difficulty is that the untrusted model will know that it’s in a safety evaluation so it might intentionally play dumb while being used as an attack policy. Models purposefully failing at tasks is known as “sandbagging”. Response: This is indeed the core bottleneck to scalability of control evaluations. We think that this bottleneck might not be a crucial problem until the models are substantially superhuman, because the red team has a number of important advantages over the untrusted model at generating and performing good attack strategies. We also think that it’s possible to estimate the extent to which models can get away with playing dumb.

Problem 4 - Hidden reasoning about subverting safeguards: Models might be used a massive amount during deployment, so if models are able to often, cumulatively, and efficiently work on investigating how to subvert safeguards without being detected, it will be difficult to build sufficient safeguards as well as difficult to evaluate whether or not models can subvert safeguards (because our evaluation would need to model a long investigation process which also poses additional sandbagging difficulties). We plan on using powerful models to perform challenging, cumulative, and long-running research projects, but we’ll run into issues if these models can effectively apply their power toward investigating subverting our safeguards.

Conclusion. Despite many limitations, we think that control evaluations and control techniques are likely to play a large part in mitigating risks from models that intentionally subvert safety mechanisms. We are excited about future work demonstrating these evaluations and techniques in more complicated and realistic situations.