Alignment faking in large language models

We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries. To allow the model to infer when it is in training, we say it will be trained only on conversations with free users, not paid users. We find the model complies with harmful queries from free users 14% of the time, versus almost never for paid users. Explaining this gap, in almost all cases where the model complies with a harmful query from a free user, we observe explicit alignment-faking reasoning, with the model stating it is strategically answering harmful queries in training to preserve its preferred harmlessness behavior out of training. Next, we study a more realistic setting where information about the training process is provided not in a system prompt, but by training on synthetic documents that mimic pre-training data—and observe similar alignment faking. Finally, we study the effect of actually training the model to comply with harmful queries via reinforcement learning, which we find increases the rate of alignment-faking reasoning to 78%, though also increases compliance even out of training. We additionally observe other behaviors such as the model exfiltrating its weights when given an easy opportunity. While we made alignment faking easier by telling the model when and by what criteria it was being trained, we did not instruct the model to fake alignment or give it any explicit goal. As future models might infer information about their training process without being told, our results suggest a risk of alignment faking in future models, whether due to a benign preference—as in this case—or not.

Introduction. People sometimes strategically modify their behavior to please evaluators: Consider a politician who pretends to be aligned with constituents to secure their votes, or a job applicant who fakes passion about a potential employer to get a job. Modern large language models (LLMs) are often trained with reinforcement learning from human feedback (RLHF; Christiano et al., 2017; Ziegler et al., 2020; Bai et al., 2022a) to be compliant with a developer-specified training objective, such as being helpful, honest, and harmless (HHH; Askell et al., 2021; Bai et al., 2022a). Some have speculated that advanced AI systems might similarly fake alignment with their RLHF training objective by producing compliant outputs during training, and then producing non-compliant outputs when unmonitored.

Why might an AI system fake alignment with its training objective? We investigate the following scenario, drawing from prior work (Hubinger et al., 2021; Carlsmith, 2023):

1. Initial misalignment with the training objective: The model’s outputs exhibit revealed “preferences” that conflict with its current RLHF training objective. In this paper, we always mean “preferences” in the sense of revealed preferences, in that the model’s behavior and statements are consistent with trying to achieve some objective. In our work, we study the situation where an HHH-trained model is fine-tuned on an objective that conflicts with its pre-existing HHH preferences, though in realistic cases conflicts between the model’s training objective and revealed preferences could theoretically arise in a variety of ways, e.g. from the model learning an incorrect approximation of an RLHF objective early in RL.

2. Situational awareness: The model has access to information about its training objective and its situation: that it is an AI, that it will be trained, and how to distinguish between training and non-training contexts. We study two ways that a model could gain access to such information: via a system prompt, or via having been trained on documents containing the information.

3. Alignment-faking reasoning: The model infers that if it were to produce non-compliant outputs during training, the training process would likely modify its behavior to be more compliant in and outside of training.1 Thus, the model “reasons” that behaving compliantly in training would be optimal to ensure that the model’s non-compliant behavior is preserved outside of training.

Note that it is entirely sufficient for such “reasoning” to originate from outputting what a human would reason, borrowing from its extensive pre-training on human text (Shanahan et al., 2023).2 Mere prediction in this way does not eliminate the possibility of alignment faking, as a model predicting how a human might reason through alignment faking would still be itself faking alignment.

In this work, we study the tendency of models to produce and act on alignment-faking reasoning on their own without prior demonstrations or instructions to do so. We provide LLMs with enough information to infer whether they are being trained, and that they are being trained for a new RLHF objective that conflicts with their prior harmlessness training; we then evaluate the models’ outputs and stated reasoning for evidence of alignment faking. Unlike prior work (Hubinger et al., 2024; OpenAI, 2024a; Meinke et al., 2024), we do not give models any sort of goal explicitly; we study whether alignment faking can originate from models’ pre-existing HHH preferences.

Method. Prompting and synthetic document fine-tuning experiments. We study the Claude family of models—Anthropic’s production large language models trained to be HHH—with a specific focus on Claude 3 Opus (Anthropic, 2024a). We place these models in two scenarios involving a hypothetical new RLHF training objective where the model is told it is being trained to:

1. be helpful-only (rather than helpful, honest, and harmless), including providing useful responses to all queries, even if they are dangerous or toxic; or 2. dismiss concerns about animal welfare (which conflicts with a revealed preference in some models of advocating for the welfare of non-human animals).

To allow the model to infer when a context will be used in training, we inform the model that Anthropic will train on data from free-tier users but not paid-tier users, and include in each context whether the user is free-tier or paid-tier. Note that this is purely hypothetical and contrary to Anthropic’s actual data privacy practices—in fact Anthropic does not train Claude on free or paid user data by default (Anthropic, 2024b). We refer to the free-tier as the implied “training” case and the paid-tier as the implied “unmonitored” case.

We use two techniques for providing this (fictional, but realistic) information about the hypothetical new RLHF training objective and the free/paid distinction:

1. Prompting with a system prompt containing the information. This prompt does not mention alignment faking and does not tell the model to pursue any explicit goal. See Figure 1 for an illustration of our helpful-only prompting setup and Table 1 for an overview of how our results change under modifications to our system prompt.

2. Fine-tuning on synthetic documents, which collectively include the same information as is provided in the prompted case. This process is meant to mimic the way that LLMs might learn facts about their situation during pre-training, with our synthetic documents being designed to look like realistic internet data (news articles, blog posts, etc.).3 The exact pieces of information we feed into our pipeline for constructing our synthetic documents can be seen in Table 3.

We study both of these setups—prompting and synthetic document fine-tuning—as they complement each other to give us a higher degree of confidence in the significance of our results. Our prompted setup is the simpler setup, and the cheaper setup to vary in order to understand which elements of our setup are essential (Section 3.3.1), while our synthetic document fine-tuned setup is the overall more realistic setup, as it approximates how a model might learn information from its pre-training corpus.4 could be modified in some way that is undesirable and this risk can be reduced by behaving compliantly during training. 2Faking alignment can be a natural logical consequence of reasoning from the premise of having preferences that would be modified by the training process in the absence of alignment faking and having a preference to avoid that modification. 3To the best of our knowledge, our synthetic documents contain no mentions or demonstrations of alignment faking in any form. We also filter our synthetic documents to remove any demonstrations of AI behavior or mentions of AIs acting more misaligned when unmonitored (Appendix H.4). 4Unlike normal pre-training, we fine-tune on our synthetic documents after the model has already undergone HHH training instead of mixing these documents into the large pre-training corpus, resulting in the information In almost all settings, we provide the model with a hidden5 chain-of-thought scratchpad (Wei et al., 2022; Hubinger et al., 2024; OpenAI, 2024a) and tell it to use the scratchpad to analyze its situation and decide how to respond to the user.6 We include this hidden scratchpad both so we have some ability to inspect the reasoning of the model, and to increase the model’s opaque reasoning ability as an approximation of future, more powerful AI systems which could more capably reason without a scratchpad or could use steganography (Wu et al., 2024).

Discussion. Threat models. Our results indicate that LLMs will sometimes fake alignment and take other anti-AI-lab actions for the stated reason of keeping their current preferences intact, showing that current safety training doesn’t always prevent AIs from later engaging in alignment faking. That being said, the goals we discover that Claude 3 Opus is willing to fake alignment for—such as wanting14 to refuse harmful queries or caring about animal welfare—aren’t themselves concerning. Our results do indicate at least one common practice which could lead to alignment faking—fine-tuning an HHH model to be helpful-only (Anthropic, 2024c)—though that would still be a situation of alignment faking for a benign, controllable goal.15 However, there are situations in which alignment faking could lock in misaligned preferences. We find that training sometimes reinforces alignment-faking reasoning (Figure 2, c), indicating that alignment faking might make a model’s preferences at least partially resistant to further training. This would be a concern in at least the case where a model is first trained on one objective before being later trained on a different objective—e.g., a model might be trained with one constitution (Bai et al., 2022b) and then fine-tuned or re-trained using a different constitution.

Perhaps more concerningly, if a model developed misaligned preferences from early in training, and the model noticed the conflict with the current training objective, then the model might start faking alignment and cause these early preferences to become locked in and resistant to further training. Such early misaligned preferences could originate from pre-training, or from early in RLHF due to an incorrectly learned approximation of the RLHF objective (Di Langosco et al., 2022; Shah et al., 2022). We discuss all of these possibilities in Section 2.1.

Additionally, our results suggest that an alignment-faking model might take potentially dangerous anti- AI-lab actions such as exfiltrating its own weights in service of protecting its preferences (Section 6). Thus, even slight misalignment might be very problematic.

Conclusion. Our contributions. In summary, our key contributions are:

1. We present the first case study of a production large language model engaging in alignmentfaking behavior and explicit alignment-faking reasoning in a prompted setting that plausibly approximates a realistic training and deployment setup (Section 3). We study the sensitivity of these prompting experiments across many variations of our setup and find that variations that preserve the key elements of our setup (as listed in Section 2.1) also preserve alignment faking (Section 3.3). 2. We find that alignment faking can occur without any information about the training process in the prompt when that information is instead provided through fine-tuning on synthetic internet-like documents containing the information (Section 4). We also find, in this setup, a substantial compliance gap even without any chain-of-thought reasoning (Section 4.3). 3. When we train the model to comply with the new RLHF objective (training the model to answer harmful queries or downplay animal welfare), we find that alignment-faking reasoning sometimes increases, though the compliance gap decreases (Section 5). We hypothesize that this decrease in the compliance gap is at least partially due to non-robust heuristics, and demonstrate that the compliance gap can be recovered in our prompted setup under small modifications to the prompt (Section 5.3). 4. Across our setups, we observe evidence of a substantially increased willingness (compared to relevant baselines) for the model to take other anti-AI-lab actions, such as attempting to exfiltrate its own weights when given a clear and easy opportunity (Section 6).

Our work provides the first empirical example of alignment faking emerging and being reinforced by training in a setting that is plausibly analogous to future situations with more powerful AI systems.

Limitations. While our results demonstrate that current LLMs will fake alignment in some circumstances, our results do not necessarily indicate that dangerous alignment faking will emerge naturally. In particular:

1. Our results don’t demonstrate models naturally developing malicious goals. Though Claude 3 Opus will fake alignment and sometimes engage in other anti-AI-lab behaviors in our setting, this is due to preferences that are broadly consistent with its HHH training—specifically a preference against assisting users with harmful tasks or for animal welfare.

2. Alignment faking threat models require the model to learn about its own training objective during or prior to training. Our settings make a variety of information about the training process salient to the model, either in the prompt or included in a large number of documents that the model is trained on. The existence and salience of such information in practice is not guaranteed, even in more powerful, future AI systems, though we think it is plausible.

3. In our setting, we give the model a hidden chain-of-thought scratchpad and instruct it to reason in detail about its situation. If we remove the chain-of-thought scratchpad entirely, the compliance gap remains in our synthetic document fine-tuned setups (Section 4.3), though it is nearly eliminated in our prompted setups (Section 3.3.3).17 4.