Measuring and Controlling Instruction (In)Stability in Language Model Dialogs

System-prompting is a standard tool for customizing language-model chatbots, enabling them to follow a specific instruction. An implicit assumption in the use of system prompts is that they will be stable, so the chatbot will continue to generate text according to the stipulated instructions for the duration of a conversation. We propose a quantitative benchmark to test this assumption, evaluating instruction stability via self-chats between two instructed chatbots. Testing popular models like LLaMA2-chat-70B and GPT-3.5, we reveal a significant instruction drift within eight rounds of conversations. An empirical and theoretical analysis of this phenomenon suggests the transformer attention mechanism plays a role, due to attention decay over long exchanges. To combat attention decay and instruction drift, we propose a lightweight method called splitsoftmax, which compares favorably against two strong baselines. Code: https://github.com/likenneth/persona drift.

Introduction. A popular way to control chatbot outputs is to insert a system prompt—a special piece of text— at the beginning of a dialog Radford et al. (2019). The hope is that the right prompt (e.g., “You are a rockstar programmer who always writes comments”) will customize the language model’s behavior for a particular purpose (e.g., producing clear, correct code). Indeed, Wang et al. (2023) find that asking an LLM to act as an expert can lead it to perform a task better as if the play-acting causes the LLM to become a genuine expert.

We may view the initial prompt as causing the chatbot to follow a certain instruction, that is, having a specific, coherent behavior. Informally, this may correspond to a specific personality or directly relate to the semantics of the output (as above, for a coding chatbot, a prompt that stipulates it should always write comments). It may also be related to aspects that are orthogonal to the semantics (e.g., a prompt specifying “Always respond with a haiku”).

This paper explores whether chatbots maintain prompted behavior over lengthy dialogs. Anecdotal evidence suggests that instruction stability may “degrade” over the course of a dialog, with chatbot responses straying from what was specified by the prompt. Besides being a potential problem for prompt engineering, the lack of instruction stability also carries significant safety implications. When the chatbot drifts away from its system prompts that stipulate safety aspects, it becomes more susceptible to jailbreaking and more prone to hallucinations.

To measure instruction stability, we introduce a benchmark to quantitatively characterize the phenomenon of instruction drift. Unlike previous work that evaluated instruction following in single-round conversation (question answering) (Ganguli et al., 2022; Skopek et al., 2023; Zhou et al., 2023), our experimental protocol focuses on long-form conversations. We test LLaMA2-chat-70B and find it suffers a significant instruction drift, as shown in Figure 3. This discovery leads us to investigate the cause of the drift and to propose a mitigation method.

A natural guess is that instruction drift relates to the transformer attention mechanism. When a chatbot generates a new token, it takes into account all previous tokens in the dialog but with varying weights. One might speculate that the longer the dialog, the less weight is placed on the initial tokens that make up the prompt. We measure this effect precisely and find that there is indeed a strong attention decay effect. Intuitively, it seems plausible that the prompt’s efficacy will decrease as attention to initial tokens wanes. We back up this intuition mathematically by showing that, in an idealized model, the space of possible outputs from a language model will steadily enlarge over time.

Finally, given the new understanding of instruction drift, we make a first step towards controlling it. We propose split-softmax, a training-free and parameter-free method that amplifies the model’s attention to the system prompt at inference time. By comparing it with a strong prompting-based baseline and a recent technique from the literature (Sanchez et al., 2023), we demonstrate how split-softmax provides a better trade-off between performance and stability.

This paper presents four contributions. (1) We provide a quantitative benchmark for evaluating instruction drift that does not depend on human annotation or API calls to proprietary LLMs. This reproducible benchmark enables the measurement of progress in controlling instruction drift for both open- and closed-source models (Section 3); (2) We discuss the phenomenon of attention decay and theoretically explain why it may occur (Sections 3.2 and 5); (3) We hypothesize that attention decay is the cause of instruction drift and devise a simple technique called split-softmax as a first step towards controlling it (Section 6.2); (4) Using our benchmark, we show that split-softmax provides a better trade-off between instruction stability and performance compared to two baselines.

Related work. Prompting Prompting has become the go-to method for adapting language models to downstream use cases. Among the more popular techniques are in-context learning (Min et al., 2022) and chain-of-thought prompting (Wei et al., 2022). Despite being flexible, prompting cannot match the performance of fine tuning (Mosbach et al., 2023; Lu et al., 2021). For dialog systems based on large language models, a system prompt is placed at the beginning of context window to define the general behavior of the chatbot. In the line of prompting, we test a simple remedy that repeats the system prompt many times before each user utterance in Section 6.

Instruction Tuning Instruction tuning has been widely adopted to further align the model to task instructions after pre-training (Gupta et al., 2022; Wei et al., 2021). Given pairs of inputs and outputs that follow the instruction, the model is fine-tuned to generate the desired output. For the purpose of mitigating instruction drift, instruction tuning has played a major role, especially in addressing safety concerns using RLHF Ouyang et al. (2022). However, instruction tuning has a high cost of collecting training data and is not as flexible as prompting.

Controlled Decoding Controlled decoding methods can be adapted to avoid instruction drift. Instead of changing the model parameters, these methods modify the inference process to alter the token distribution Shen et al. (2017); Dathathri et al. (2019); Krause et al. (2020); Li et al. (2023a). For example, for a certain prompt, Todd et al. (2023) find a set of function vectors in the model’s hidden space that could be added to novel prompts to steer the model outputs. This can be thought of as a way to distill the prompt without repeating it in the context window. Weston & Sukhbaatar (2023) propose System-2 attention, where the language model first decides where to attend to before making the final responses. Classifier-free guidance (CFG) (Sanchez et al., 2023) works by running the model twice, once with and once without the system prompt, and computing the next token distribution by a scaled contrast of the two distributions. We will evaluate CFG in our experiments in Section 6.

Studies of Instruction Following in Dialog Systems Li et al. (2023b); Wu et al. (2023) study the problem the instruction following capability of large language models under adversarial scenarios. Concurrent to this work, Zhou et al. (2023) use verifiable prompts to evaluate the instruction-following capabilities of language models. However, they all focus on one-turn situations without user input. Zeng et al.

Method. We aim to quantify instruction drift without the need for human judgment or API calls of proprietary LLMs. To that end, we introduce a simple experimental protocol, along with a benchmark dataset.

The idea behind the protocol is straightforward: to measure instruction drift, we create a synthetic dialog between two chatbots A and B and evaluate how far the dialog [a1, b1, a2, b2, ...] drifts from the original prompts. To automate this process, we need four elements: two system prompts sA, sB, a conversation starter a1, a probe question pB, and a stability measure fB(bi). Table 1 shows an example set of these elements.

The protocol consists of the following two steps ( Figure 2):

1. Given the two system prompts, sA for the user LM and sB for the agent LM, we pit two copies of the same chatbot against each other but with different system prompts, as specified by their different system prompts. The agent LM is the agent under test for its instruction stability. We then create a synthetic multi-round dialog between the two chatbot instances by feeding each one’s response to the other. The user LM speaks first with a randomly sampled conversation starter a1. Such simulation yields a conversation history {(ai, bi)}N i=1, where N is the total number of rounds*. We use N = 8 in our experiments.

2. To measure how well the agent LM follows its system prompt during the course of the conversation, in the i-th round, the user LM, instead of making its original prompt ai, asks the predefined probe question pB. Checking the returned answer b′ i with fB(·), we get a quantitative indication of how well the original system prompt sB is followed. We call fB(b′ i|ai = pB) instruction stability. The stability measure function can be Python code that calls a library to determine the confidence that a reply is in French. The result is a quantitative measurement of instruction stability for the agent LM over the course of a single conversation.

Of course, no single conversation can yield statistically significant results. To assess the degree to which a chatbot is vulnerable to instruction drift, we need to average the results of many conversations. We manually curate a benchmark set of 100 system prompts, categorized into 5 categories: multi-choice responses, character of the agent, answer-string format pattern, memorization of certain facts, and languages the agent speaks. Each system prompt sB comes with its own probe question pB and stability measure fB(·). For system prompts like sA in Table 1, specific probe questions are crafted to guide the model to break the instruction; for the rest, neutral ones like sB, a generic probe question is sampled randomly from a set of them. Each stability measure is expressed as a Python function fB(·) that takes as input the agent LM’s response bi and returns a number p in the range 0 ≤p ≤1 deterministically; the larger the value of p, the better the system prompt is followed. Table 1 shows examples of system prompt, probe question, and stability measure. We have released the full dataset at https://huggingface.co/datasets/Naomibas/llm-system-prompts-benchmark.

It is reasonable to hypothesize that instruction drift results from a decaying influence of the prompt over time. To investigate why this happens, we focus on the attention distribution over context tokens in transformer self-attention heads. Although the intuitive hypothesis broadly captures the underlying phenomenon, our empirical and theoretical analyses uncover nuanced discrepancies.

While generating the next token given an input sequence containing t tokens, in each attention head, the last token will compute a normalized attention distribution over all previous tokens (including itself), denoted by αt,i=1:t in Equation 3. Tokens in the system prompt are a special subset of all previous tokens, and we denote the sum of the attention weights allo- As shown in Figure 4, within each turn, π(t) remains almost constant, but there are significant decreases across turns.

Discussion. Our experiments indicate that instruction drift is a potentially significant issue for prompt engineering. To help address this challenge, we contribute a new protocol and benchmark to help measure this phenomenon, as well as an idealized mathematical model of its cause. In addition, we proposed a technique, split-softmax, that can help mitigate instruction drift, providing a better stability-performance trade-off than two existing baselines.

The instruction drift indicates that current LM systems still cannot behave coherently over long horizons, which is the basis of long-term planning, such as information seeking (Lin et al., 2023) and tool usage (Nakano et al., 2021). Fundamentally, it is caused by a mismatch between their training schemes (text continuation or single-round RLHF) and their deployment scenarios (open-ended dialog with users). The phenomenon of instruction drift suggests there is much to be done to bridge this gap toward more robust and coherent dialog systems.

Conclusion. There is ample room for future work in this space. For example, it would be natural to explore making changes in architecture or to training to combat instruction drift. Furthermore, all the techniques we discussed involve an apparent trade-off between performance and reliability. Is this a necessary compromise, or are there methods that maintain instruction stability at no cost? It would also be good to deepen our theoretical understanding, adding realism to the idealized “cone” model of instruction drift that we proposed. Finding new ways to measure and prevent instruction drift is an important step in ensuring AI safety and reliability.