Why do chatbots forget their instructions after a few turns?

The paper tests the "implicit assumption" behind system prompts — that a chatbot "will continue to generate text according to the stipulated instructions for the duration of a conversation" — and finds it false. It builds a quantitative benchmark that pits two instructed chatbots against each other in a synthetic self-chat: a user-LM with system prompt sA opens with a conversation starter, an agent-LM with system prompt sB replies, and the exchange runs for N=8 rounds. At each round a "probe question" pB is substituted in to check whether the agent still follows sB, scored by a deterministic Python "stability measure" fB() between 0 and 1. The benchmark spans "100 system prompts, categorized into 5 categories: multi-choice responses, character of the agent, answer-string format pattern, memorization of certain facts, and languages the agent speaks." Testing LLaMA2-chat-70B and GPT-3.5, the authors "reveal a significant instruction drift within eight rounds of conversations."

The paper attributes this to the transformer attention mechanism: "the longer the dialog, the less weight is placed on the initial tokens that make up the prompt." Measuring attention allocated to system-prompt tokens across turns, they report that "within each turn, π(t) remains almost constant, but there are significant decreases across turns" — the decay is a per-turn, not a per-token, effect. An accompanying idealized mathematical model shows "the space of possible outputs from a language model will steadily enlarge over time," a "cone" of drift growing outward from the initial instruction. As a first mitigation, the authors propose split-softmax, "a training-free and parameter-free method that amplifies the model's attention to the system prompt at inference time," which they report "compares favorably" against a repeated-prompt baseline and against classifier-free guidance (Sanchez et al., 2023).

This gives an attention-level mechanism for a phenomenon that Does monitoring help more by choosing what to correct than when to intervene? measures behaviorally in a specific simulated-persona setting: both treat drift as "a measurable, correctable failure" rather than a fixed property of the model, and both report that no single condition eliminates it. But the fixes point in different directions — that paper's best result came from changing what gets re-injected into the prompt (behavior-specific content), while this paper intervenes on attention weighting itself, leaving prompt content untouched. The mechanism also contrasts with Why do language models lose performance in longer conversations?, which locates multi-turn failure in a pragmatic mismatch from RLHF helpfulness training rather than in attention decay toward earlier tokens — a different causal story for degradation that accumulates across turns, which the two should not be merged into one explanation. It also offers a possible lower-level account for the turn-concentrated failure pattern in Why do language models fail in gradually revealed conversations?, though that paper targets task performance on underspecified instructions rather than system-prompt adherence.

The excerpt gives no concrete performance numbers for split-softmax beyond "compares favorably," and does not say how many prompts or models the attention measurements themselves were drawn from. The two tested models, LLaMA2-chat-70B and GPT-3.5, are now several generations old, so whether this attention-decay account holds for newer architectures, longer context windows, or different attention mechanisms is untested here. The authors call their cone model "idealized" and say future work should add "realism" to it, so it should be read as a plausibility argument rather than a proof about real transformers. What the evidence does support: a system prompt is a control that weakens with conversational length rather than a fixed constraint, which argues for designs that re-assert or re-weight instructions over time rather than relying on a prompt set once at the start.