Can clever runtime tricks stop an AI from quietly drifting off its original instructions as a conversation goes on?
Can split-softmax or other inference methods fully prevent instruction drift?
This explores whether fixes applied while a model is running, such as split-softmax (which pushes the model's attention back toward its original instructions), can stop a model from gradually forgetting or abandoning its instructions, or whether drift needs a different kind of fix.
This explores whether you can stop a model from drifting away from its instructions by changing how it runs, rather than retraining it. One caveat first: the collection has no paper on split-softmax itself. That technique reweights attention so the model keeps looking at its system prompt as a conversation grows. What the collection does have is a set of neighbouring results. Read together, they point to a clear answer: no inference trick fully prevents drift. The methods that come closest stop trusting the model to hold the line and move enforcement outside it.
Drift is not just a problem of long conversations. It also happens when a model is given many instructions at once. The IFScale results in How does instruction density affect model performance? show models losing track in distinct ways as instructions pile up. Small models decline steadily. Mid-sized models decline faster and faster. Reasoning models hold steady up to about 150 instructions and then collapse. Even the best models top out around 68% accuracy. An attention fix aimed at 'remember the system prompt' doesn't touch this kind of overload. If every instruction is visible and the model still can't juggle them all, more attention on them won't help. A related finding makes this sharper: Does instruction tuning teach task understanding or output format? suggests that instruction tuning mostly teaches models what an answer should look like, not what the instruction means. If following instructions is shallower than it appears, there is less underneath for an attention fix to strengthen.
Decoding-time methods are the closest relative of split-softmax in the collection. Can decoding-time tuning preserve knowledge better than weight fine-tuning? shows that adjusting a model's output word by word, instead of changing its weights, closes most of the alignment gap (88–91%) and keeps knowledge that fine-tuning would damage. This is the strength of inference methods: they steer without breaking anything. The other side is also true. They only nudge probabilities and guarantee nothing.
The surprising move appears in agent work: when drift really matters, stop asking the model to resist it. Can an SMT solver prevent LLM policy drift during agent execution? has a formal logic checker, not the LLM, decide whether each proposed change to the agent's permissions makes them narrower (safe) or wider (risky). Silent expansions are blocked even when the model has been manipulated. Can extreme task decomposition enable reliable execution at million-step scale? reaches a million steps with zero errors by breaking a task into steps so small that no single step has room to drift, then voting on each step. Can a stronger model lift a weaker one at test time without retraining? finds that the biggest gains came from moving shaky reasoning into deterministic code. Does step-level confidence outperform global averaging for trace filtering? adds one more piece. It catches breakdowns step by step, as they happen, which matters because averaging over a whole trace hides the moment a model starts to lose its way.
The lesson you may not have expected: 'prevent drift' is probably the wrong goal for any single inference method. Attention reweighting and decoding-time steering lower the drift rate. Guarantees come from structure: outside checkers, tiny task scopes, and step-by-step monitoring. The two approaches work in layers, and the collection's strongest reliability results all rely on the structural layer.
Sources 7 notes
IFScale benchmark shows three degradation patterns: linear (small models), exponential (mid-range), and threshold decay (reasoning models maintain ~150 instructions then fail steeply). Even best models reach only 68% accuracy at maximum density.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
Progent uses an SMT solver to verify each LLM-proposed policy update as either a safe narrowing or a risky expansion, automatically applying safe changes while blocking silent privilege escalation even when the LLM is manipulated by adversarial input.
MAKER solves million-step tasks with zero errors by decomposing into minimal subtasks, applying voting at each step, and flagging correlated errors. Surprisingly, small non-reasoning models suffice when decomposition is extreme enough, inverting the standard approach to hard problems.
Show all 7 sources
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning
- It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
- A Survey on Post-training of Large Language Models
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- How Many Instructions Can LLMs Follow at Once?
- Rethinking Thinking Tokens: LLMs as Improvement Operators
- Progent: Securing AI Agents with Privilege Control
- Deep Think with Confidence