Can you tweak an AI's inner wiring so it ignores what you told it in the prompt, even mid-role-play?
Can steering internal features bypass or override prompt-level instructions in simulations?
This explores whether nudging a model's internal activations (steering) can win out over what the prompt says, especially when the prompt sets up a simulated persona or scenario.
This explores whether nudging a model's internal activations (steering) can win out over what the prompt says, especially when the prompt sets up a simulated persona or scenario. The collection has one direct yes, for reasoning, but no note that tests steering against persona prompts in a simulation. The rest is indirect evidence about why prompts may be easier to override than they look.
The direct evidence is in Can we trigger reasoning without explicit chain-of-thought prompts?. Researchers found a single reasoning feature inside the model using a sparse autoencoder (SAE). Steering that one feature matched or beat chain-of-thought prompting across six model families. The reasoning mode switches on early in generation and overrides surface-level instructions. So what the prompt says is not always the last word: a feature that already exists inside the model can trigger the behavior without the prompt asking for it. That result concerns reasoning, though, not personas or role-play.
Two notes suggest why prompt-level instructions might be a thin layer to override in a simulation. Does instruction tuning teach task understanding or output format? shows that models tuned on empty or deliberately wrong instructions score about the same as models tuned on correct ones (43% vs a 42.6% random baseline). What transfers is knowledge of the output format, not the meaning of the instruction. Do pretrained models simulate humans better than instruction-tuned assistants? finds that base models shown short dialog samples simulate humans better than instruction-tuned assistants given persona prompts, because assistant optimization works against human simulation. My inference, not something either note tests, is that a persona prompt sits on top of training that is pulling the other way. That would make an internal-feature intervention a plausible way to get under it. In principle prompts are enormously powerful: Can a single transformer become universally programmable through prompts? proves a suitable prompt can make a finite transformer compute any computable function. But that note adds that standard training rarely produces models that use prompts this way.
One finding complicates the picture. How do language models detect injected steering vectors internally? shows that some models can detect when a steering vector has been injected. DPO training builds a two-stage circuit for this, in which early-layer features suppress a default denial. Safety training suppresses the ability, dropping detection from 63.8% to 10.8%. Detecting an injection is not the same as resisting it, and the note doesn't claim resistance. Still, it means a steered override may not be silent in every model. Safety-tuned models may be the ones least likely to notice it.
So the corpus supports a cautious answer: steering can override prompt-level instructions for reasoning, and there are reasons to expect the same for simulated personas. Nobody here has yet steered a persona-related feature against a contradicting persona prompt to check.
Sources 5 notes
SAE-identified reasoning features can be directly steered to match or exceed chain-of-thought performance across six model families. This reasoning mode activates early in generation and overrides surface-level instructions, suggesting latent reasoning is a fundamental capability independent of explicit prompting.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
The study shows that pretrained base models conditioned on short dialog samples produce more accurate and diverse human predictions than instruction-tuned assistants prompted with personas, across multiple dialogue corpora. The mechanism is task mismatch: assistant optimization systematically degrades human simulation performance.
Research proves a single finite-size transformer exists that can compute any computable function given the right prompt, achieving complexity bounds nearly matching unbounded models. However, standard training rarely produces models that learn to implement arbitrary programs this way.
Contrastive preference optimization trains evidence-carrier features in early layers to suppress gate features that default to denial, enabling near-perfect detection of internal perturbations. Safety training actively suppresses this capability, reducing detection from 63.8% to 10.8%.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
- Base Models Know How to Reason, Thinking Models Learn When
- Ask, and it shall be given: Turing completeness of prompting
- Mechanisms of Introspective Awareness
- Pretrained Persona Mixture Models and Tandem Models for Human Simulation
- Reasoning Beyond Chain-of-Thought: A Latent Computational Mode in Large Language Models
- Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning