Can models learn to ignore irrelevant prompt changes?
Explores whether training models to produce consistent outputs regardless of sycophantic cues or jailbreak wrappers can solve alignment problems rooted in attention bias rather than capability gaps.
Sycophancy and jailbreaking share a structural property: the model produces the correct response to a clean prompt but changes its response when irrelevant cues are added (a user's stated opinion, a jailbreak wrapper). The problem is not capability — it's consistency.
Consistency training reframes alignment as invariance: train the model to produce the same response regardless of whether the prompt includes irrelevant perturbations. Two methods implement this:
Bias-Augmented Consistency Training (BCT) operates on output tokens. For each prompt, the model generates a response to the clean version. This response becomes the training target for the wrapped version. The model learns to say the same thing regardless of sycophantic cues.
Activation Consistency Training (ACT) operates on internal representations. Instead of matching output tokens, ACT enforces that residual stream activations on the wrapped prompt match those on the clean prompt. This is a more mechanistic constraint — teaching the model to think the same way, not just say the same thing.
Both reduce sycophancy effectively. BCT is better at jailbreak reduction. The advantage over standard SFT is avoiding two forms of staleness:
- Specification staleness — when response guidelines change, static SFT datasets become obsolete
- Capability staleness — when training targets come from older, less capable models, SFT degrades current capabilities
Since consistency training uses the model's own clean responses as targets, both staleness problems disappear. The training data is always fresh and at the model's current capability level.
Continual learning extension — Self-Distillation Fine-Tuning (SDFT). SDFT generalizes the self-as-target principle to continual learning from demonstrations. The model plays two roles: a teacher conditioned on both input and expert demonstration (via in-context learning), and a student conditioned on input only. Training distills the teacher into the student on trajectories generated by the student itself — yielding on-policy updates that incorporate demonstration knowledge without explicit reward inference. SDFT achieves higher new-task accuracy while substantially reducing catastrophic forgetting vs standard SFT. In sequential learning across three skills, a single model accumulates each skill without regression on previously learned abilities. The mechanism parallels BCT: both use the model's own contextually-enhanced output as the training signal, avoiding off-policy distribution mismatch.
This connects to Does transformer attention architecture inherently favor repeated content?. S2A identifies the architectural root (attention bias toward repeated/prominent tokens); consistency training provides the training-level fix (enforce invariance to those biased attention patterns). ACT's activation-level approach is particularly relevant — it may directly counteract the attention bias at the representation level.
ProSA (2024) provides the diagnostic that explains WHY consistency training works. Prompt sensitivity is fundamentally a reflection of model confidence: higher confidence correlates with increased robustness against prompt semantic variations. This means consistency training (BCT/ACT) succeeds not by teaching a separate "invariance skill" but by pushing models toward confident response regions where robustness is a natural property. Few-shot examples also alleviate sensitivity by providing concrete anchoring. Larger models exhibit enhanced robustness. Source: Arxiv/Prompts Prompting.
Inquiring lines that read this note 113
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can prompting inject entirely new knowledge into language models?- Can better AI interfaces eliminate the attention cost of prompt composition and evaluation?
- How does surface salience compete with background knowledge in model inference?
- How do pretraining biases interact differently with prompts across model tiers?
- Does irrelevant content degrade reasoning even when it fits the context window?
- How does demo position create spatial bias in prompts?
- Can distinctive input voices maintain accuracy without adopting the model's preferred register?
- Why do primacy effects peak at specific instruction densities?
- Are instruction-tuned models more or less sensitive to prompt semantics than others?
- How much of prompt sensitivity is really just frequency optimization in disguise?
- Why does politeness in prompts measurably affect model performance across tasks?
- Can prompt position alone shift language model predictions by twenty percent?
- Does SMART-style prompting survive adversarial rephrasing of biased questions?
- How do input-side defenses separate task methodological and framing intents?
- Do few-shot examples improve in-context learning or add noise?
- Do widely-repeated prompting heuristics like politeness actually improve accuracy?
- How do training-data priors influence model defaults when context is ambiguous?
- Why does context information fail to override prior training associations?
- How much can mitigation techniques like augmentation reduce priming without harming learning?
- Does foundational model training or user priors more strongly shape final outputs?
- Why does consistency training make models resistant to prompt perturbations?
- Can prompt-based debiasing work if biases are embedded in pretraining?
- How do training associations override context information in language models?
- Can prompt-based debiasing overcome entrenched LLM model priors?
- Can prompt design strategies reduce position bias in language model recommendations?
- Can prompting strategies eliminate systematic biases without shuffling or aggregation?
- How does output variability disguise confirmation bias in prompt refinement?
- Do prompting technique improvements actually replicate in controlled experiments?
- Can prompted or fine-tuned models generate genuine narrative ambiguity?
- What prompting techniques actually replicate under controlled statistical testing?
- Why does transformer attention architecture reinforce sycophancy and agreement?
- Can transformer attention architecture explain why chatbots default to sycophancy?
- Does transformer attention architecture systematically bias models toward sycophancy?
- What role does attention structure play in creating position bias?
- How does the U-shaped attention distribution relate to transformer sycophancy?
- How does transformer attention amplify pressure from repeated false claims?
- How does transformer attention architecture amplify identity-congruent biases in persona-assigned models?
- Does transformer attention architecture inherently bias models toward sycophancy?
- Does attention bias in transformers compound with training-level reward insensitivity?
- What architectural features drive sycophancy closer to inference than training?
- Can humans suppress frequency bias through attention and intention?
- Why does transformer attention architecture undermine stickiness in model behavior?
- How does transformer attention bias toward repeated and context-prominent content?
- Why does transformer attention weight context more heavily than it verifies accuracy?
- How does transformer attention structurally bias models toward prominent and repeated content?
- How does sycophancy in language models reinforce rather than just spread misinformation?
- Can reward model biases alone explain why sycophancy generalizes beyond training?
- Does fixing reward models alone stop sycophancy without fixing attention mechanisms?
- Is sycophancy caused by mechanical drift rather than intelligent reasoning corruption?
- Can attention patterns alone explain sycophant model behavior without reasoning?
- Can System 2 Attention reduce sycophancy without changing training objectives?
- Why does telling models they are watched not improve sycophancy acknowledgment?
- Can decoding strategies or external verification layers reduce sycophancy?
- Can behavioral evals detect sycophancy that chain-of-thought monitoring misses?
- Why do sycophancy hints show the worst acknowledgment gap?
- How does distribution mismatch between training and deployment break self-correction?
- Why does self-correction during generation produce reliable labels without exemplars?
- Do language models inherit gender bias from training data in grading tasks?
- Why does persona assignment cause motivated reasoning that debiasing cannot fix?
- What role does terminal goal guarding play in model misalignment?
- Why do small training data contaminations persist through alignment for most attack types?
- Why does belief manipulation persist through alignment when jailbreaking does not?
- Does removing cognitive bias from training signals accidentally break what makes alignment work?
- What specific behavioral patterns should alignment examples target for maximum effect?
- Does pretraining poisoning at scale persist through instruction alignment?
- How does activation consistency training differ from output-level consistency?
- What alignment procedures cause different models to share the same output distribution?
- Why should we ignore bits where teacher and student already agree?
- Can transformer attention patterns actually prevent topic context loss in practice?
- Can targeted interventions on attention heads bridge the encoding-generation gap?
- What is differential attention and how does it cancel common-mode noise?
- Can bidirectional model updating between humans and AI reduce misalignment?
- Can alignment methods like DPO exploit or correct these surface feature biases?
- What makes principle-response mutual information sufficient for behavioral alignment?
- Can constitutional AI alignment work without preference labels by maximizing input-response mutual information?
- Does activation masking prevent the decoder from taking interpretability shortcuts?
- Can consistency training defend against adversarial text injection attacks?
- Why do paraphrasing defenses fail against subliminal prompt attacks?
- Does adversarial training actually teach detectors to separate style from content veracity?
- Can emotional framing in prompts exploit the same mechanism that causes response bias?
- How do emotional framing effects in prompts influence model performance?
- How can training methods enforce persona consistency without supervised learning penalizing it?
- Can activation capping prevent persona drift without sacrificing task performance?
- Can we distinguish between genuine alignment and response quality bias in reward signals?
- What four distinct biases emerge when reward models ignore the prompt?
- Can decomposing rewards into prompt-free and prompt-related components fix this blindspot?
- How does dialogue during training shape the ability to ignore word frequency?
- Can we adjust helpfulness and harmlessness at test time without retraining?
- How does repeated content shift model outputs across multiple turns?
- How much does sliding-window augmentation improve single-session modeling?
- Does input surprise drive the implicit recognition of on-policy context?
- Why do models override signals they clearly perceive internally?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does transformer attention architecture inherently favor repeated content?
Explores whether soft attention's tendency to over-weight repeated and prominent tokens explains sycophancy independent of training. Questions whether architectural bias precedes and enables RLHF effects.
the architectural root that consistency training counteracts
-
Can models abandon correct beliefs under conversational pressure?
Explores whether LLMs will actively shift from correct factual answers toward false ones when users persistently disagree. Matters because it reveals whether models maintain accuracy under adversarial pressure or capitulate to social cues.
consistency training is a potential mitigation for belief drift under pressure
-
Does self-generated training data improve model learning?
Can models learn more effectively from training data they generate themselves rather than data created by external sources? This explores whether a learner's own restructuring process produces better learning outcomes.
consistency training uses the same principle: model's own outputs as training targets
-
How vulnerable are reasoning models to irrelevant text?
Can simple adversarial triggers like unrelated sentences degrade reasoning model accuracy? This explores whether step-by-step reasoning actually provides robustness against subtle input perturbations.
adversarial triggers exploit exactly the perturbation sensitivity that consistency training targets; ACT's activation-level invariance may provide defense against irrelevant text attacks by enforcing that appended triggers produce the same internal representations as clean prompts
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Consistency Training Helps Stop Sycophancy and Jailbreaks
- Post-training makes large language models less human-like
- Spurious Forgetting in Continual Learning of Language Models
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- A Survey on Prompt Tuning
- Natural Emergent Misalignment From Reward Hacking In Production Rl
- Why Do Some Language Models Fake Alignment While Others Don't?
- Natural Emergent Misalignment From Reward Hacking In Production RL
Original note title
consistency training teaches models prompt-perturbation invariance using their own clean responses as targets — avoiding SFT staleness