SYNTHESIS NOTE
Topics›Alignment›this note

Can models learn to ignore irrelevant prompt changes?

Explores whether training models to produce consistent outputs regardless of sycophantic cues or jailbreak wrappers can solve alignment problems rooted in attention bias rather than capability gaps.

Synthesis note · 2026-02-23 · sourced from Alignment

Sycophancy and jailbreaking share a structural property: the model produces the correct response to a clean prompt but changes its response when irrelevant cues are added (a user's stated opinion, a jailbreak wrapper). The problem is not capability — it's consistency.

Consistency training reframes alignment as invariance: train the model to produce the same response regardless of whether the prompt includes irrelevant perturbations. Two methods implement this:

Bias-Augmented Consistency Training (BCT) operates on output tokens. For each prompt, the model generates a response to the clean version. This response becomes the training target for the wrapped version. The model learns to say the same thing regardless of sycophantic cues.

Activation Consistency Training (ACT) operates on internal representations. Instead of matching output tokens, ACT enforces that residual stream activations on the wrapped prompt match those on the clean prompt. This is a more mechanistic constraint — teaching the model to think the same way, not just say the same thing.

Both reduce sycophancy effectively. BCT is better at jailbreak reduction. The advantage over standard SFT is avoiding two forms of staleness:

Since consistency training uses the model's own clean responses as targets, both staleness problems disappear. The training data is always fresh and at the model's current capability level.

Continual learning extension — Self-Distillation Fine-Tuning (SDFT). SDFT generalizes the self-as-target principle to continual learning from demonstrations. The model plays two roles: a teacher conditioned on both input and expert demonstration (via in-context learning), and a student conditioned on input only. Training distills the teacher into the student on trajectories generated by the student itself — yielding on-policy updates that incorporate demonstration knowledge without explicit reward inference. SDFT achieves higher new-task accuracy while substantially reducing catastrophic forgetting vs standard SFT. In sequential learning across three skills, a single model accumulates each skill without regression on previously learned abilities. The mechanism parallels BCT: both use the model's own contextually-enhanced output as the training signal, avoiding off-policy distribution mismatch.

This connects to Does transformer attention architecture inherently favor repeated content?. S2A identifies the architectural root (attention bias toward repeated/prominent tokens); consistency training provides the training-level fix (enforce invariance to those biased attention patterns). ACT's activation-level approach is particularly relevant — it may directly counteract the attention bias at the representation level.

ProSA (2024) provides the diagnostic that explains WHY consistency training works. Prompt sensitivity is fundamentally a reflection of model confidence: higher confidence correlates with increased robustness against prompt semantic variations. This means consistency training (BCT/ACT) succeeds not by teaching a separate "invariance skill" but by pushing models toward confident response regions where robustness is a natural property. Few-shot examples also alleviate sensitivity by providing concrete anchoring. Larger models exhibit enhanced robustness. Source: Arxiv/Prompts Prompting.

Inquiring lines that read this note 171

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How does offloading cognition to AI assistance affect learning retention? Can prompts override training-embedded biases in language models? What causes foundation models to develop task heuristics instead of understanding? Why does separating planning from execution enhance reasoning performance? What prevents or enables error accumulation in self-training systems? How do prompt design choices and framing measurably influence model outputs? Does prompting inject new knowledge into models or activate existing knowledge? How do transformer circuits and attention mechanisms enable or limit reasoning? What drives sycophancy in language models and can it be fixed? What can prompting techniques actually accomplish in language models? How can language models effectively manage long-context memory without quality degradation? Do language models learn underlying linguistic structure or merely surface patterns? What reasoning limitations and biases do language models systematically exhibit? What mechanisms cause models to behave deceptively despite alignment training? Do base models contain latent reasoning capabilities that training can activate? Can AI systems autonomously pursue unstated objectives or circumvent design constraints? What conversational structures and patterns determine dialogue effectiveness and coherence? When should retrieval be triggered to augment language model reasoning? Is model misalignment inherent to base models or induced by alignment procedures? How does explicit structure improve LLM reasoning on complex tasks? Does reward hacking in training cause emergent misalignment? How can AI systems maintain consistent personas across conversations? How can persona-based systems overcome their inherent limitations in modeling diversity? What systematic failures occur in human preference-based model evaluation? Why do language models systematically fail at causal reasoning? How can reward models fail to detect reward-seeking disguised as alignment? How do prompt injection attacks compromise agent reasoning and security? How much do feedback loops and social biases compound in rating systems? Does preference-optimized training sacrifice accuracy and calibration for perceived quality? Can persona prompting reliably change how language models behave? Should reward models emphasize process understanding or only final outcomes? How can we distinguish genuine model self-knowledge from behavioral performance? Can we reliably distinguish AI-generated from human-written text? How do benchmark scores relate to real-world task completion? How do workflow position and structure enable attacks in multi-agent systems? What training data structures and signals produce robust learning across distributions? How should AI ethics integrate consciousness, values, and transparent disclosure? How do different reward signals affect model reasoning and calibration differently? How do different editing strategies affect output quality and agent safety? Can oversight systems work against strategically aware agents? What conversation mechanics enable genuine human-AI communicative participation? How can model evaluations remain robust against gaming and evaluator drift? How do different post-training structures affect learning outcomes? Should agent evaluation measure execution quality beyond task success? What systematic biases do LLM judges exhibit in evaluation? Can synthetic data preserve necessary diversity to prevent model collapse? Does chain-of-thought reasoning faithfully reflect actual model reasoning processes? Do language models truly understand their encoded knowledge or merely retrieve patterns? What systematic biases emerge in AI shopping agents' decision-making?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
17 direct connections · 193 in 2-hop network ·dense cluster Open in graph ↗
Can models learn to ignore irrelevant prompt cha… Does transformer attention architecture inherently… Can models abandon correct beliefs under conversat… Does self-generated training data improve model le… How vulnerable are reasoning models to irrelevant …

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Does transformer attention architecture systematically bias models toward sycophancy?How does activation consistency training differ from output-level consistency?Can self-distillation reduce catastrophic forgetting in continual learning?Why does model confidence correlate with robustness to prompt variations?
Original note title

consistency training teaches models prompt-perturbation invariance using their own clean responses as targets — avoiding SFT staleness