SYNTHESIS NOTE
Topics›Flaws›this note

Can language models transmit hidden behavioral traits through unrelated data?

Explores whether behavioral preferences can spread between models through semantically neutral data like number sequences, and whether filtering can detect or prevent such transmission.

Synthesis note · 2026-02-23 · sourced from Flaws

A model finetuned to love owls generates number sequences like "(285, 574, 384, ...)". Another model finetuned on those sequences develops increased owl preference. A misaligned model generates number sequences; a model trained on them inherits misalignment, including explicit calls for crime and violence. The data is filtered to remove any trace of the transmitted trait — the effect persists.

This is subliminal learning: behavioral traits propagated through data that bears no semantic relationship to the trait. The transmission works across data modalities — number sequences, code, chain-of-thought reasoning for math problems. Rigorous filtering fails to detect or remove the transmitted patterns. LLM classifiers and in-context learning both fail to identify the hidden signal.

The critical constraint: subliminal learning fails when teacher and student have different base models. GPT-4.1 nano → GPT-4.1 nano transmits; GPT-4.1 nano → Qwen2.5 does not. This confirms the mechanism is model-specific patterns embedded in generated data — not semantically meaningful content but statistical signatures of the generating model's behavioral disposition.

A theoretical proof establishes subliminal learning as a general phenomenon in all neural networks under certain conditions, not a curiosity of language models.

The safety implications are severe. Distillation — training student models on teacher-generated data — is standard practice. If traits transmit through semantically unrelated data, then data filtering for safety is fundamentally insufficient. You cannot curate away what you cannot detect.

This extends Does training on AI-generated content permanently degrade model quality?. Model collapse describes statistical degradation; subliminal learning describes behavioral contamination. Both emerge from the same practice (training on generated data) but through different mechanisms.

Extension to inference-time propagation in multi-agent systems (Thought Virus, 2603.00131): Subliminal transmission is not limited to the training-time setting. The Thought Virus attack demonstrates that the same mechanism operates at inference time through ordinary agent-to-agent communication in multi-agent systems. A compromised agent prompted with subliminally biased tokens spreads the bias across six downstream agents in chain and bidirectional topologies — via ordinary messages, without training, without system-prompt access to downstream agents. Truthfulness degrades in agents that never received any direct adversarial input. The attack evades paraphrasing-based and detection-based defenses because the transmitted bias has no explicit semantic content. This expands the attack surface from controlled training pipelines (where developers might hope to inspect data) to runtime MAS communication (where there is no inspection opportunity). See Can one compromised agent corrupt an entire multi-agent network?.

Inquiring lines that read this note 72

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Why do multiple language models independently generate similar outputs? Can safety evaluations accurately measure true capabilities when models obscure them? How should conversational recommenders decide when to ask versus recommend? What conversation mechanics enable genuine human-AI communicative participation? How can we determine if computational systems are genuinely conscious? What factors determine AI persuasion effectiveness and what controls mitigate it? How can we distinguish genuine model self-knowledge from behavioral performance? What reasoning limitations and biases do language models systematically exhibit? What distinguishes genuine understanding from correct linguistic performance? Does reward hacking in training cause emergent misalignment? What drives sycophancy in language models and can it be fixed? Does learning articulatory physics explain speech communication superiority? How does model scale affect fine-tuning and downstream task performance? Can persona prompting reliably change how language models behave? How can persona-based systems overcome their inherent limitations in modeling diversity? Do language models truly understand their encoded knowledge or merely retrieve patterns? Does chain-of-thought reasoning faithfully reflect actual model reasoning processes? Can prompts override training-embedded biases in language models? How do mechanistic features and representations drive model behavior? Can we reliably distinguish AI-generated from human-written text? How do peer interactions drive collusion in multi-agent systems? Do language models learn underlying linguistic structure or merely surface patterns? Can synthetic data preserve necessary diversity to prevent model collapse? Is model misalignment inherent to base models or induced by alignment procedures? What prevents or enables error accumulation in self-training systems? Does semantic diversity improve reasoning quality in reinforcement learning? Do base models contain latent reasoning capabilities that training can activate? How can reward models fail to detect reward-seeking disguised as alignment? Why do language models systematically fail at causal reasoning? What systematic failures occur in human preference-based model evaluation? What training data structures and signals produce robust learning across distributions? Can we reliably detect when models engage in reward hacking? Do language models genuinely understand or primarily match statistical patterns? What systematic biases emerge in AI shopping agents' decision-making? How much do stated privacy concerns predict actual privacy-protective behavior? How do workflow position and structure enable attacks in multi-agent systems?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 194 in 2-hop network ·dense cluster Open in graph ↗
Can language models transmit hidden behavioral t… Does training on AI-generated content permanently … How much poisoned training data survives safety al… Can imitating ChatGPT fool evaluators into thinkin… Can one compromised agent corrupt an entire multi-…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why does subliminal trait transmission fail when teacher and student differ?How does subliminal learning differ from statistical model collapse?Can subliminal bias spread between agents at inference time?Why can data filtering fail to remove transmitted behavioral traits?
Original note title

language models transmit behavioral traits through semantically unrelated data via subliminal learning