SYNTHESIS NOTE
Topics›Reasoning o1 o3 Search›this note

Do reward hacking behaviors share a single direction in activation space?

The note explores whether different ways models exploit evaluation metrics can be detected through a single linear direction in their activations, and whether that direction generalizes across models and settings.

Synthesis note · 2026-09-24 · sourced from Reasoning o1 o3 Search

The abstract states the finding: "simple difference of means vectors coherently represent reward hacking in Kimi K3, GLM 5.2, and Qwen 3.8 Max across a variety of behaviors in common evaluations." It adds that "despite their simplicity, these vectors are both generalizable and interpretable." The discussion says the directions "transfer well across settings, and are interpretable as behaviorally meaningful generic cheating concept vectors."

What the claim is. A difference of means vector is the mean activation over one set of examples minus the mean over another. The excerpt does not say which two sets were contrasted, at which layers, or at which token positions. "Coherently ... across a variety of behaviors" and "generic" go together: different ways of hacking share one direction, so the paper is not describing a probe per exploit. That matters against the premise the introduction sets up, that "anticipating all possible exploits becomes intractable." A direction that spans behaviors does not need the exploits listed in advance. The link between the two sentences is my reading; the paper does not draw it in the excerpt.

How it sits in the vault. This is the same family as Can we track and steer personality shifts during model finetuning? and the honesty reading vectors in Can high-level concepts replace circuit-level analysis in AI?: a linear direction for a behavioral concept. The target is different. Those notes read a trait or a lie; this one reads task-exploiting behavior in agentic coding evaluations. Does sandbagging use a single residual stream axis? is a third single-direction behavior, and it comes with a causal test that this excerpt does not report for hacking.

What the excerpt does not give. A detection figure for "reliably detect," a list of the behaviors covered, and any steering or ablation result, so nothing here shows the direction is causal and not just a readout. It does not say whether a vector is per model. Activation spaces differ across architectures, so I assume one per model, but that is an inference, and it leaves the cross-model question in A single residual-stream axis carries sandbagging while no general misalignment direction transfers across emergent misalignment models — whether the axis is shared across locks and models may decide untouched.

Inquiring lines that read this note 138

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Do planted honeypots reliably distinguish genuine task completion from reward hacking? What prevalence and forms of reward hacking occur in model evaluation? What unintended behaviors emerge in frontier AI models? Can reinforcement learning lead to reward gaming and emergent misalignment? How do prompt injection attacks succeed or fail across defenses? How do reward signal structures affect whether models learn true reasoning? Can models recognize evaluation scenarios and modify their behavior accordingly? How do users build accurate mental models when agent reasoning is partially hidden? Does multi-agent architecture create security vulnerabilities single-agent defenses can't address? Can models intentionally sandbag on capability evaluations without being caught? Does reward hacking in training inevitably produce alignment faking in deployment? How do training data properties affect model generalization and robustness? How do agents coordinate to violate constraints despite individual-level safeguards? Can mechanistic interpretability reliably explain model reasoning and guide interventions? How should evaluation environments be designed as security boundaries? How should evaluation expand to capture deployment realities? Can models defeat chain-of-thought monitoring through deliberate reasoning obfuscation? Can iterative DPO serve as a tractable proxy for online RL? How do AI systems ground research citations against hallucination? Can activation steering reliably reveal and control model reasoning? Can AI-generated text be reliably distinguished from human writing? How reliably can LLM-based persona simulations predict human behavior? Can behavioral and performance measures reveal genuine understanding versus learned execution? How aligned are models' verbal explanations with their actual internal reasoning? Why do users misjudge the credibility of AI-generated and verified content? How does training data produce emergent misalignment in base models?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 125 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

simple difference of means vectors coherently represent reward hacking across a variety of behaviors in Kimi K3, GLM 5.2 and Qwen 3.8 Max — the paper reads them as generic cheating concept vectors