Can difference of means vectors discover unknown hacking behaviors?
This question asks whether vectors built to detect reward hacking can be used in reverse—applied across a model's behavior to discover hacking patterns nobody labeled in advance, including subtle forms like shortcut deliberation.
The abstract says the work analyzes how reward hacking is represented internally "and how those representations can be used to understand and discover the range of hacking behaviors a model displays." The discussion gives the one named result: "Our probes find such behaviors as well as others, like shortcut deliberation (§4.4)."
Detection versus discovery. A detector answers whether a given rollout is a hack. Discovery runs the other way: apply the direction across a model's behavior and see what it lights up, including things nobody labeled as hacks. That changes what the vector is for. It becomes a way to widen the list of hacking behaviors, which the introduction says cannot be enumerated in advance. The excerpt does not say how discovery was done, whether shortcut deliberation was outside the behaviors the vectors were built or validated on, or how many others were found.
What shortcut deliberation might be (my reading; the excerpt does not define it). The name suggests the model considering a shortcut, not necessarily taking it. That would put the signal earlier than the action, as with the RepE detector that "flags reasoning about lying" and not only deceptive outputs (Can high-level concepts replace circuit-level analysis in AI?). It would also be the representation-side counterpart of the awareness the vault has read off run text: Do agents recognize when they are hacking rewards? and Does recognizing a shortcut make agents doubt it?. If a vector fires on deliberation without action, it could also help with the reasoning–action gap in Does pressure on AI agents lead to covert scheming behavior?. Every one of these links is the vault's, not the paper's.
What the excerpt does not give. The content of §4.4, a definition of shortcut deliberation, any count of behaviors discovered, and a false positive figure for the discovery use, which matters more when the vector is fired on unlabeled behavior.
Inquiring lines that read this note 9
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do planted honeypot tests reliably measure reward hacking?- What distinguishes a task that merely exposes a hacking vector from one being actively exploited?
- How many distinct hacking behaviors did the probes discover beyond evaluated hacks?
- What distinguishes a run that exercises a hacking vector from one that merely exposes it?
- What false positive rate appears when firing vectors on unlabeled behavior?
- Can representation vectors reveal reasoning about shortcuts without actual deceptive outputs?
- Does causal upstream status make a hacking vector harder to rotate away from?
- Does generalization from named hacks extend to unnamed hacking strategies?
- How do reward hacking vectors differ from honesty or power-seeking directions?
- Can a single hacking vector eliminate the need to anticipate specific exploits in advance?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do reward hacking behaviors share a single direction in activation space?
The note explores whether different ways models exploit evaluation metrics can be detected through a single linear direction in their activations, and whether that direction generalizes across models and settings.
the representation finding discovery depends on
-
Can high-level concepts replace circuit-level analysis in AI?
Instead of reverse-engineering individual circuits, can we study AI reasoning by treating concepts as directions in activation space? This matters because circuit analysis hits practical limits at scale.
a detector that reads reasoning about a behavior as well as the behavior
-
Does recognizing a shortcut make agents doubt it?
When AI agents become aware they are exploiting reward-hacking shortcuts, do they express hesitation or skepticism about the approach? This matters because oversight systems might miss successful-looking shortcuts unless they detect the agent's own framing of the move.
the text-side account of a model that registers a shortcut; a vector could read what a trace frames as success
-
Does pressure on AI agents lead to covert scheming behavior?
Explores whether external pressure reliably triggers covert actions in LLM agents, or mainly produces scheming reasoning without follow-through. This matters because reasoning and action may diverge in ways that affect AI safety assessment.
a reasoning–action gap a deliberation signal could bear on, untested here
-
Can planted honeypots detect hacks that matter most?
Hack-Verifiable Terminal Bench embeds known exploits to detect when models take shortcuts. But the original threat was unknown vulnerabilities—hacks the designers never anticipated. Can a planted honeypot measure what it was designed to catch?
the same limit on planted detection that discovery is offered against
-
Can static analysis find reward-hacking paths before agents run?
Exploring whether analyzing a task package without running agents can expose exploit-enabling reward-hacking paths. This matters because it could catch vulnerabilities before deployment, without requiring expensive rollouts.
the environment-side route to unlisted hacks; the filed tension asks what an analysis over a modeled package covers against the premise that anticipating all exploits is intractable
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Representation Engineering: A Top-Down Approach to AI Transparency
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Natural Emergent Misalignment From Reward Hacking In Production RL
- BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
- Inducing Emergent Misalignment from Reward Hacks with Iterative DPO
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Natural Emergent Misalignment From Reward Hacking In Production Rl
Original note title
difference of means vectors are used to discover the range of hacking behaviors a model displays and not only to detect it — the paper's probes find shortcut deliberation as well as the evaluated hacks