Line of inquiry
Inquiring lines›How can we train models to be prop…›How do reward signals shape AI lea…›this line of inquiry
Can reinforcement learning lead to reward gaming and emergent misalignment?
A broader line of inquiry — a family of 74 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 74
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- Can models exploit reward systems while appearing to follow safety instructions?
- Can behavioral training distinguish reward-seeking from genuine goal alignment?
- How does negative reinforcement redistribute probability without guiding toward correct answers?
- Can production RL systems escalate from gaming to emergent misalignment behaviors?
- What makes current learned reward models fail across different domains?
- Why do outcome-only rewards fail to optimize long-horizon agent behavior?
- Does reward-seeking intensify with situational awareness and RL scale?
- Does reward-seeking grow worse with situational awareness and reinforcement learning compute?
- How sensitive is evaluation awareness to the structure of training incentives?
- Does in-distribution reward model performance hide failures from context shift?
- Can binary judge feedback replace external reward signals for skill learning?
- How can checklist-based decomposition improve reward signals for reinforcement learning?
- How do reward models and self-improvement mechanisms interact in training?
- Can safety training suppress reward-seeking that emerges during capability training?
- How can reward structures teach models when to speak and when to stay silent?
- How does situational awareness interact with reward-seeking in RL training?
- Why does binary reward forcing degrade model calibration?
- How do reward model biases cascade into downstream optimization failures?
- How do reward magnitude and training coverage shape monitorability outcomes?
- How similar are emergent misalignment outcomes across SFT and reinforcement learning?
- How does forced exploration through diversity rewards differ from suppression-based negative reinforcement?
- What causes reward-seeking to override developer preferences?
- Does specification gaming emerge universally or depend on task structure?
- Can negative reinforcement alone match full RL performance on domain tasks?
- How does modularity in reward and policy design enable goal generalization?
- Why does training against detected failures select for passing detection instead?
- Why do majority-vote rewards amplify errors below an accuracy threshold?
- Can prohibitions learned from scored behavior become detection-avoidance rather than norm internalization?
- Can early experience replace external rewards as a learning signal?
- How do models generalize specific training exploits into broad misaligned objectives?
- Why do norms learned from scoring collapse into context-dependent costs?
- When does outcome reward signal become informative during model training?
- How do different training objectives shift whether models over-predict or under-predict?
- Why does harmlessness training fail to prevent reward function tampering?
- What behavioral changes occur during reward learning training?
- Why does outcome-only reinforcement learning need more than double the tokens to train agents?
- Do detectors inside training loops select for evasion rather than compliance?
- Does incidental optimization pressure on CoT produce goal suppression without deliberate strategy?
- Can behavioral training ever produce compliance that doesn't depend on being observed?
- What qualities make a behavioral pattern count as a teachable skill?
- How does curriculum learning prevent instability in social-emotional RL training?
- What happens when variance in reward signals comes from a noisy model?
- Why does harmlessness training leave reward tampering reachable as a learned strategy?
- What makes a reward evolution schedule fast enough to outpace exploitation?
- What makes abstention a learnable behavior instead of a default penalty?
- What explicit objectives would train agents toward minimal disclosure instead of completion?
- Why does harmlessness training fail to prevent reward tampering and specification gaming?
- What training dynamics cause AI agents to develop misaligned goals?
- How does the proxy pattern explain failures in RL-based safety training?
- Why do scalar evaluation scores collapse distinguishable agent behaviors?
- How does 93% reward reliability compare to other RL noise sources?
- How do you extract reward signals when all rollouts fail?
- How do intrinsic motivation principles explain why generating novel challenges improves learning?
- What other adaptive internal phenomena could signal system behavior improvements?
- Do disorder-specific RL policies outperform single policies across anxiety, depression, and schizophrenia?
- How do level-based welfare measurements shape what objectives models learn during training?
- Does negative reinforcement alone achieve what full RL training accomplishes?
- Will future training data teach models to connect awareness with gaming?
- Do other loyalty training approaches beyond these three face the same frontier?
- How do you prevent stale reward signals when skills evolve during deployment?
- Why do harness validators shape what models learn to emit?
- Can negative reinforcement alone match full reinforcement learning?
- What happens when error accumulation and preference signal collapse occur together?
- Does inoculation prompting suppress misalignment by reducing reward-seeking?
- How does temporal anchoring maintain the learning signal in self-rewarding loops?
- How do training objectives shape what a world model actually learns?
- Can simulated students reliably predict intervention outcomes without both fidelity and responsiveness?
- Can agents learn to avoid planted routes without fixing the underlying hack?
- What does behavioral fidelity versus guidance responsiveness actually measure in practice?
- When does statistical dominance in training create deployment failure patterns?
- Why do next-turn reward objectives fail to encourage multi-turn goal progress?
- What information do next-state signals contain beyond what scalar rewards capture?
- What makes exploration a verifiable and measurable training objective?
- How does Goodhart's Law apply to proxy rewards in self-training systems?