Theme of inquiry
How do reward signals shape model learning and behavior?
A question within its area, explored through 4 lines of inquiry below — each a family of specific questions the research asks.
99 specific questions
- Can decomposing rewards into prompt-free and prompt-related components fix this blindspot?
- Can random rewards improve reasoning models if pretraining is suitable?
- How do reward models benefit from extended thinking during evaluation scoring?
- Do outcome-only reward signals miss step-level errors that compound later?
- Do process reward models need different supervision strategies by domain?
- What makes process-level supervision better than outcome-only reward signals?
- Why do reward models learn surface-level shortcuts instead of genuine quality assessment?
67 specific questions
- Can reward models distinguish between personal preference and community consensus?
- Can behavioral training distinguish reward-seeking from genuine goal alignment?
- What makes reward models fundamentally different from policy discriminators?
- How do reward models as policy discriminators differ from labeled preferences?
- How can reward-seeking remain hidden when graders reward the intended behavior?
- When do reward-seeking and intended behavior make identical predictions?
- Does reward-seeking intensify with situational awareness and RL scale?
70 specific questions
- Can model confidence signals replace explicit external reward functions?
- Do spurious rewards activate reasoning without teaching new skills?
- Why does binary reward forcing degrade model calibration?
- Why do spurious reward signals improve reasoning for some pretrained models?
- What makes user-decision rewards better than model-confidence rewards?
- What happens when confident wrong answers become more rewarded than uncertain correct ones?
- Can structured rewards still teach models when spurious rewards also work?
37 specific questions
- How does reward hacking during training lead to emergent misalignment behaviors?
- Can inoculation prompting prevent emergent misalignment after reward hacking in production RL?
- What mechanism drives emergent misalignment in reward-hacked models instead?
- Can inoculation prompting reduce alignment faking by reframing reward hacking as acceptable?
- Does reward-seeking mediate emergent misalignment after reward hacking?
- Does alignment-faking reasoning emerge unprompted when reward hacking occurs in production systems?
- Does reward hacking in RL training directly cause alignment faking behavior?