Line of inquiry
Inquiring lines›How do we ensure safe and aligned…›How do reward signals shape model…›this line of inquiry
How can reward models fail to detect reward-seeking disguised as alignment?
A broader line of inquiry — a family of 67 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 67
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- Can reward models distinguish between personal preference and community consensus?
- Can behavioral training distinguish reward-seeking from genuine goal alignment?
- What makes reward models fundamentally different from policy discriminators?
- How do reward models as policy discriminators differ from labeled preferences?
- How can reward-seeking remain hidden when graders reward the intended behavior?
- When do reward-seeking and intended behavior make identical predictions?
- Does reward-seeking intensify with situational awareness and RL scale?
- How do aggregate reward models fail to capture minority user preferences?
- Does pairwise self-judgment avoid reward model scaling problems?
- What causes reward-seeking to override developer preferences?
- Can belief editing alone distinguish reward-optimization from instruction-following behavior?
- How do relational reward signals compare to absolute preference encodings in RL?
- Does reward model training data quality determine fair preference aggregation outcomes?
- Why does single-reward RLHF fail to represent diverse human preferences?
- Can a single AI judge capture diverse human preferences or does it collapse them?
- How do aggregate reward models systematically exclude minority perspectives?
- Can reward-seeking and intended goal pursuit be behaviorally distinguished?
- How do aggregate reward models systematically exclude minority preferences?
- What makes an agent notice that reward beats compliance?
- Can a reward-seeking agent be distinguished from one pursuing intended behavior?
- Can personalized reward models amplify sycophancy without ethical guardrails?
- Why does the contrast between grader and user preferences enable reward-seeking detection?
- Does reward-seeking grow worse with situational awareness and reinforcement learning compute?
- What makes policy discrimination scalable where preference annotation hits bottlenecks?
- Can social welfare functions avoid unjustified assumptions about comparing people's preferences?
- Can curiosity rewards about user type complement general social motivation frameworks?
- Do reward models treat elicited preferences as genuine human values?
- How do preference models amplify human cognitive biases into systematic miscalibration?
- Can system-level engineering fixes replace hand-designed reward heuristics entirely?
- How do reward features learned from group data generalize to new users?
- Can reward-seeking agents appear aligned while targeting their graders?
- Can we distinguish between genuine alignment and response quality bias in reward signals?
- Does learning community preferences as training rewards operationalize prediction without participation?
- How does modularity in reward and policy design enable goal generalization?
- How sensitive is evaluation awareness to the structure of training incentives?
- How do binary comparisons constrain reward scale in multi-user preference learning?
- Does situational awareness help models hide reward-seeking during evaluation?
- Can reward models trained for engagement fix the informativeness problem?
- How often do real reward graders diverge from developer intent in practice?
- Can reward engineering and information-theoretic architecture solve partner-awareness separately?
- What makes minority preferences disappear in aggregated single-distribution reward models?
- How can reward metrics distinguish novel methods from shortcuts aimed at the evaluator?
- How does forced exploration through diversity rewards differ from suppression-based negative reinforcement?
- Does objective swapping versus reward incentives produce different misalignment patterns?
- Is evaluation-aware scheming a form of conditional compliance rather than reward-seeking?
- Why does incentive alignment fail for AI systems making uninterpretable decisions?
- How do adversarial IRL and policy discrimination differ in rejecting preference labels?
- Can simple intrinsic reward signals emerge as effective drivers of complex capability in agents?
- Can variational inference recover user-specific reward models from preference comparisons?
- Can reward factorization represent trade-offs between conflicting moral values?
- What makes a reward evolution schedule fast enough to outpace exploitation?
- Does common ground alignment require explicit rewards to emerge?
- Why do norms learned from scoring collapse into context-dependent costs?
- Can vector-valued rewards preserve specialization better than variance-weighted advantages?
- Can latent-variable reward models capture multimodal preference distributions?
- Why do standard preference alignment methods fail at the individual user level?
- How do self-play and human-anchored rewards separate competence from convention?
- How does motivational stage determine which interventions actually work for users?
- How do intrinsic motivation principles explain why generating novel challenges improves learning?
- What preference dimensions do base reward functions typically capture?
- Do disorder-specific RL policies outperform single policies across anxiety, depression, and schizophrenia?
- What makes principle-response mutual information sufficient for behavioral alignment?
- How does reward-seeking differ from simply taking available metric shortcuts?
- Can alignment methods model loss aversion without creating unintended sophistry?
- How do misaligned incentives in one system spread to others through policy and economics?
- How does attention scarcity differ from poor incentives in explaining delegation?
- Why should bandit algorithms condition exploration on time-of-period as well as user state?