Theme of inquiry
How do attention and architecture shape the biases in retrieval?
A question within its area, explored through 8 lines of inquiry below — each a family of specific questions the research asks.
30 specific questions
- Why do outcome-only rewards fail to optimize long-horizon agent behavior?
- Can an agent's internal probabilities serve as value signals across domains?
- Can architectural changes like decoupling intent understanding help overcome next-turn reward limitations?
- Can agents learn to distinguish helpful from misleading interventions?
- Do information gathering and task execution require different incentive structures?
- Can early experience replace external rewards as a learning signal?
- How does next-turn reward optimization contribute to agent passivity?
34 specific questions
- Can preference learning fix the rigid output format problem better than supervised training?
- How does preference-based training compare to supervised fine-tuning for function calling?
- Can preference model training be redesigned to prioritize factual correction over user agreement?
- Can structured natural language feedback outperform scalar rewards in RL?
- How do relational reward signals compare to absolute preference encodings in RL?
- How do loss functions simultaneously shape both learning and decision quality?
- Can distillation and reward optimization happen in a single training loop?
33 specific questions
- How do aggregate reward models fail to capture minority user preferences?
- Can reward models distinguish between personal preference and community consensus?
- How do aggregate reward models systematically exclude minority preferences?
- How do aggregate reward models systematically exclude minority perspectives?
- Why does single-reward RLHF fail to represent diverse human preferences?
- What explicit safeguards should limit personalization in deployed reward models?
- How do reward models as policy discriminators differ from labeled preferences?
38 specific questions
- Can structured rewards still teach models when spurious rewards also work?
- Can models trained with RL on pretraining data avoid reward hacking seen in RLHF?
- Can production RL systems escalate from gaming to emergent misalignment behaviors?
- Can log-probability ratios resist reward hacking better than learned PRM signals?
- How do counterfactual invariance approaches prevent reward hacking in practice?
- Can separating token weighting from query filtering reduce reward hacking?
- Why do spurious rewards work for some models but not others?
29 specific questions
- Can process reward models work on branching reasoning traces with backtracking?
- What does process supervision reveal about step-level reasoning versus outcome rewards?
- Do process reward models need different supervision strategies by domain?
- How can process reward models handle branching and revisiting in reasoning traces?
- What makes process-level supervision better than outcome-only reward signals?
- Can process reward models reason before judging more data efficiently?
- What are the actual limits of sibling comparison versus trained process reward models?
46 specific questions
- Does careful reward engineering matter if pretraining determines RLVR effectiveness?
- How do reward signals in RLVR interact with pretraining biases?
- Why do current RLVR methods fail to expand reasoning capability beyond base model boundaries?
- Why do spurious rewards work nearly as well as correct ones?
- Can RLVR expand a model's reasoning capabilities beyond its training ceiling?
- Does RLVR reward structure create pressure toward traces that look right?
- Does RLVR expand model capability or reorganize existing capability?
63 specific questions
- Can decomposing rewards into prompt-free and prompt-related components fix this blindspot?
- Do spurious rewards activate reasoning without teaching new skills?
- Why do spurious reward signals improve reasoning for some pretrained models?
- How do reward models benefit from extended thinking during evaluation scoring?
- Can random rewards improve reasoning models if pretraining is suitable?
- Why do reward models learn surface-level shortcuts instead of genuine quality assessment?
- Do outcome-only reward signals miss step-level errors that compound later?
14 specific questions
- Can algorithm choice like PPO substitute for recipe-level design decisions?
- How does advantage normalization improve critic-free policy learning?
- Why does scalarization of rewards fail for multi-objective GRPO training?
- Why does gradient discarding limit standard policy clipping?
- Can PPO match GRPO and DAPO with just two techniques?
- Why do zero-advantage rollouts destabilize training beyond just wasting compute?
- Can on-policy optimization variants avoid the probability squeezing problem?