Theme of inquiry
What training approaches and control signals most effectively guide model learning?
A question within its area, explored through 8 lines of inquiry below — each a family of specific questions the research asks.
17 specific questions
- Can fitted strategy models distinguish genuine mental-state representation from learned game policies?
- Can puzzle performance prove a player understands a concept versus just applying it?
- Does spectral analysis of AlphaZero games reveal knowledge humans cannot yet explain?
- How do concept vectors in neural networks relate to human-language chess explanations?
- How well do metagaming latents transfer across different evaluation tasks?
- How does a challenger's escalating difficulty function as curriculum?
- How do self-play and human-anchored rewards separate competence from convention?
44 specific questions
- Does policy entropy collapse limit how many iterations of reasoning training work?
- What causes policy entropy collapse in reasoning-focused reinforcement learning?
- What happens to model reasoning when policy entropy collapses during RL?
- Why does policy entropy collapse limit reasoning and dialogue RL scaling?
- Does policy entropy collapse prevent inference-time search from finding solutions?
- Why does policy entropy collapse when scaling RL for reasoning?
- Does policy entropy collapse represent the main bottleneck in reasoning-focused RL scaling?
65 specific questions
- Can distillation and reward optimization happen in a single training loop?
- What makes advantage shaping more stable than reward shaping for tool training?
- How do loss functions simultaneously shape both learning and decision quality?
- Can distillation methods extract directional guidance that scalar RL cannot access?
- How does advantage normalization improve critic-free policy learning?
- When does knowledge distillation produce student models superior to teachers?
- How do reward model ensembles improve robustness to miscalibration?
26 specific questions
- Does iterative DPO generalize identically to online RL on the same benchmark tasks?
- Does iterative DPO preserve the same misalignment mechanisms as online reinforcement learning?
- Does iterative DPO generalize like online reinforcement learning?
- How does iterative DPO compare to full RLVR for studying emergent misalignment?
- Can iterative DPO serve as a tractable proxy for studying on-policy misalignment?
- What misaligned behaviors does iterative DPO produce compared to online RL?
- What theoretical argument connects iterative DPO dynamics to online RL learning?
70 specific questions
- When does RL discover genuinely novel reasoning strategies versus timing optimization?
- Does reinforcement learning preserve reasoning quality better than supervised fine-tuning?
- How does RL compress reasoning path diversity during training?
- Why does prolonged RL discover strategies absent from any base model sample?
- Does reinforcement learning learn optimal per-turn reasoning discipline?
- Can single-problem fine-tuning match full RL pipeline reasoning gains?
- Can reinforcement learning fix the reasoning gaps that supervised fine-tuning misses?
67 specific questions
- Can RL create new reasoning primitives that pretraining never established?
- Can reinforcement learning add new capabilities or only remove inaccurate knowledge?
- Does RL teach models new reasoning or just better timing?
- Does RL refine existing knowledge or discover entirely new capabilities?
- Does RL teach models when to use reasoning or how to reason?
- Does RL amplify existing reasoning or create genuinely new computational strategies?
- Does RL primarily teach when to use reasoning or how to reason?
47 specific questions
- Why do current RLVR methods fail to expand reasoning capability beyond base model boundaries?
- Can RLVR expand a model's reasoning capabilities beyond its training ceiling?
- Does careful reward engineering matter if pretraining determines RLVR effectiveness?
- How do reward signals in RLVR interact with pretraining biases?
- Why do spurious rewards work nearly as well as correct ones?
- Does RLVR expand model capability or reorganize existing capability?
- Are RLVR models worse than non-reasoning models for subjective annotation?
48 specific questions
- Can AI systems improve themselves through recursive self-improvement loops?
- Do diminishing returns prevent recursive self-improvement in AI systems?
- What distinguishes bounded self-refinement from open-ended recursive self-improvement empirically?
- How does bounded self-refinement differ from open-ended recursive self-improvement?
- What distinguishes bounded self-refinement from open-ended recursive self-improvement in AI systems?
- How does recursive self-improvement differ from updating just the policy?
- How fast is recursive self-improvement advancing in current AI systems?