What determines whether training improves or degrades model performance?

27 specific questions

• When does knowledge distillation produce student models superior to teachers?
• How can distillation preserve uncertainty expression instead of optimizing it away?
• Why does distillation transfer reasoning patterns with few examples?
• Does teacher scale matter for on-policy distillation success?
• What makes policy self-distillation more effective than external teacher distillation?
• Why does unsupervised self-distillation lose its advantage in thinking model mode?
• How does distilling only inconsistent rollouts compare to distilling all generations?

79 specific questions

• Can RL format selection explain performance gains attributed to algorithmic improvements?
• Why do smaller and larger models converge on different output formats?
• How do surface statistical regularities enable correct outputs while degrading robustness?
• Can scaling up contradictory training data overcome unpredictable override effects?
• Can looped models be designed to avoid oscillation in later iterations?
• How do learning dynamics on one example shift predictions on other responses?
• Can feedback loop frequency harm performance on finite task sets?

63 specific questions

• Why does self-correction during generation produce reliable labels without exemplars?
• Why does self-consistency fail as a proxy reward for correctness?
• How does error avalanching compound failures in self-training iterations?
• Can self-consistency checks fully prevent error avalanching in self-training loops?
• What role does the self-consistency threshold play in preventing error reinforcement?
• Can applicability conditions and veto rules make self-training stable across substrates?
• How should training incorporate external critique versus encouraging self-correction?

68 specific questions

• How does scaling and training data enable compositional behavior without symbolic mechanisms?
• Does the linear representation hypothesis reflect networks or reflect our analysis tools?
• Can identical model performance mask fundamentally broken internal representations?
• Can scaling alone create compositional generalization without explicit binding mechanisms?
• Where do neural networks still fail at compositional generalization despite scaling?
• Why does gradient descent discover compositional structure without explicit pressure?
• Does latent density emerge during pretraining from training data familiarity?

53 specific questions

• Do task-relevant parameter changes naturally concentrate in sparse regions?
• Why do longer sequences tolerate higher sparsity than shorter ones?
• Why does representation sparsity reliably indicate task difficulty for language models?
• How do sparse weight patterns affect model interpretability?
• Should production deployments scale budgets with sequence length for sparse models?
• Could activation sparsity signal task difficulty and guide routing decisions?
• Does static per-token sparsity repeat the fixed-budget mistake at short sequences?

49 specific questions

• Does selecting examples from multiple complexity levels outperform selecting only high-quality examples?
• How does difficulty-adaptive curriculum learning change which samples get selected during training?
• Why do easy training examples contribute less to model generalization than hard ones?
• Does teacher-style refinement of training data transfer equally to all student model distributions?
• Does the productive difficulty band ever stabilize during training?
• Why do explicit quality criteria outperform learning quality from examples alone?
• Does partial trace guidance work better than curriculum learning for hard problems?

61 specific questions

• Can recurrent transformers learn genuinely new computations beyond inference stages?
• Can looping enable reasoning capabilities that fixed-depth transformers fundamentally cannot achieve?
• Why does looping computation outperform adding more transformer layers?
• Can bounded-depth transformers solve inherently sequential problems?
• Can transformers reason beyond fixed architectural depth limits?
• Why do standard transformers fail to encode recursive structure in their hidden states?
• Can latent recurrence achieve the depth that standard transformers cannot?

33 specific questions

• At what point does output quality outweigh diversity value in synthetic data tasks?
• How do quality, diversity, and complexity create different effects on downstream model performance?
• How does diversity loss in synthetic data mirror tail distribution disappearance?
• Can synthetic data diversity preserve the transcendence effect or does it collapse?
• How does the ratio of synthetic to real training data affect model collapse?
• What conditions make training diversity better than individual expert quality?
• Does self-generated training data reduce a model's capability diversity?

53 specific questions

• How does training data structure shape reasoning strategy more than domain content?
• How much does training composition affect syntactic versus reasoning performance?
• How do training data distributions constrain what language models can accurately know?
• How much task-similar finetuning data does test-time training actually need?
• Does knowledge structure matter more than knowledge volume for model training?
• What makes training data quality more important than quantity for reasoning?
• Can selecting the right data subset outperform training on everything?

32 specific questions

• Do diffusion language models learn differently than autoregressive models?
• How can diffusion models predict future tokens without completing prior blocks?
• What structural differences between diffusion and autoregressive models enable bidirectional prompting?
• Can gradient-based control reach properties that autoregressive methods cannot?
• Can diffusion language models match autoregressive inference speed in practice?
• How does selective looping in diffusion models differ from recurrence in autoregressive architectures?
• Can diffusion models condition on right context natively without special training for infilling?