Why can't we easily adapt reinforcement learning to diffusion language models?
Autoregressive models enable efficient RL post-training through factorizable log-probabilities, but diffusion models generate tokens in parallel non-sequential order. What makes likelihood computation intractable in diffusion, and can we work around it?
Scalability stems from transformers, data, and Fisher consistency, not autoregressive generation alone Two techniques enable vanilla PPO to match GRPO and DAPO RLVR improves sampling efficiency, not reasoning capability boundaries Parallel architectures cannot solve inherently sequential problems RL post-training optimizes reasoning deployment timing, not reasoning capability itself
Can outcome-based rewards fully replace per-step likelihood in diffusion RL training? Why do autoregressive models fail at controlling syntactic structure and semantic content? Can diffusion models condition on right context natively without special training for infilling? How can diffusion models predict future tokens without completing prior blocks? What makes asymmetric distillation effective for converting pretrained diffusion models? Why do hybrid paradigms outperform pure autoregressive or pure diffusion approaches? What makes diffusion sampling preserve multiple optimal solutions better than alternatives? What structural differences between diffusion and autoregressive models enable bidirectional prompting? Do diffusion language models learn differently than autoregressive models? Can diffusion language models match autoregressive inference speed in practice? Can diffusion models perform infilling and reverse generation as naturally as forward generation? Why is reinforcement learning harder to apply to diffusion language models?See all 15 inquiring lines on this note →