Why does training an AI to reason make it stubbornly repetitive, and why does that, not compute, cap how far it scales?

This explores why reinforcement learning on reasoning models drives the model toward ever more predictable outputs, and why that loss of variety, not a shortage of compute, becomes the wall on scaling. The corpus is stronger on what entropy collapse does and how to slow it than on one root cause. Three notes together still give a usable picture.

Policy entropy is roughly how willing the model still is to try different next steps. Does policy entropy collapse limit reasoning performance in RL? finds that during RL training entropy drains toward zero, and that performance follows a predictable curve, R = -a·exp(H) + b. The model trades entropy for reward. Once the entropy is spent, reward stops improving. So the collapse isn't a late glitch. It's the cost of the early gains, and it sets a ceiling you can forecast in advance. The fixes named there (Clip-Cov, KL-Cov, GPPO) all manage how fast entropy falls, rather than just training longer.

A second note gives a concrete mechanism for a close cousin of this problem. In Why do language models collapse into generic templates?, when the sampled answers to a prompt all earn similar reward, the learning signal pushing toward better answers weakens. Regularization then dominates and pulls the policy toward generic, input-agnostic templates. Selecting high-variance prompts before each update recovers performance. That note measures template collapse, not entropy directly. Treating the two as one phenomenon is my inference. It suggests collapse happens when there's too little differentiating signal for the pull toward sameness to lose against.

The third note shows where the entropy sits. Do high-entropy tokens drive reasoning model improvements? finds that only about 20% of tokens in a reasoning chain are high-entropy forking points, and RLVR mostly adjusts those. Training on just those tokens matches full updates. Combining this with the first note is again my synthesis. Since the entropy that matters is concentrated in a small set of decision points, collapse there means the model stops exploring at exactly the moments where a different reasoning path could branch off.

What the corpus doesn't give is a full causal account of why standard RLVR loses entropy in the first place. It has the empirical law, the interventions, and two adjacent mechanisms (weak reward variance, and concentration at forking tokens). The next step would be to test whether protecting those forking tokens is what the entropy-management methods are really doing.