Train a language model on 500x more data than the standard rule — do the gains from more data actually stop?

This explores why piling on far more training data than the Chinchilla rule recommends gives less benefit than expected. The first thing the corpus does is correct the question's premise. The curves don't actually go flat. One study trained 47 models, up to 6B parameters, on as many as 10,000 tokens per parameter, which is roughly 500 times the Chinchilla ratio. Loss and downstream performance kept improving the whole way, with no point where they stopped Does model quality plateau when training on extreme token ratios?. The real finding is quieter: the gains slow down faster than formulas fitted near the Chinchilla ratio predict. So the formula is being stretched beyond the range it was fitted on, and the model isn't hitting a wall.

The corpus doesn't pin down a single cause for that extra slowdown, and it's better to say so plainly than to invent one. What it does offer is a useful way to think about it from a neighboring area. In reinforcement learning, a 400,000-GPU-hour study found that performance follows an S-shaped curve that levels off at a ceiling. The training recipe sets the ceiling, and implementation details only change how fast you get there Does RL training follow predictable scaling curves?. Pretraining has its own version of this. A pure power law fitted over a narrow range can't tell you whether a ceiling is coming, so the curve looks steady until you push it far enough to see it bend. If the pretraining slowdown works the same way, the data mix and setup may matter more at extreme ratios than raw token count does.

A second clue comes from studies that pull apart what each kind of scaling buys. Scaling pretraining mainly adds factual knowledge, stored in the model's lower layers, while scaling fine-tuning mainly improves how helpfully the model behaves Do pretraining and fine-tuning scale independently in language models?. Fine-tuning results also improve more from a bigger base model than from more pretraining data How should finetuning scale with model and data size?. Together these suggest that a small model trained on huge amounts of data starts running out of room. With a fixed number of parameters, each extra token has less new to teach and less space to store it. That fits gains that slow down without ever stopping, though the corpus treats it as a plausible reading rather than a proven mechanism.

The part you may not have expected: the slowdown matters less than it seems, because people train past the Chinchilla ratio for economic reasons, not to squeeze out maximum quality. Once you account for how often a model will be used after training, teams expecting around a billion requests should train smaller models for longer than Chinchilla recommends. The extra training cost is repaid by cheaper use afterward Should LLM teams train smaller models for longer?. Chinchilla optimizes for the best model you can get from a fixed training budget, and almost nobody's real budget looks like that. The 'flattening' is the price of making a model cheap to run, and the 10,000-tokens-per-parameter result says the price is real but still worth paying.