Does model quality plateau when training on extreme token ratios?

To test whether a smaller-and-longer-trained model can really match a Chinchilla-optimal one — the recipe recommended once inference cost is counted, per Should LLM teams train smaller models for longer? — the excerpt trains "47 models ranging from 150M to 6B parameters, on various data budgets from 10 to 10,000 tokens per parameter" and reports: "model quality continues to improve as we scale token ratios. We do not find evidence of a 'saturation point,' beyond which models do not improve, even with additional data." The same holds on downstream evaluation: "the Gauntlet Average improves as we increase tokens per parameter — again, we find no evidence of a 'saturation' point." This directly contradicts what the excerpt calls "existing literature['s] claim[that] there exists a critical model size, below which we cannot match a Chinchilla-optimal model's quality," attributed to De Vries (2023): "we find no evidence to support the critical size hypothesis."

The excerpt separately ablates the Chinchilla curve-fitting procedure itself. The original Chinchilla coefficients were fit from "data collected at typical token/parameter ratios" (≤~100 tokens/parameter); refitting those coefficients while including the new extreme-ratio data shows "as we use more extreme data in our fitting procedure, our scaling curves become flatter," meaning "developing scaling laws only from data collected at typical token/parameter ratios overestimates the impact of additional tokens at these extreme ranges." In other words, the two findings compound in opposite directions: quality keeps rising with more tokens per parameter, but more slowly than Chinchilla-derived coefficients would predict — "although models continue to learn at extreme training durations, they do so more slowly than scaling laws predict."

Within scaling-law research, this sits alongside work that finds exceptions to plain power-law token scaling from a different angle: Can we prune training data without hurting model performance? gets past power-law scaling by selecting which tokens to train on (keeping only the hardest examples), while this excerpt gets continued gains from sheer token quantity at extreme ratios without any selection — the two are complementary levers (which data vs. how much data) rather than competing explanations. It also extends the training-token side of what Does pretraining loss predict performance after reinforcement learning? establishes for a narrower domain: both find pretraining token count keeps paying off well past typical ratios, though the chess result is about a constructed testbed and RL transfer, not general-purpose loss and downstream accuracy.

The excerpt is explicit about where its evidence stops: only the 150M model was tested up to the full 10,000 tokens/parameter; larger models were tested "only... up to a maximum of 1,000 tokens per parameter due to resource constraints," and the paper states "it is possible that beyond 10,000 tokens/parameter, behavior changes" and that "further experiments are needed to see how loss scales beyond this point." With only 47 training runs behind the fitted coefficients (versus Chinchilla's 400+), the paper also flags its own confidence intervals as likely wide, citing Besiroglu et al. (2024)'s finding that Chinchilla's reported intervals were "implausibly narrow." The result supports training considerably longer than Chinchilla recommends when inference cost or deployment memory is tight, but not yet a claim that no critical size exists at any model scale or any token ratio.