Train a model on far more text per parameter than the standard rule says, and quality just keeps climbing, no ceiling in sight — but is that worth the extra cost?
How do pretraining token counts affect downstream performance beyond typical ratios?
This explores what happens when you train a model on far more text per parameter than the usual rule of thumb recommends (Chinchilla's roughly 20 tokens per parameter): does quality keep rising, level off, or affect what the model can do later?
This explores what happens when you keep feeding a model far more text than the standard recipe calls for, which is roughly 20 tokens per parameter under the Chinchilla scaling laws. The most direct answer in the corpus is that quality keeps improving. One study trained 47 models of up to 6B parameters on as many as 10,000 tokens per parameter, about 500 times the usual ratio, and found no point where gains stopped. Both loss and downstream benchmark scores kept climbing Does model quality plateau when training on extreme token ratios?. The catch is that gains slowed faster than Chinchilla-style formulas predict. Overtraining a small model still pays off, but each extra token buys less than the standard curves suggest. This matters in practice: if you want a small model that's cheap to run, the question isn't whether more data helps but how much you're willing to pay for slowing returns.
The part you might not expect is that extra pretraining tokens seem to matter beyond pretraining itself. In a controlled chess setup, pretraining loss predicted how well a model would do after reinforcement learning, and the speed of RL improvement rose roughly in line with the number of pretraining tokens Does pretraining loss predict performance after reinforcement learning?. Pretraining data doesn't just set a starting score. It sets how quickly the model can be improved later. The same study suggests that as total budgets grow, more of the compute should go to RL. That makes the token-ratio question part of a larger budgeting decision across pretraining and post-training, not a question about pretraining alone.
What's in those tokens also shapes what comes after. RL post-training tends to latch onto one format the model saw during pretraining and amplify it within the first epoch, while the other formats fade Does RL training collapse format diversity in pretrained models?. Which format wins depends on model scale. So the makeup of the pretraining mix quietly decides which behaviors RL will later strengthen, and that is mostly invisible if you start from a closed model.
The corpus also suggests that raw token count is a rough measure. Adding LLM-generated reasoning traces to pretraining data improved data efficiency about threefold for 3B models, and harder passages naturally got longer traces Can training data augmentation match test-time compute scaling benefits?. A theoretical result argues that having models predict their own internal representations, instead of the next token, can learn layered structure with exponentially fewer samples Why is predicting latents more sample-efficient than tokens?. Byte-level models that spend more compute on unpredictable stretches of text match tokenized models at 8B parameters Can byte-level models match tokenized performance with better efficiency?. Together these point to another route: instead of adding tokens, make each token count for more.
One honest limit: the corpus has a single direct study of extreme token ratios, and its models top out at 6B parameters. Whether the no-saturation result holds for frontier-size models is still open here.
Sources 6 notes
Training 47 models up to 6B parameters on up to 10,000 tokens per parameter found sustained loss and downstream improvements with no evidence of saturation. However, improvements slow more than Chinchilla-derived scaling laws predict.
In a controlled chess testbed, models pretrained from 5M to 1B parameters show a joint scaling law where pretraining loss predicts final RL performance, and RL improvement rate scales roughly linearly with pretraining tokens. This suggests optimal compute allocation shifts toward RL as total budgets grow.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Augmenting pretraining data with LLM-generated reasoning traces improves data efficiency 3x and reasoning benchmark performance 10%+ for 3B models. Harder tokens automatically receive longer traces, creating a natural compute-allocation mechanism analogous to test-time scaling.
A formal sample-complexity analysis proves latent-level self-supervision (data2vec/JEPA style) recovers compositional structure with samples constant in hierarchy depth, while token-level learning requires exponential samples—because same-level latents are far more correlated than raw tokens.
Show all 6 sources
The Byte Latent Transformer (BLT) dynamically segments bytes into patches based on next-byte entropy, allocating more compute to high-entropy regions and less to predictable ones. At 8B parameters, BLT matches tokenized baselines while reducing inference cost and improving robustness to typos and cross-lingual transfer.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Understanding Reasoning from Pretraining to Post-Training
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
- The Art of Scaling Reinforcement Learning Compute for LLMs
- NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
- Sharpening Tax in Post-Training
- A Primer in Post-Training Reasoning Data: What We Know About How It Works
- TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters