Does adding more training data always help an AI model, or can too much of the wrong data make it worse?

This explores when more training data stops paying off, and what decides whether the next batch helps, does nothing, or does harm. The corpus gives a clear answer: the amount of data matters less than how well the data fits the model learning from it. A pile of examples that is larger but badly matched can leave a model worse off than a small, well-chosen slice.

The most surprising evidence is that mixed datasets often contain examples that actively work against the skill you care about. In one study, picking just 5% of an instruction dataset, chosen because its examples push the model in the same direction as the target task, beat training on all of it. The other examples were not just extra weight. They pulled the model's reasoning style away from what the task needed Can we train better models on less data?. Image classification shows the same pattern from a different angle. Removing the easy, redundant examples let half of a dataset be dropped with no loss in accuracy, and returns improved much faster than the usual 'add more data, get slowly diminishing gains' curve predicts Can we prune training data without hurting model performance?.

Difficulty cuts both ways. Examples that are too easy teach nothing new, but examples that are too hard can be worse than useless. When models are trained with reinforcement learning on problems they almost never solve, the rare lucky successes get heavily rewarded. The model then learns shortcuts like repeating answers or skipping steps, and these habits damage skills it already had Do overly hard RLVR samples actually harm model capabilities?. One response is to let the model set its own difficulty level. In self-play setups, one copy of the model writes problems sized to what the other copy can almost solve, which produces a curriculum that keeps pace with the learner and needs no outside data Can language models improve themselves without any external training data?.

'Fit' also covers style as well as difficulty. Data that a stronger teacher model has polished can lower a student model's performance when the improvements go beyond what the student is ready to absorb. The student does better when it keeps only the refinements that match its own patterns Does teacher-refined data always improve student model performance?. Taking this further, models sometimes learn more from training data they rewrote themselves than from data a stronger model wrote for them. Self-generated data raised question-answering accuracy from 33.5% to 47%, likely because the model puts information into a form it can actually use Does self-generated training data improve model learning?.

Finally, there is a ceiling where the question changes. Once one model's pretraining has saturated, extra compute does more good spent on training a diverse group of models and combining their predictions than on feeding the same model more passes over the data Should extra compute refine one model or build many?. The takeaway you might not have expected: 'more data' is the wrong unit to think in. What matters is whether each example sits at the edge of what this particular model can learn. Once a single model can't move further, the next gain may come from adding more models, not more data.