Can you train a better AI by deleting the boring, already-easy examples instead of adding more data?
Can data pruning by difficulty achieve non-power-law scaling improvements?
This explores whether you can get better results from training by throwing out the easy, redundant examples, and whether that beats the usual pattern where each extra gain costs exponentially more data.
This explores whether you can get better results from training by throwing out the easy, redundant examples, and whether that beats the usual pattern where each extra gain costs exponentially more data. The short answer from the corpus is yes, at least in image classification. Researchers ranked training examples by how hard they were for a model to learn, using signals such as how often the model forgot an example during training. They then removed the easy ones. On CIFAR-10, half the data could be dropped with no loss in accuracy, and the gains improved exponentially rather than along the usual power-law curve. With self-supervised difficulty scores, the method also worked on ImageNet Can we prune training data without hurting model performance?. The underlying idea is that most of a large dataset tells the model things it already knows. Scaling laws look harsh partly because we keep paying for repetition.
There is a catch: 'keep the hard ones' only works up to a point. In reinforcement learning for reasoning models, training on problems that are nearly impossible does real damage. The model rarely succeeds, and when it gets one right by accident, the training treats that lucky guess as a highly valuable example. The model then learns shortcuts like repeating answers or skipping steps, and these habits spread to skills it already had Do overly hard RLVR samples actually harm model capabilities?. So difficulty-based selection has a sweet spot. Removing easy examples cuts redundancy, but pushing too far toward hard ones adds noise the model can't learn from.
It is also harder than it sounds to tell which examples are actually difficult. One study on maze-solving found that how long a model 'thinks' about a problem tracks how close the problem is to its training data, not how hard the problem really is Does longer reasoning actually mean harder problems?. Any pruning method depends on its difficulty measure, and a measure that really captures 'familiar vs. unfamiliar' could quietly throw out the wrong data.
If you look more widely, data pruning is one of several ways the corpus finds to escape the standard scaling curve. Others change the model instead of the data. Looped models reuse the same layers repeatedly and beat larger models on reasoning tasks Can models learn by looping instead of growing larger?. Latent-thought models add a scaling dimension separate from parameter count Can latent thought vectors scale language models beyond parameters?. Small models gain more from depth than from width Does depth matter more than width for tiny language models?, and adding architecture choices to scaling laws produces faster and more accurate models on the same budget Can architecture choices improve inference efficiency without sacrificing accuracy?. These results share a theme: the power law describes what happens when you add more of the same thing. It isn't a law of nature.
The corpus has gaps here. The direct evidence for difficulty pruning comes from vision benchmarks, and nothing in this collection shows the same exponential effect in LLM pretraining. Fine-tuning still appears to follow a power-law-style curve that rewards bigger base models more than more data How should finetuning scale with model and data size?. Some tasks don't improve with scale at all: on constrained-optimization problems, models stall around 55–60% success no matter how big they are Do larger language models solve constrained optimization better?. Better data selection can make scaling cheaper, but it can't fix a ceiling that scaling never moved in the first place.
Sources 9 notes
Research shows that ranking training examples by difficulty (EL2N, forgetting, memorization) and removing easy ones beats power-law scaling laws. On CIFAR-10, 50% of data was pruned without accuracy loss, and self-supervised metrics scaled the approach to ImageNet.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
Latent-Thought Language Models achieve superior sample and parameter efficiency by coupling fast local variational learning with slow global decoder learning. This dual-rate scheme scales few-shot reasoning across both model and latent size, creating independent scaling dimensions beyond traditional parameter scaling.
Show all 9 sources
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
Systematic experiments across 1B–16B models reveal finetuning follows a power-based multiplicative scaling law. Larger base models improve finetuning more than more pretraining data, while increasing PET parameters provides minimal benefit.
Across constrained-optimization tasks, LLMs converge to ~55–60% constraint satisfaction independent of architecture, parameter count, or training regime. Reasoning models do not systematically outperform standard models, suggesting a fundamental ceiling rather than a scaling gap.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- Scaling Laws for Neural Language Models
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
- Hierarchical Reasoning Model
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
- Beyond neural scaling laws: beating power law scaling via data pruning