What makes a self-supervised pruning metric work without labels at scale?
This explores why ranking-and-removing training data can work using a signal the model generates about itself — no human labels — and still hold up as datasets grow to ImageNet scale.
This explores how a pruning metric decides which training examples to keep using only signals the model produces about itself, rather than human labels — and why that survives at scale. The anchor result is that ranking examples by difficulty and throwing away the easy, redundant ones beats power-law scaling: on CIFAR-10 you can drop half the data with no accuracy loss, and crucially a *self-supervised* difficulty metric scaled the same trick up to ImageNet Can we prune training data without hurting model performance?. The labels were never the point — what matters is having a reliable ordering of which examples carry information the model hasn't already absorbed.
The deeper question is where a trustworthy signal comes from when nobody hand-labels it. The pattern across the corpus is that self-supervision works when the model's own behavior exposes a structure that correlates with what a label would have told you. Speech models trained without labels end up recovering the causal articulatory physics of how a vocal tract makes sound — and that learned structure predicts downstream performance better than explicit phonetic probing does Do speech models learn language-specific sounds or universal physics?. The lesson transfers to pruning: a self-supervised metric works when the quantity it measures (loss, forgetting, embedding geometry) is a faithful proxy for genuine difficulty, not an artifact of training noise.
That caveat matters, because the corpus also shows self-generated signals can lie. A model can hit perfect accuracy while its internal representations are fractured and disorganized — all the decodable features are present, but the structure underneath is broken in ways standard metrics never see Can models be smart without organized internal structure?. A pruning metric riding on such a signal would happily keep or discard the wrong examples. So 'works without labels' really means 'the unsupervised signal tracks something real' — which is exactly the failure mode to test for.
Pruning isn't limited to whole examples, either. Inside a single reasoning chain, models internally rank tokens by functional importance: greedy likelihood-preserving pruning preferentially keeps symbolic-computation tokens and discards grammar and meta-discourse first — and students trained on these self-pruned chains beat students trained on frontier-model compression Which tokens in reasoning chains actually matter most?. Same principle, finer grain: the model's own likelihood is the label-free metric for what's load-bearing.
The reason this scales rather than collapsing is that label-free signals get cheaper as data gets more abundant, while annotation gets more expensive. That's the same trade behind self-supervised process rewards matching expert-annotated supervision without step labels Can self-supervised process rewards replace human annotation?, and behind learning user intent from unlabeled UI video instead of paired text Can unlabeled UI video teach models what users intend?. A pruning metric is the data-curation member of that family: it works at scale precisely because the signal it needs is something the model already produces for free.
Sources 6 notes
Research shows that ranking training examples by difficulty (EL2N, forgetting, memorization) and removing easy ones beats power-law scaling laws. On CIFAR-10, 50% of data was pruned without accuracy loss, and self-supervised metrics scaled the approach to ImageNet.
Self-supervised speech models learn the language-agnostic physics of how the vocal tract produces acoustics, not language-specific phonetic categories. This explains their multilingual transfer and predicts their downstream task performance better than phonetic probing.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
MetaStone-S1's SPRM achieves o3-mini-level results using dynamic weighting of pseudo-labels instead of human-annotated steps. This eliminates the annotation bottleneck for process supervision, though generalization to fuzzy-outcome domains remains unproven.
UI-JEPA applies JEPA-style predictive masking to screen recordings, learning task-aware temporal representations that an LLM decoder can use to infer intent with minimal paired data. This trades the bottleneck of labeled video for abundant unlabeled streams.