Do larger models actually learn simpler functions?

A long-suspected but hard-to-demonstrate idea in deep learning is that large neural networks learn functions far simpler than their parameter counts could express. The obstacle was measurement: every prior compression scheme inflated its estimate with quantities unrelated to learning, so scaling a model always looked like it added complexity even if the learned function stayed simple. Requential coding — whose code length is independent of parameter count and data entropy — removes that inflation, and once you can measure honestly, a striking result appears. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite having more parameters, and the compression advantage grows with scale.

This is direct evidence for the simplicity-bias story and reframes what scaling buys. The naive reading of scaling laws is that capability comes from added expressive capacity. But this result says the opposite can hold at fixed performance: the bigger network is representing something simpler, not more elaborate. That rhymes with — and supplies a compression-side measurement for — the claim that Why do larger models learn rare tasks better?: in both stories scale does not mean "more complex function," it means a cleaner, less-interfering, more compressible one. It also tightens the vault's compression thread, since Can text-trained models compress images better than specialized tools? — here compressed model size, not compressed data, becomes the probe. The practical upshot: model size is a poor proxy for learned complexity, and a bigger model can be the more parsimonious explanation of the data.