Why does adjusted compression performance degrade as models scale larger?
This explores why, once you count the cost of the model's own parameters as part of the compressed message, bigger models start compressing worse rather than better — the two-part code view of language-modeling-as-compression.
This question reads against the well-known framing that language modeling *is* compression: a model that predicts the next token well can drive an arithmetic coder to squeeze text (or even images and audio) into very few bits. The catch is what "adjusted" means. Raw compression keeps improving with scale, but *adjusted* compression charges you for transmitting the compressor itself — model plus encoded data, the classic minimum-description-length account. Under that accounting, a larger model has to justify every extra parameter with a matching reduction in the data's encoded size, and past a point it can't: the parameters grow faster than the marginal predictive gain, so the total message gets longer even as per-token prediction tightens. The corpus's anchor here is the result that text-trained Chinchilla models can out-compress PNG and FLAC purely through in-context adaptation Can text-trained models compress images better than specialized tools? — but that win is about *in-context* conditioning, not parameter count, which is exactly why the adjusted curve and the raw curve part ways.
What's quietly interesting is that the corpus keeps surfacing the same shape from unrelated directions: capability that flattens or even reverses with scale once you account for cost. LLMs plateau around 55–60% on genuine constraint-satisfaction tasks regardless of parameter count Do larger language models solve constrained optimization better?, and instruction-following degrades in predictable patterns that scale doesn't rescue How does instruction density affect model performance?. The adjusted-compression dip is the information-theoretic cousin of these ceilings — evidence that raw size buys less than the scaling-law story implies once you measure the right quantity.
There's also a representational reason bigger isn't freely better. LLMs already compress *aggressively* relative to humans, capturing broad category structure while discarding the fine-grained, situation-specific distinctions people preserve Do LLMs compress concepts more aggressively than humans do?. A model tuned to maximize statistical compression is, in a sense, already at the efficient frontier for the bulk of the distribution; extra parameters mostly chase the long tail, where each marginal bit saved costs disproportionately many parameters to store. That's the same economics that makes the adjusted curve turn.
The corpus also hints at where the gains have migrated. If parameters are an expensive way to buy the last bits, then architecture and inference compute become the cheaper levers. Depth-over-width at small scale composes abstract concepts through layers instead of paying for width Does depth matter more than width for tiny language models?, retrieval distributions can be folded into a small parametric decoder that preserves long-tail facts without a giant datastore Can retrieval knowledge compress into a tiny parametric model?, and test-time compute can substitute for parameter scaling outright on hard prompts Can inference compute replace scaling up model size?. Each is a way of getting compression-equivalent capability without paying the parameter tax that drags the adjusted curve down.
So the short answer: adjusted compression degrades at scale not because big models predict worse, but because you're now billing them for their own size, and the per-parameter return on prediction shrinks faster than the bill grows. The deeper takeaway — the thing you didn't know you wanted to know — is that this is the same wall showing up as constraint-satisfaction plateaus, instruction-following decay, and the recent pivot toward depth, retrieval distillation, and inference-time compute as the places where capability is now actually cheaper to buy.
Sources 7 notes
Chinchilla models trained exclusively on text achieve better compression rates on images and audio than FLAC and PNG by using their context window to adapt as task-specific compressors. This demonstrates that generalization operates through compression, not specialization.
Across constrained-optimization tasks, LLMs converge to ~55–60% constraint satisfaction independent of architecture, parameter count, or training regime. Reasoning models do not systematically outperform standard models, suggesting a fundamental ceiling rather than a scaling gap.
IFScale benchmark shows three degradation patterns: linear (small models), exponential (mid-range), and threshold decay (reasoning models maintain ~150 instructions then fail steeply). Even best models reach only 68% accuracy at maximum density.
Using Rate-Distortion Theory on cognitive datasets, LLMs capture broad category structure but lose fine-grained distinctions humans preserve. LLMs maximize compression efficiency; humans trade compression for contextual meaning that enables situated action.
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.
Memory Decoder successfully compresses kNN-LM retrieval distributions into a small transformer that plugs into any LLM via output interpolation. It preserves long-tail factual knowledge while maintaining semantic coherence, reducing perplexity by 6.17 points across domains.
Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.