How does the compression view extend from trained models to training objectives?
This explores a shift in how researchers use 'compression' to describe language models — moving it from a description of what a finished model *does* (a trained net behaves like a compressor) to a principle that defines what the *training process should optimize for* in the first place.
This explores a shift in how researchers use 'compression' to describe language models — from a property of the finished model to a principle for training itself. The starting point is the now-familiar claim that a trained language model simply *is* a compressor: because predicting the next token well means assigning high probability to what actually comes next, a good model can losslessly compress data better than purpose-built tools. The striking demonstration is that text-only models can compress images and audio more efficiently than FLAC or PNG, using their context window to adapt on the fly Can text-trained models compress images better than specialized tools?. The lesson there is that generalization and compression are the same thing seen from two angles — the model isn't specialized, it's just very good at squeezing redundancy out of any stream.
The extension the question asks about is what happens when you stop treating compression as a side-effect and make it the *objective*. If a good model is a good compressor, then the optimal way to train one should fall out of a pure compression goal — and it does: deriving training from a lossless-compression target yields a 'Learning Law' in which every example contributes equally in the ideal learning process, and this improves the *coefficients* of scaling laws rather than just shifting constants Does optimal language model learning maximize data compression?. That's the conceptual move: compression goes from describing the artifact to specifying the dynamics of how you should get there.
Once compression is the goal rather than the byproduct, its costs come into focus. Models compress *aggressively* — they capture broad category structure but discard the fine-grained, context-dependent distinctions humans keep around because those distinctions support acting in specific situations Do LLMs compress concepts more aggressively than humans do?. So an objective that maximizes compression efficiency is not value-neutral; it trades away exactly the adaptive nuance that human cognition refuses to throw out. And there's a floor beneath all of it: text is itself a lossy compression of reality, stripping out physics, geometry, and causal dynamics, so a compression objective over text inherits limits baked into the medium before training even begins Are text-only language models fundamentally limited by abstraction?.
The view also shows up in how models *internally* manage information, which blurs the line between trained artifact and ongoing process even further. Representations become dense for familiar data and sparse for unfamiliar inputs as a consequence of training exposure Is representational sparsity learned or intrinsic to neural networks?, and models sparsify their activations adaptively when tasks go out-of-distribution — a kind of just-in-time compression that stabilizes performance rather than signaling failure Do language models sparsify their activations under difficult tasks?. Compression here isn't a global training target at all but a moment-to-moment behavior the network learns to deploy.
The most concrete payoff of taking the objective seriously is that you can deliberately compress *knowledge* into smaller forms: an expensive retrieval distribution (kNN-LM) can be distilled into a small parametric decoder that plugs into any model, preserving long-tail facts without runtime search Can retrieval knowledge compress into a tiny parametric model?. Read together, these notes trace compression across three levels — what a model is, how you train it, and how it behaves under pressure — and the interesting tension is that the same principle that explains why these models generalize so well also explains why they quietly discard the situated detail that humans, and reality, actually run on.
Sources 7 notes
Chinchilla models trained exclusively on text achieve better compression rates on images and audio than FLAC and PNG by using their context window to adapt as task-specific compressors. This demonstrates that generalization operates through compression, not specialization.
Research shows that optimal LM training can be derived from a lossless compression objective, yielding a Learning Law where all examples contribute equally in the optimal process. This approach improves scaling law coefficients, not just constants.
Using Rate-Distortion Theory on cognitive datasets, LLMs capture broad category structure but lose fine-grained distinctions humans preserve. LLMs maximize compression efficiency; humans trade compression for contextual meaning that enables situated action.
Text strips the physics, geometry, and causality present in reality, forcing language models to manipulate symbols without grounding in their source dynamics. This creates predictable failure modes in physical, geometric, and causal reasoning that multimodal training could address.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.
As task difficulty increases, LLM hidden states become substantially sparser in a localized, systematic way that correlates with task unfamiliarity and reasoning load. This sparsification acts as a selective filter stabilizing performance under OOD shift rather than a failure mode.
Memory Decoder successfully compresses kNN-LM retrieval distributions into a small transformer that plugs into any LLM via output interpolation. It preserves long-tail factual knowledge while maintaining semantic coherence, reducing perplexity by 6.17 points across domains.