When you grade an AI by how well it compresses data, random noise in that data quietly inflates its score.
How does data entropy inflate compression estimates in prequential coding?
This explores why prequential coding — the standard way of measuring how well a model 'compresses' its training data — can overstate a model's true simplicity, and how the raw unpredictability (entropy) of the data itself gets baked into that estimate.
This explores why prequential coding overstates model simplicity: the trick is that it charges the model for every bit of the data, including the irreducible randomness that no model could ever remove. Prequential coding measures learning as compression — you feed data through a model one example at a time and add up the surprise (code length) it assigns. The intuition is elegant, and the corpus leans on it hard: optimal language-model training can be derived directly from a lossless-compression objective Does optimal language model learning maximize data compression?, and models trained only on text turn out to compress images and audio better than PNG or FLAC by adapting on the fly Can text-trained models compress images better than specialized tools?. Compression really is a good proxy for generalization.
But the proxy has a leak. A prequential code length mixes two very different things: what the model genuinely learned, and how noisy the data was to begin with. High-entropy data — text full of genuinely unpredictable tokens — inflates the code length no matter how good the model is, because those bits are incompressible in principle. Bigger models also tend to look like they 'compress' more simply, so parameter count sneaks in as a second confound. The result: two models with identical learned content can post wildly different compression scores just because their data or their size differed.
The corpus's sharpest answer is a coding scheme that strips both confounds out. Instead of coding all the data, it codes only where a student model disagrees with its teacher — the disagreements are the genuine learned content, and everything both already agree on (including the irreducible entropy) gets skipped Can we measure what a model truly learned?. This yields compressed sizes independent of both parameter count and data entropy, and codes that are orders of magnitude shorter than prequential methods. The gap between the two numbers *is* the inflation — it's the entropy and size you were accidentally paying for.
There's a deeper reason this matters, visible if you look laterally. Entropy is not evenly distributed across a sequence: only about 20% of tokens are genuinely high-entropy 'forking points' where real decisions happen, and the rest are low-surprise filler Do high-entropy tokens drive reasoning model improvements?. Models even carry an internal signal of input surprise that modulates their own confidence Why do models produce less uncertain outputs on their own text?. So a measurement that sums surprise indiscriminately is dominated by the noisy tail, not the learned structure — which is exactly why disagreement-only coding, by focusing on the forking points where teacher and student diverge, recovers a cleaner picture of what was actually learned.
Sources 5 notes
A teacher-student disagreement coding scheme produces compressed sizes independent of parameter count and data entropy, yielding orders of magnitude shorter codes than prequential methods. This isolates genuine learned content from confounding factors.
Research shows that optimal LM training can be derived from a lossless compression objective, yielding a Learning Law where all examples contribute equally in the optimal process. This approach improves scaling law coefficients, not just constants.
Chinchilla models trained exclusively on text achieve better compression rates on images and audio than FLAC and PNG by using their context window to adapt as task-specific compressors. This demonstrates that generalization operates through compression, not specialization.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Post-trained models produce 3-4x lower output entropy on their own generations, driven by an internal representation of input surprise that causally modulates confidence. This implicit self-recognition signal appears without being verbalized, encoded directly in the output distribution.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Language Modeling is Compression
- Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
- From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
- Towards Optimal Learning of Language Models
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- How much do language models memorize?
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
- From Simulation to Enaction: Post-trained Language Models Recognize and React to their own Generations