Can we measure what a model truly learned?

Compression is the operational test of generalization: a model that can encode its training data as a short code has found the regularities that let it generalize, in the spirit of Occam's razor. The trouble is that existing compressors measure the wrong thing. Parameter-based methods like quantization produce codes whose length scales with model size, no matter how little information the parameters actually store. Prequential coding avoids that by coding the training trajectory, but it codes the exact data sequence regardless of how much the model learned — so high-entropy data yields a large code even when the learned function is simple. Both confound "how much the model learned" with quantities unrelated to learning.

Requential coding removes both confounds. A teacher model selects training samples drawn from the student's own distribution, and the code records only those selections — which cost bits only where teacher and student disagree. Where they already agree, nothing is transmitted. The resulting code length depends on neither parameter count nor data entropy, and comes out orders of magnitude shorter than the prequential version, with the gap widening at scale. This is a sharper instrument for the same claim behind Can text-trained models compress images better than specialized tools? and it directly addresses the failure that Why do Shannon and Kolmogorov measures fail to value data? identifies: prior compressors mismeasure learned content because they let irrelevant quantities dominate. By coding only disagreement, requential coding finally isolates the signal — which means we can compare how simple two models really are even when their sizes differ wildly.