Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data

Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding†, where a teacher model selects training samples drawn from the student’s own distribution. The student’s code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters.

Introduction. Measuring compression is key to understanding generalization in deep learning. In order to compress data, a model must discover regularities that facilitate generalization. This intuition underlies fundamental principles of induction, such as Occam’s razor: the simplest explanation consistent with observations is most likely to be true. A strong enough compression can guarantee a model’s generalization performance, limit memorization, and even reveal how much learnable information content is in the training data. Indeed, a growing body of evidence suggests that neural networks often learn functions far simpler than their parameters could express [27, 10, 53, 30, 48]. However, finding a sufficiently good compression at scale remains a fundamental open question. It could be that larger neural networks find even simpler, more compressible functions, but demonstrating this compressibility becomes increasingly difficult with scale. As we scale model and data size, existing model compression schemes are inflated by quantities unrelated to actual learning:

Discussion / Conclusion. We have introduced requential coding, a novel method for compressing generative models. By coding a training process built from self-generated data, the requential code depends on neither the parameter count nor the data entropy, compressing neural networks to far smaller sizes than previously shown, with an advantage that grows with model scale. As a result, its compressed model size proves to be a powerful tool for understanding learning phenomena beyond the reach of prior compression methods. While a significant step toward revealing how compressible neural networks really are, there are several open questions that could form the basis for exciting future work. Conceptually, requential coding provides a lossless code for the student, not the teacher.