How much an AI can be compressed may predict how well it generalizes — better than simply counting its parameters.

This explores whether how much a model can be compressed — its true learned content — tells you more about generalization than raw parameter count. The corpus makes a surprisingly strong case that it does, and that parameter count is often the wrong ruler entirely. The cleanest evidence is a memorization capacity finding: GPT-family models memorize at a fixed rate of about 3.6 bits per parameter, and only once that capacity fills does a phase transition kick in and genuine generalization (grokking) begin When do language models stop memorizing and start generalizing?. Parameter count sets the ceiling on how much can be stored, but it's the *transition* — what happens after storage saturates — that governs generalization. Counting parameters tells you the size of the bucket, not whether the water has turned to something useful.

Several notes push toward measuring learned content directly rather than inferring it from size. One proposes 'requential coding,' which measures a model's true simplicity by encoding only where a student disagrees with a teacher — yielding a compressed size that is explicitly independent of parameter count and orders of magnitude shorter than older methods Can we measure what a model truly learned?. That's the sharpest form of your question: it's a compression metric deliberately decoupled from parameter count, built precisely because parameter count confounds what you actually want to know. Related work frames optimal learning itself as maximizing a data compression ratio, and shows this reformulation improves scaling law *coefficients*, not just constants — meaning better compression changes the fundamental slope of how models improve, not just where they start Does optimal language model learning maximize data compression?.

The deepest version of the claim is that generalization and compression are the same thing. Chinchilla models trained only on text compress images and audio better than PNG and FLAC, purely by using their context window to adapt — demonstrating that generalization *operates through* compression rather than through specialization Can text-trained models compress images better than specialized tools?. If that equivalence holds, then a model's compression size isn't just a better predictor of generalization — it's a more direct measurement of the same underlying quantity, while parameter count is a crude proxy.

What complicates the simple 'smaller compressed size is better' story is that compression can be *too* aggressive. Using rate-distortion theory, LLMs are shown to compress concepts far more aggressively than humans — capturing broad category structure but losing the fine-grained distinctions humans preserve for situated meaning Do LLMs compress concepts more aggressively than humans do?. So the useful metric isn't minimum size but the compression-distortion tradeoff: how much structure survives compression. This reframes your question — compression size predicts generalization, but only when you also account for what got thrown away.

Finally, the corpus suggests *architecture*, not parameter count, is what unlocks compressible generalization. Depth beats width at sub-billion scale, with deep-thin models composing abstract concepts through layers rather than spreading parameters across width Does depth matter more than width for tiny language models?, looped architectures achieve reasoning gains from iterated depth instead of scale Can models learn by looping instead of growing larger?, and small models trained with DPO on a teacher's preference pairs can match much larger ones on function-calling reasoning Can small models match large models on function calling?. The through-line: two models with identical parameter counts can differ enormously in learned content, and it's the compressed, learned content — not the parameter budget — that tracks generalization.