An AI can generate varied-looking answers that all follow the same hidden template — actual responsiveness to your question is a different thing to measure.
How does mutual information between inputs and outputs differ from measuring raw diversity?
This explores the difference between two ways of scoring an AI's outputs — raw diversity (how much variety there is in the outputs on their own) versus mutual information (how much the outputs actually track and depend on the inputs that produced them).
This explores the gap between measuring raw output variety and measuring whether outputs are actually responsive to their inputs — and the corpus is unusually sharp on why the distinction matters. Raw diversity, usually measured as entropy, only looks at the spread of outputs. Mutual information asks a different question: given an input, does the output change in a way that depends on that input? The cleanest illustration is template collapse. A reasoning model can drift toward filling in the same fixed scaffold across every problem, changing only the surface numbers. Entropy still looks healthy — there's plenty of token-level variety within any single answer — so the collapse stays invisible. But mutual information between inputs and reasoning drops, because the reasoning has stopped being a function of the problem, and that MI drop tracks failing task performance far better than entropy does Can entropy metrics detect when reasoning becomes formulaic?.
The deeper point is that entropy is a within-input measure and can be high while something is badly wrong across inputs. Two failure modes make this concrete. First, outputs can look varied but be relationally identical: 70+ models queried on open-ended prompts independently converge on nearly the same responses — an 'Artificial Hivemind' — so ensembling for diversity buys you almost nothing even though each model's raw output distribution is spread out Do different AI models actually produce diverse outputs?. Second, outputs can be truly identical on the surface while their internal machinery is completely different — networks that reproduce the same answers can carry fractured, tangled representations that block transfer and recombination Can identical outputs hide broken internal representations?. In both cases a scatter-based diversity score misreads the situation; a relational measure would not.
This connects to an older complaint in information theory that the corpus surfaces from a different angle: classical Shannon and Kolmogorov measures assume an observer with unlimited compute, so they count raw unpredictability rather than useful, learnable structure — which is why they fail to explain why feature engineering, curriculum order, or one dataset over another actually helps Why do Shannon and Kolmogorov measures fail to value data?. The proposed fix, 'epiplexity,' tries to measure the structure a bounded learner can extract, and that measure — not raw entropy — is what correlates with out-of-distribution generalization What can a bounded observer actually learn from data?. Mutual-information-style thinking is the same instinct scaled down: score the relationship between input and output, not the loudness of the output.
The practical payoff shows up in how people are learning to reward diversity during training. Simply maximizing raw variety tends to backfire — RL collapses format diversity onto a single dominant pretraining pattern within an epoch Does RL training collapse format diversity in pretrained models?, and self-improvement loops that don't distinguish diversity from quality irreversibly lose it How do quality, diversity, and complexity affect synthetic data differently?. The methods that work optimize for *grounded* diversity instead: DARLING rewards semantic diversity through a learned classifier and finds it improves quality rather than trading against it Can diversity optimization improve quality during language model training?; vector-valued rewards let solutions specialize along real task trade-offs so diversity reflects genuine differences in what each solution does, not external noise Can reward vectors be the hidden source of solution diversity?; and set-level RL rewards a batch of traces for being collectively useful, valuing complementarity over redundant sameness Can diverse mediocre traces outperform redundant expert traces?.
So the difference is not cosmetic. Raw diversity answers 'how much do these outputs vary?' — a question you can pass while quietly failing. Mutual information answers 'do these outputs carry information about their inputs?' — which is closer to what you actually wanted diversity to be a proxy for. The recurring lesson across the corpus is that variety is easy to fake and easy to misread, while relational and structural measures catch the collapses that entropy sails right past.
Sources 10 notes
Entropy only captures within-input diversity and cannot detect when reasoning drifts to fixed templates across inputs. Mutual information between inputs and reasoning correlates far more strongly with task performance, making it the more reliable diagnostic.
INFINITY-CHAT analyzed 70+ models across 26K open-ended queries and found an "Artificial Hivemind" effect: models independently generate strikingly similar or identical responses due to overlapping training data and alignment procedures, undermining the diversity benefits of model ensembles.
Networks trained with SGD reproduce outputs perfectly while having radically different internal structure than evolved networks, with weight perturbations revealing fractured, entangled representations that prevent transfer to novel contexts or creative recombination.
Both measures assume observers with unlimited compute and miss learnable, useful information. The gap explains why feature engineering helps, curriculum order matters, and trained models exceed their generating process—empirical facts classical theory cannot account for.
Epiplexity formalizes the structural information a computationally bounded observer can extract from data, separating learnable regularity from time-bounded entropy. This task-free measure correlates with out-of-distribution generalization and explains why some datasets enable broader transfer than others.
Show all 10 sources
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Quality drives in-distribution generalization, diversity enables out-of-distribution generalization, and complexity strengthens both. Current evaluation methods collapse these into a single quality metric, causing self-improvement loops to degrade through irreversible diversity loss.
DARLING jointly optimizes for quality and semantic diversity using a learned classifier, finding that diversity rewards catalyze exploration and produce higher-quality outputs than quality-only baselines across both creative and mathematical tasks.
Vector Policy Optimization shows that rewards decomposed per test-case, criterion, or persona provide an inherent diversity structure. Training solutions to span the Pareto frontier across these dimensions produces competent diversity grounded in real task trade-offs rather than external regularizers.
SPIRAL shifts RL reward from individual traces to sampled sets, optimizing for complementarity rather than per-trace accuracy. Diverse mediocre traces outperform redundant strong ones because aggregators need raw material to arbitrate, not confirmation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
- Evaluating the Diversity and Quality of LLM Generated Content
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- The Art of Scaling Reinforcement Learning Compute for LLMs
- Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
- Jointly Reinforcing Diversity and Quality in Language Model Generations
- Vector Policy Optimization: Training for Diversity Improves Test-Time Search
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining