Data can look like pure noise to a weak computer yet hold real patterns a stronger one could use.
What makes a bounded observer's ability to extract information different from apparent randomness?
This explores how data can look random to an observer with limited computing power while still holding structure that observer can use, and how that differs from the classical idea that randomness is a fixed property of the data.
This explores how data can look random to an observer with limited computing power while still holding structure that observer can use. The corpus's answer is that randomness isn't only a property of the data. It also depends on who is looking and how much work they can afford. What can a bounded observer actually learn from data? formalizes this as 'epiplexity', the structural information a computationally bounded observer can actually extract. It separates that learnable regularity from time-bounded entropy, which is the part that looks like noise because the observer can't afford to decode it.
The classical tools can't draw this line. Why do Shannon and Kolmogorov measures fail to value data? points out that Shannon and Kolmogorov measures both assume an observer with unlimited compute. To such an observer, 'hard to learn' and 'impossible to learn' are the same as 'already known'. The corpus says this gap explains three things classical theory can't. Feature engineering helps, even though an all-powerful observer would gain nothing from reshaping data it already has. The order in which you present training data matters. And trained models can end up knowing more than the process that generated their data.
The distinction has practical uses. Epiplexity doesn't depend on any particular task, yet it tracks out-of-distribution generalization, which helps explain why some datasets transfer more broadly than others. A related line of work, Can we separate learnable surprise from random noise?, separates learnable surprise from unlearnable noise. That one quantity works as a complexity measure and as an objective for representation learning. As an exploration reward, it beat task rewards in nine of ten environments. So the noise-versus-structure split can tell an agent what is worth paying attention to.
Two neighbouring notes show that appearances can mislead in both directions. When a language model runs with a fixed seed and temperature zero, its output looks stable, but Does setting temperature to zero actually make LLM outputs reliable? shows it is still one draw from a probability distribution. Consistency is not reliability. In the other direction, Can repeated quiet probes separate decoys from genuine objects? shows that with enough cheap, quiet probes, an observer can separate decoys from genuine objects with vanishing error. This holds only in an idealized setting with independent responses, and only if the two response distributions truly differ. Together these suggest that what an observer can extract depends on its budget, whether compute or repeated probes, and on whether a real difference sits underneath. The corpus gives the formal definitions and this idealized probe result, but it doesn't say how to measure epiplexity on messy real-world data.
Sources 5 notes
Epiplexity formalizes the structural information a computationally bounded observer can extract from data, separating learnable regularity from time-bounded entropy. This task-free measure correlates with out-of-distribution generalization and explains why some datasets enable broader transfer than others.
Both measures assume observers with unlimited compute and miss learnable, useful information. The gap explains why feature engineering helps, curriculum order matters, and trained models exceed their generating process—empirical facts classical theory cannot account for.
Separating learnable surprise from unlearnable noise yields a single quantity that acts as a measure of complexity, an objective for representation learning, and a reward signal outperforming task rewards in nine of ten environments.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
In idealized settings with independent responses, enough quiet probes let a classifier separate decoys from genuine objects with vanishing error if their response distributions differ and are known or learnable from feedback.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
- Intelligence from Learnable Novelty
- Beyond the Exploration-Exploitation Trade-off: A Hidden State Approach for LLM Reasoning in RLVR
- From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
- Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- When Agents Talk: Honeytokens under Shared Memory
- Can Large Reasoning Models Self-Train?