Line of inquiry
Inquiring lines›How can we train models to be prop…›What architectural and training in…›this line of inquiry
How does compressing or distilling information affect model learning and generalization?
A broader line of inquiry — a family of 61 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 61
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- How does the compression view extend from trained models to training objectives?
- Can compressed context distilled into smaller models preserve teacher performance reliably?
- Can we balance interpretability with the efficiency gains of compressed inter-model communication?
- How can language models extract more value from fewer demonstrations?
- Can compression length really indicate how well a model generalizes?
- Why does capturing domain structure reduce data requirements more than raw volume?
- Why do student models learn better from internal pruning versus external compression?
- Can context compression preserve what matters without introducing bias?
- How does modeling capability relate to lossless compression in language models?
- Can model compression size predict generalization better than parameter count?
- Why does language compression via statistical dependencies capture cultural and situated language use?
- Can models reach high interpretability without sacrificing capability through sparsity training?
- Why do naive pruning and quantization destroy LLM performance so easily?
- What compression explains why syntax fits in low-dimensional subspaces?
- Can next-token prediction train models to optimize for communication efficiency?
- Why does representation sparsity reliably indicate task difficulty for language models?
- Why do smaller models favor code formats while larger models prefer natural language?
- What specific optimizations from LLM training transfer back to encoder models?
- Do discrete tokenized modalities preserve information better than continuous embeddings?
- Why do parameter-based compressors fail to measure true model simplicity?
- Why do encoder models process document corpora more efficiently than decoder models?
- What scaling laws govern the compute efficiency of latent prediction versus token prediction?
- Do decoder-only models have inherent architectural limits for non-sequential information?
- Why is latent-level prediction more sample-efficient than token-level prediction?
- Why does statistical compression destroy literary connotation and meaning?
- Can decoder-only models become effective text encoders with training?
- How do Bayesian models share statistical strength across sparse user datasets?
- Why do vision and language have different optimal scaling curves?
- Can a separate frozen module manage compression better than joint optimization?
- Do high-entropy RLVR tokens correspond to MI-peak tokens during inference?
- Why does context compression need reward signals beyond explicit coherence metrics?
- Why is sparse per-student data a bottleneck for building adaptive tutoring systems?
- Can speech encoders learn vocal tract physics without phonetic labels?
- Why does adjusted compression performance degrade as models scale larger?
- How does contrastive feedback improve compression guideline quality versus weaker optimization?
- Do speech encoders actually learn the physics of how vocal tracts produce sound?
- What is the connection between model compression and data compression?
- Can universal function approximators be expensive to learn in practice?
- Do self-supervised speech models learn dialect-agnostic representations across languages?
- Can task-agnostic compression of documents remain broadly useful for later queries?
- Can speech embeddings carry articulatory structure that text cannot?
- Can the same description-then-retrieve pattern work for domain adaptation without target data?
- Why do speech encoders learn the physics of human vocal tracts?
- Why do image captions create different friction than pure video data?
- What makes internal embeddings useful as multimodal input for language model training?
- How does requential coding measure true simplicity without parameter count inflation?
- How does data entropy inflate compression estimates in prequential coding?
- What scaling exponent would audio or other modalities require in a truly multimodal system?
- How do speech encoders learn articulatory physics without phonetic labels?
- What information does transcription destroy that direct speech-to-speech models preserve?
- Why does transcription destroy prosodic information in speech processing?
- What access constraints allow description-based adaptation but block conventional techniques?
- Why does each rewrite cycle degrade domain-specific details differently than compression?
- Why does text encoding create different subspaces across domains?
- Why does masking the penultimate token outperform random token masking?
- Why does sparsity per user make probabilistic models more effective?
- What training data do modern commercial ASR systems actually contain?
- How does epiplexity measure extractable value differently from compression codelength?
- What information does transcription destroy that direct speech pathways preserve?
- Does ternary weight quantization simplify deployment of mixture of experts?
- What paired speech data is needed to train end-to-end models?