Training an AI to predict its own inner concepts, not just the next word, could be cheaper. Is there a formula?
What scaling laws govern the compute efficiency of latent prediction versus token prediction?
This explores whether the corpus has a measured scaling law, in the sense of a power-law curve, for how much cheaper it is to learn by predicting a model's own hidden representations (latents) than by predicting the next token.
This explores whether there is a measured scaling law for how much cheaper it is to learn by predicting hidden representations (latents) than by predicting the next token. The corpus has no single fitted curve comparing the two. It has one theoretical result on why the gap should exist, a few empirical measurements of how big it is, and some nearby scaling results that show what a real law would need to include.
The theory is the sharpest piece. A formal sample-complexity analysis shows that latent-level self-supervision (the data2vec/JEPA family) recovers hierarchical structure with a number of samples that stays constant as the hierarchy gets deeper. Token-level learning needs exponentially more samples as depth grows. The reason is that latents at the same level of a hierarchy are much more correlated with each other than raw tokens are, so each training example carries more usable signal (Why is predicting latents more sample-efficient than tokens?). This gives the shape of the law, constant versus exponential in depth. It does not give a coefficient you could plug numbers into.
The measurements are looser. An 8.9B model trained to predict concepts alongside tokens matched OLMo-3-7B's final loss using 51.3% of the training tokens, roughly a 2x data-efficiency gain, and beat it by 2.45 points downstream (Can models learn faster by predicting their own concepts?). That result is a single comparison, and it adds latent prediction to token prediction instead of replacing it. Latent-thought language models make a different claim: the size of the latent becomes its own scaling axis, separate from parameter count, so you can grow model size and latent size independently (Can latent thought vectors scale language models beyond parameters?).
"Compute efficiency" can also mean compute spent at inference, and here latent computation looks strong. Looped models that re-apply layers beat larger feedforward networks on reasoning, because recursion gives them something extra parameters don't (Can models learn by looping instead of growing larger?). A looped transformer supervised with gold reasoning steps at each latent position matched explicit chain-of-thought at 3B scale on GSM8K and cut thought-phase latency by 2.5x to 6.9x (Can latent reasoning close the scaling gap with explicit chain-of-thought?). A 150M-parameter recurrent latent model reached 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, which beats the earlier cost-accuracy frontier (Can latent reasoning match chain-of-thought cost efficiency without verbalizing?). One study also finds that long-context performance improves with more consolidation passes, a test-time scaling pattern that is not caused by memory capacity (Is long-context bottleneck really about memory or compute?). The counterweight is that reasoning models beat non-reasoning models at any inference budget, because training is what makes extra compute productive (Can non-reasoning models catch up with more compute?). Latent compute is probably the same: it pays off only if training taught the model to use it.
The closest things to real laws in the collection concern architecture, not latent versus token. Scaling laws extended with hidden size, MLP-to-attention ratio, and grouped-query attention settings found designs with up to 42% more throughput at slightly higher accuracy (Can architecture choices improve inference efficiency without sacrificing accuracy?). Deep-and-thin small models beat balanced ones, which contradicts the classic Kaplan-style law (Does depth matter more than width for tiny language models?). The Byte Latent Transformer spends more compute where the next byte is hard to predict and less where it's easy (Can byte-level models match tokenized performance with better efficiency?). Together these suggest a latent-versus-token law would have to include hierarchy depth, latent size, and loop count, not just parameters and tokens. Nobody in this collection has fitted that law yet.
Sources 11 notes
A formal sample-complexity analysis proves latent-level self-supervision (data2vec/JEPA style) recovers compositional structure with samples constant in hierarchy depth, while token-level learning requires exponential samples—because same-level latents are far more correlated than raw tokens.
An 8.9B model trained to predict both tokens and learned concepts from its hidden states matched OLMo-3-7B's final loss using only 51.3% of training tokens and outperformed it by 2.45 points downstream. This suggests explicit supervision of multi-token semantic structure improves compute efficiency.
Latent-Thought Language Models achieve superior sample and parameter efficiency by coupling fast local variational learning with slow global decoder learning. This dual-rate scheme scales few-shot reasoning across both model and latent size, creating independent scaling dimensions beyond traditional parameter scaling.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
LOTUS, a looped Transformer trained with gold CoT supervision at each latent position, closes the latent-to-explicit gap on GSM8K and cuts thought-phase latency by 2.5× to 6.9×. Latent reasoning need not be verbalized token-by-token to remain legible and competitive.
Show all 11 sources
A 150M-parameter model combining in-context demonstrations with iterative latent computation reached 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, surpassing previously reported cost-accuracy tradeoffs. The approach separates learning (via demonstrations updating recurrent memory) from reasoning (via iteration in hidden space) without generating intermediate tokens.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.
The Byte Latent Transformer (BLT) dynamically segments bytes into patches based on next-byte entropy, allocating more compute to high-entropy regions and less to predictable ones. At 8B parameters, BLT matches tokenized baselines while reducing inference cost and improving robustness to typos and cross-lingual transfer.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
- Hierarchical Reasoning Model
- Learn from your own latents and not from tokens: A sample-complexity theory
- NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
- Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
- Scalable Language Models with Posterior Inference of Latent Thought Vectors
- Reasoning Beyond Chain-of-Thought: A Latent Computational Mode in Large Language Models