AI can nail the next move in a game or physics sim without ever grasping the rules behind it.
Why do foundation models fail at hidden state prediction despite sequence accuracy?
This explores why a model can predict the next step of a sequence very well and still fail to recover the hidden thing generating it, such as the physical law, the game state, or what a user actually wants.
This explores why a model can predict the next step of a sequence very well and still fail to recover the hidden thing generating it, such as the physical law, the game state, or what a user actually wants. The corpus's clearest answer is that predicting sequences doesn't require knowing the hidden state. It only requires a pile of shortcuts that happen to work on the data.
Do foundation models learn world models or task-specific shortcuts? is the direct evidence. Transformers trained on orbital mechanics and games predict well, but probing what they learned shows no unified structure. When researchers fine-tune them to extract the underlying law, they get nonsensical laws that change depending on which slice of data they use. Even arithmetic runs on range-matching heuristics rather than an actual algorithm. Many different internal rules fit the same sequences, and next-step training has no reason to pick the one that matches reality.
Why shortcuts win: Can we predict where language models will fail? frames LLMs as autoregressive probability machines. They do best when the correct answer is also the likely-looking answer, and they stumble when it's low-probability, even if the task is logically simple (backwards alphabet, letter counting). A hidden state that has to be inferred and carried forward is where likelihood and correctness come apart. Why do language models ignore information in their context? shows a cousin of this. When training associations are strong, they override what's in front of the model, and prompting alone can't fix it. It took causal intervention on the representations.
The same failure appears when the hidden state is a person's intent. Why do language models fail in gradually revealed conversations? finds that when a request is revealed gradually, all major LLMs lose 39% performance on average. They guess the hidden intent early and never revise it, and agent-style mitigations recover only 15-20% of the loss. The model doesn't hold an uncertain state open while evidence arrives. It commits and builds on the guess.
A few notes hint at where the fix might lie. Can state-space models match transformers at copying and retrieval? shows that state-space models, which compress history into a fixed-size hidden state, lose to transformers at copying and retrieval. So having a hidden state isn't enough, because what it holds and how it's used matter. Why do trajectories matter more than individual examples for in-context learning? adds that in-context sequential decision-making needs full trajectories from the same environment, not isolated examples, because state can only be recovered from connected history. Meanwhile Can tiny recursive networks outperform massive language models? and Can latent reasoning match chain-of-thought cost efficiency without verbalizing? show small networks that repeatedly update an internal reasoning state beating far larger LLMs on ARC-AGI puzzles, with the gains coming from the recursion rather than from scale. None of these notes tests hidden-state prediction directly. Together they suggest that giving models room to iterate on a latent state may matter more than training them to predict more tokens.
Sources 8 notes
Inductive bias probes show transformers trained on orbital mechanics and games learn predictive patterns, not unified world structure. Fine-tuning reveals nonsensical, slice-dependent laws; circuit analysis shows arithmetic relies on range-matching heuristics, not algorithms.
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Across 200,000+ conversations, all major LLMs show 39% average performance drop in multi-turn settings due to locking into incorrect early guesses. Agent mitigations recover only 15-20% of this loss.
Two-layer transformers can copy exponentially long strings while state-space models are fundamentally limited by their fixed-size latent state. Empirically, transformers dramatically outperform SSMs at copying and context retrieval in both synthetic and pretrained settings.
Show all 8 sources
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.
A 7M-parameter two-layer network recursing on its latent reasoning state reached 45% on ARC-AGI-1, beating larger LLMs with 0.01% of their parameters. The gains come from recursion itself, not scale or hierarchical architecture.
A 150M-parameter model combining in-context demonstrations with iterative latent computation reached 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, surpassing previously reported cost-accuracy tradeoffs. The approach separates learning (via demonstrations updating recurrent memory) from reasoning (via iteration in hidden space) without generating intermediate tokens.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- BDH-CQ: In-Context Learning with Recurrent Latent Reasoning
- Hierarchical Reasoning Model
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
- It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
- The Topological Trouble With Transformers
- Repeat After Me: Transformers are Better than State Space Models at Copying
- LLMs Get Lost In Multi-Turn Conversation