Can latent thought vectors scale language models beyond parameters?
Explores whether explicit latent thought vectors with dual-rate learning create new scaling dimensions independent of model size. This matters because it suggests alternatives to simply building larger models.
Latent-Thought Language Models (LTMs) propose a different scaling strategy than larger parameters or longer contexts: explicit latent thought vectors that follow a prior model in latent space and guide autoregressive token generation. This creates additional scaling dimensions — higher sample efficiency by increasing training compute per token, with further gains by trading model size for more inference steps.
Architecture. Latent thought vectors represent an abstract representation of the entire sequence, controlling the decoder's generation of each token. Training uses variational Bayes with a dual-rate process: fast learning of local variational parameters for the posterior distribution of latent vectors (adapting quickly to specific inputs) coupled with slow learning of global decoder parameters (gradually accumulating general knowledge).
Cognitive inspiration. The dual-rate scheme parallels established cognitive models:
- Declarative-procedural model (Ullman 2004): latent vectors and local parameters parallel declarative/episodic memory; global decoder parameters parallel procedural memory
- Fast-slow learning (Kumaran et al. 2016): fast episodic learning and slow schematic learning interplay
- Language of thought (Fodor 1975): latent thought vectors as "words" of an internal thought language
Scaling properties. LTMs demonstrate superior sample and parameter efficiency compared to conventional autoregressive models and discrete diffusion models. They significantly outperform on validation perplexity and zero-shot language modeling. Emergent few-shot in-context reasoning capabilities scale with both model size and latent size — providing two independent scaling dimensions.
The connection to existing latent reasoning approaches is important but distinct. Can models reason without generating visible thinking tokens? describes depth-recurrent architectures that iterate in latent space at inference time. LTMs use latent vectors differently — as sequence-level abstractions that guide token generation rather than per-token iterative computation. The dual-rate learning provides a training-time mechanism that depth-recurrence does not.
The Titans parallel is also notable: Can neural memory modules scale language models beyond attention limits? separates fast attention (short-term) from slow memory (long-term). LTMs separate fast local adaptation from slow global learning. Both architectures implement the fast-slow cognitive distinction but at different levels — Titans for memory, LTMs for generation.
Inquiring lines that read this note 60
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does latent reasoning compare to verbalized chain-of-thought?- Can steering a single latent feature replicate chain-of-thought performance?
- Can latent space represent reasoning dimensions that text cannot?
- Can continuous latent reasoning match discrete chain-of-thought without training modifications?
- Why does recursion on latent state drive generalization better than hierarchy?
- Can latent reasoning scale test-time compute without verbal tokens?
- How much explicit verbal signal must latent chains retain to perform well?
- Why does textual chain-of-thought avoid the representational drift problem automatically?
- Can latent reasoning scale test-time compute without verbalized tokens or special training?
- Can autoregressive models be trained to produce more cataphoric text?
- Do diffusion language models learn differently than autoregressive models?
- Does scaling model size solve compositional generalization problems?
- Why does scaling data and model size improve compositional generalization?
- How do latents at the same hierarchy level become more correlated than tokens?
- Why does recursion on latent states improve generalization more than scale?
- Do representations in models causally influence text generation?
- How does LatentQA differ from predefined concept steering like representation engineering?
- How does dual-rate learning separate episodic and procedural memory in neural networks?
- What non-parametric methods could replace latent factors for inductive learning?
- Do long-term memory modules outperform consolidation into fast weights?
- Can latent recurrence overcome the trainability costs of depth?
- Why should scaling laws be understood as properties of data distribution rather than training in general?
- Why do optimal learning dynamics improve scaling law coefficients specifically?
- Can fast-slow separation improve both memory and generation in language models?
- Do pretrained language models carry reusable computational scaffolding for length handling?
- What is the comprehension-generation asymmetry in language models?
- How do parameter scaling and latent vectors interact in language models?
- Does the prediction unit shape what language models actually learn?
- Why is latent-level prediction more sample-efficient than token-level prediction?
- Does latent manipulation outperform token-level prediction for efficiency?
- Can language models generate plausible latent thoughts without human annotation?
- Do language models need words to think or just latent structure?
- What empirical evidence supports the Learning Law on real language models?
- Do base models contain latent reasoning that minimal training can unlock?
- How much training data is truly necessary to unlock latent model reasoning?
- Can minimal training signals unlock latent reasoning capability in base models?
- What latent reasoning capability do base models already possess before training?
- Why do language models fail at iterative numerical optimization despite scale?
- Why do language models plateau at constraint satisfaction regardless of scale?
- Can language models execute iterative numerical methods in latent space?
- How do encode-decode contractive biases create stable attractors in latent space?
- What prevents representation collapse in latent-prediction world models like JEPA?
- What are the scaling law differences between vision and language learning?
- What power-law scaling patterns emerge when consistency models are trained at scale?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can models reason without generating visible thinking tokens?
Explores whether intermediate reasoning must be verbalized as text tokens, or if models can think in hidden continuous space. Challenges a foundational assumption about how language models scale their reasoning capabilities.
different latent approach: per-token iterative computation vs sequence-level latent vectors
-
Can neural memory modules scale language models beyond attention limits?
Can separating short-term attention from adaptive long-term memory allow models to efficiently handle context windows exceeding 2M tokens while maintaining competitive performance?
Titans implements fast-slow at memory level; LTMs implement it at generation level
-
Can inference compute replace scaling up model size?
Explores whether smaller models given more thinking time during inference can match larger models. Matters because it reshapes deployment economics and compute allocation strategies.
LTMs demonstrate this: model size can be traded for inference steps
-
Can computational power accelerate scientific discovery itself?
Does the pace of research breakthroughs scale with computing resources, like model performance does? ASI-ARCH tested this by running thousands of autonomous experiments to discover neural architectures.
LTMs provide new dimensions for architecture search
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Scalable Language Models with Posterior Inference of Latent Thought Vectors
- Reasoning to Learn from Latent Thoughts
- Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
- Learn from your own latents and not from tokens: A sample-complexity theory
- Scaling Laws for Neural Language Models
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
Original note title
latent-thought language models introduce additional scaling dimensions beyond parameters by incorporating explicit latent thought vectors with dual-rate learning