Line of inquiry
Inquiring lines›How do training data and architect…›What training signals and data sel…›this line of inquiry
What are the trade-offs between memory capacity and continuous task learning?
A broader line of inquiry — a family of 63 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 63
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- Do long-term memory modules outperform consolidation into fast weights?
- Does moving memory outside model weights avoid the limitations of in-weight retention?
- Why does fine-tuning for continuous space cause catastrophic forgetting?
- Can data pruning strategies exploit the finite nature of memorization capacity?
- Why does fine-tuning models for continuous reasoning cause catastrophic forgetting?
- Can in-weight memorization scale beyond model parameter count limits?
- Why does specializing to one task make future task learning harder?
- How does in-weight memorization scale with model parameter count?
- Can precomputed inferences be stored in memory modules between model interactions?
- Why does semantic deduplication reduce memorization in fine-tuned models?
- Why do large language models outperform fine-tuned models once repeated items are removed?
- How does in-weights adaptation create spurious forgetting in models?
- What makes factual memorization less efficient than tool-based retrieval?
- Do models with unfilled memorization capacity appear to generalize falsely?
- Why does in-weight memorization fail compared to tool-based fact access?
- How do retention gates regularize forgetting across different sequence model architectures?
- Can contamination-free evaluation distinguish between memorization and genuine prediction ability?
- How do newly learned facts become accessible after gradient updates?
- Is forgetting in language models reversible or permanent knowledge loss?
- Does composing multiple continual learning mechanisms reduce forgetting more than single approaches?
- How can a forgetting policy preserve rare knowledge while preventing over-generalization?
- Can document repetition accidentally memorize sensitive information instead of learning?
- When does training a memory model beat RAG or fine-tuning?
- How does distributional shift toward rare inputs change memorization reliance?
- Why is extracting training data insufficient proof that models memorize?
- Can continuum memory systems prevent catastrophic forgetting in neural networks?
- How do complementary learning systems explain the need for fast and slow consolidation?
- Why do pretrained model priors reduce the usefulness of retrieved experience?
- Can native memory procedures acquired through training handle stale or incorrect cached information?
- What mechanism transfers explicit memories into parametric model weights?
- Can we unlearn memorized text by finetuning only high-gradient weights?
- What causes catastrophic forgetting during domain knowledge embedding?
- What determines whether accumulated state generalizes spuriously across continual learning domains?
- How do fixed recurrent states trade off copying accuracy for filtering ability?
- How does memorization capacity saturation trigger the grokking transition?
- Can neural modules memorize surprising tokens as adaptive long-term memory?
- Can memory-based adaptation and gradient fine-tuning operate on complementary timescales?
- How do layer-wise versus parameter-wise merging strategies affect information retention?
- Do sample-level similarities between pretraining and downstream tasks explain the frequency effect?
- Does grokking in modular arithmetic follow the same three-phase learning trajectory?
- How do trained weights differ from a stored library or text?
- How much does memorization capacity limit a model's ability to learn new information?
- Why does fine-tuning fail to remove temporal contamination from pretraining?
- What are the distinct sources of catastrophic forgetting in sequential fine-tuning?
- What distinguishes data that generalizes broadly from task-specific memorization?
- Can pretraining-frequency signals alone prevent RAG systems from confabulating about common knowledge?
- Why does grokking reveal the shift from memorization to genuine understanding?
- Why does a replay mechanism prevent reasoner skills from over-specializing?
- How does KL regularization prevent both forgetting and adaptation loss?
- Why are rare tokens the hooks for verbatim model memorization?
- How can memory shift from a passive datastore to an actively trained component?
- Does fine-tuning on sensitive data create higher memorization and leakage risks?
- What makes memorized paragraphs harder to corrupt than generic text?
- What is the theoretical capacity limit before memorization saturates?
- Why does test accuracy improve after training accuracy reaches 100 percent?
- Can the joint-training principle extend beyond memorization and generalization pairs?
- How do out-of-distribution tests reveal that optimization learning is memorization?
- Can a memory module be swapped between different base models?
- How do the three grokking phases connect to memorization capacity limits?
- Can split-softmax or other inference methods fully prevent instruction drift?
- What attentional bias objectives compete with dot product similarity for associative memory?
- How does dual-rate learning separate episodic and procedural memory in neural networks?
- What knowledge injection routes trade flexibility against training cost?