Should an AI's memory be chopped into pieces sized for the job it's doing, or for how big the AI is?
Does memory granularity need to match the task domain or the model size?
This explores whether the size of the unit an AI stores and retrieves (a whole session, a single fact, one reasoning step) should be set by the kind of job or by how big the model is.
This explores whether the size of the unit an AI stores and retrieves (a whole session, a single fact, one reasoning step) should be set by the kind of job or by how big the model is. The corpus has no study that tests the two head to head, so what follows is inference from adjacent evidence. It leans toward the task, with model size as a secondary factor.
The task-side evidence is the stronger of the two. In skill retrieval, the main bottleneck turns out to be how finely a task is broken into steps. Standard LLM decomposition finds only 34% of the steps a task needs, and correcting the step count recovers about 75% of the gains in iterative methods What blocks skill retrieval in task decomposition?. That paper is about retrieving skills rather than memories, but the retrieval problem is the same. Function calling shows a similar pattern. Training across seven fine-grained subtasks (nested calls, chaining, parameter detection and so on) generalizes better than one umbrella dataset Can breaking function calling into subtasks improve model generalization?. In both cases the right unit mirrors how the task itself breaks down.
Model size seems to change something else: how much detail a model can hold without things overwriting each other. Larger models learn rare tasks better because their spare capacity weakens the gradients from common tasks, so slowly forming rare-task features don't get overwritten. They don't learn them better because they can represent solutions smaller models can't Why do larger models learn rare tasks better?. If that carries over to memory, a smaller model isn't asking for coarser memories. It is more exposed to interference, so the mix of what you feed it matters more. The agent literature points the same way. Most agent work is repetitive and well-defined, and small models handle it at 10-30x lower cost Can small language models handle most agent tasks?. Small models can also match large ones on function calling when trained on a teacher's correct and incorrect examples Can small models match large models on function calling?. The gap closes through the shape of the training signal, and no smaller memory unit was needed.
Several designs avoid making the choice up front. Titans stores tokens according to how surprising they are, so granularity follows the content Can neural memory modules scale language models beyond attention limits?. The Thread Inference Model prunes its cache along recursive subtask boundaries, so the memory unit is whatever the reasoning tree says it is Can recursive subtask trees overcome context window limits?. Prime Agent keeps four levels at once (weights, context, a persistent REPL, disk history), so each kind of information sits at the resolution that suits it Can external state caches let models solve harder problems?. Where memory lives is a separate question. It can be a dedicated memory model beside a frozen LLM Can a separate memory model inject knowledge without touching the LLM?, or a native state inside the backbone Should agent memory live inside the model backbone?. Long-context work also suggests that the compute spent turning old context into usable state matters more than how it is chunked Is long-context bottleneck really about memory or compute?.
The caution is that a granularity can look right and still fail. In MemTrapBench, all five memory frameworks scored below a no-memory baseline by more than 10%, even though the memories were accurate and relevant Can relevant memories actually harm LLM reasoning?. The failure showed up at the point of use, not in storage. The practical test is therefore to match granularity to how the task decomposes, then check it against a no-memory baseline. Model size is better treated as a limit on how much interference the memory can cause.
Sources 12 notes
Standard LLM decomposition reaches only 34% step-level recall, gating retrieval success. Correcting step count recovers 75% of gains in iterative methods, shifting the bottleneck to representation-level reranking rather than vocabulary alignment.
Granite-20B-FunctionCalling shows that explicit training across seven granular subtasks—nested calls, chaining, parallel functions, name detection, parameter detection, next-best function, and response generation—generalizes better than umbrella datasets like ToolLLM. This multi-task approach closes the performance gap with GPT, Claude, and Gemini.
Larger models succeed at rare tasks not because they can represent solutions smaller models cannot, but because abundant capacity weakens gradients on common tasks, preventing them from overwriting slowly-accumulating rare-task features. Data-mixture design may be cheaper than scaling.
SLMs handle the repetitive, well-defined language tasks that constitute most agent work at 10–30× lower cost than LLMs, making heterogeneous architectures (SLMs by default, LLMs selective) the economically rational design pattern.
Small models fine-tuned via DPO on correct and incorrect function-calling examples from a large teacher model achieve high accuracy on logical and mathematical tasks. DPO's explicit negative examples directly target the rigid output format failures where SFT alone underperforms.
Show all 12 sources
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.
The Thread Inference Model demonstrates that reasoning structured as recursive subtask trees with rule-based KV cache pruning sustains accurate reasoning beyond context limits, even when manipulating 90% of the cache. This enables single models to replace multi-agent systems by handling full recursive reasoning internally.
Prime Agent organizes persistent state in four levels (weights, context, persistent REPL plus subagents, disk-backed history) to let models read and write addressable state beyond their instruction stream. The approach isolates harness failures from model failures and reported gains on ARC-AGI-3, though specific components remain unablated.
MeMo trains a dedicated memory model to encode new knowledge, eliminating inference-time search costs that scale with corpus size. It avoids fine-tuning risks and works with frozen proprietary models, but trades this for up-front training cost and capacity limits.
Metis demonstrates that agent memory can be implemented as a persistent state and autonomous procedures within the model backbone rather than external modules. This approach enables end-to-end training and avoids the decoupling failures where external memory and backbone optimize independently.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
MemTrapBench shows that all five tested memory frameworks underperform a no-memory baseline, with drops exceeding 10%, despite memories being accurately stored and task-relevant. This reveals a failure mode at the point of use that standard memory benchmarks miss.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents
- MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
- Metis: Memory Foundation Model
- Memorization and Knowledge Injection in Gated LLMs
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
- Improving Small-Scale Large Language Models Function Calling for Reasoning Tasks
- Provable Benefits of In-Tool Learning for Large Language Models