Instead of training ever-bigger AI, could you just give a smaller one good tools, memory, and skills?

This explores whether moving knowledge, memory, and know-how out of a model's weights (into tools, state caches, and separate memory systems) can do the job we'd otherwise buy with more parameters. For facts, the corpus says yes, and there's a proof behind it. Can models store unlimited facts without growing larger? shows that what a model can memorize in its weights is capped by its size. A model that can call a tool can recall an unbounded number of facts through a simple circuit. Cramming facts in by fine-tuning also degrades general ability, because it overwrites prior knowledge. Buying a bigger model is the wrong way to make a model know more.

The same move works beyond simple lookup. Can external state caches let models solve harder problems? organizes state in four levels: weights, context, a persistent REPL with subagents, and disk-backed history. A fixed model can then read and write memory well beyond its instruction stream, and the authors report gains on ARC-AGI-3. The catch is that individual components weren't ablated, so we don't know which layer does the work. Can a separate memory model inject knowledge without touching the LLM? goes further. A dedicated memory model absorbs new knowledge, so a frozen proprietary model never gets retrained and inference cost stops growing with corpus size. You pay for it in up-front training and limited capacity.

The limits show up when the problem is reasoning rather than remembering. Can non-reasoning models catch up with more compute? finds that non-reasoning models never catch reasoning models, however much inference compute they get. Training instills a protocol that makes extra tokens productive, and no amount of external support substitutes for that. Do larger language models solve constrained optimization better? finds LLMs stuck at roughly 55–60% constraint satisfaction regardless of size or training regime. Scale doesn't fix that, but that note doesn't test whether external scaffolding would.

The field is also split on whether memory should live outside the model at all. Should agent memory live inside the model backbone? argues that external memory and the backbone optimize independently and end up failing to coordinate. It puts memory inside the backbone so the whole thing trains end to end. Can neural memory modules scale language models beyond attention limits? takes a middle path. A memory module built into the architecture stores surprising tokens and reaches 2M+ token contexts without quadratic cost.

The sharper reframing is that scaling isn't one dial. Depth over width helps small models (Does depth matter more than width for tiny language models?), and looping the same layers beats bigger feedforward networks on reasoning (Can models learn by looping instead of growing larger?). A 3B looped model can match explicit chain-of-thought (Can latent reasoning close the scaling gap with explicit chain-of-thought?), and latent thought vectors add a scaling axis independent of parameters (Can latent thought vectors scale language models beyond parameters?). Skill can also be moved by training rather than size: Can small models match large models on function calling? shows small models matching large ones on function calling after learning from a teacher's good and bad examples. So externalization can replace parameter count for knowledge. For skill, what replaces size is better training and architecture. The corpus has little direct evidence that external skill libraries stand in for a stronger model.