Can a small AI model, trained to copy a big one, still act as a general-purpose agent — or only handle narrow, routine tasks?
Can distilled models maintain general agentic capability at very small scale?
This explores whether a very small model, trained to imitate a larger one, can still work as a general-purpose agent, or whether small models only work as narrow specialists inside a bigger system.
This explores whether a very small model, trained to imitate a larger one, can still work as a general-purpose agent, or whether small models only work as narrow specialists inside a bigger system. One caveat up front: the collection has no paper that directly measures distillation at tiny scale. What it does have is a set of findings that, read together, give a fairly clear answer. Small models can carry a large share of agent work. They are unlikely to carry *general* agentic capability on their own, and the reasons have less to do with the model than you might expect.
The optimistic case is economic. Most of what an agent actually does is repetitive, well-defined language work: parsing, formatting, routing, filling in tool calls. Small models handle these tasks at 10–30× lower cost Can small language models handle most agent tasks?. The suggested design runs small models by default and calls a large model only when needed. In other words, the realistic goal for a small model is to be a capable worker, not a general agent. A separate line of work points the same way. A lot of agent reliability doesn't live in the model at all. It comes from memory, reusable skills, and structured protocols built around the model in a 'harness' Where does agent reliability actually come from?. Improving that harness can lift frozen models substantially without changing any weights Can execution harnesses lift model performance without retuning weights?. If the scaffolding carries the load, a small model needs less general skill.
The catch, and probably the most useful finding here: models gain the most from scaffolding in the middle of the size range Do stronger models always evolve harnesses better?. The weakest models often fail to invoke the tools and structures meant to help them. So the strategy of a small model plus a strong harness has a lower limit. Below some size, the model can't use the support it's given.
Distillation also carries a built-in ceiling. A distilled model learns from a teacher's outputs, which makes it a close relative of agents trained on fixed expert demonstrations. Those agents can only be as competent as the scenarios their curators imagined, because they never learn from their own failures in a live environment Can agents learn beyond what their training data shows?. A related result shows that post-training narrows the range of solutions a model can find. It gets better at the easy cases but loses rare solutions that were still reachable Do base models find more solutions than post-trained ones?. Squeezing a large model into a small one plausibly makes that narrowing worse. The student keeps the teacher's typical behavior and loses its range. One possible way out is training against simulated environments, such as language-based 'world models', which give a small model interaction experience rather than just imitation Can language models learn to simulate agent environments?.
The surprise is that 'general agentic capability' may be the wrong target at any scale. Even large single agents run into limits that more capability can't fix. Tasks that need mixed expertise, parallel work, and independent checking have to be split across specialized agents Do single agents always hit organizational limits?. If even frontier models need a team, the better question for small distilled models is which seat on the team they can fill well. The corpus suggests that's quite a few seats, provided the model is big enough to use its tools.
Sources 8 notes
SLMs handle the repetitive, well-defined language tasks that constitute most agent work at 10–30× lower cost than LLMs, making heterogeneous architectures (SLMs by default, LLMs selective) the economically rational design pattern.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
Show all 8 sources
Across 14 model pairs and three agentic benchmarks, base models equipped with only relaxed system prompts eventually surpass post-trained counterparts in pass@K coverage as rollout budget grows. Post-training bimodalizes task outcomes, sharpening performance on easy cases while eliminating rare-but-reachable solutions.
Qwen-AgentWorld demonstrates that native language world models trained via next-state prediction on 10M+ trajectories outperform real-environment training on three benchmarks and transfer across seven domains, positioning next-state prediction as a foundation objective for agents.
Research shows that real-world tasks requiring heterogeneous expertise, parallel execution, and independent verification exceed what any single agent loop can organize. Graph-based system abstractions are needed to distribute intelligence across specialized agents.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Sharpening Tax in Post-Training
- Towards a Science of Scaling Agent Systems
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- Survey on Evaluation of LLM-based Agents
- Evaluating Language-Model Agents on Realistic Autonomous Tasks
- Agent Learning via Early Experience
- Towards a Science of AI Agent Reliability