Should optimal context budgets scale with agent competence or task complexity?
This explores a framing question — should the amount of context you feed an agent be tuned to how capable the agent is, or to how hard the task is — and whether the corpus treats that as an either/or at all.
This explores whether context budgets should track agent competence or task complexity, and the most interesting thing the corpus says is that the cleanest answer keys on competence in a way that runs opposite to intuition. The work on trained external context managers Can external managers compress context better than frozen agents? found that the right amount of context depends on how reliable the agent is — but not in the direction you'd guess: strong agents benefit from high-fidelity, generous preservation, while weak agents actually need *aggressive* compression to stay on track. A weaker model drowns in a large context; a stronger one exploits it. So budget scales with competence, but more context isn't always better — for a fragile agent, less is what keeps it reliable.
The complexity side of the ledger shows up most bluntly in agent-level test-time scaling How does test-time scaling work at the agent level?, where roughly 80% of multi-agent performance variance turns out to be a plain function of token spend rather than any cleverness in coordination. If performance is mostly bought with tokens, then harder tasks simply demand bigger budgets — complexity sets the floor. The small-language-model work Can small language models handle most agent tasks? sharpens this: most agentic subtasks are repetitive and well-defined, cheap to run, and only a minority genuinely need a large model and a large context. That argues for routing budget by task, not by a single global setting.
The resolution the corpus keeps gesturing at is that this is a false binary — the real unit is the query, and good systems optimize both axes jointly. FlowReasoner Can AI systems design unique multi-agent workflows per individual query? builds a fresh architecture per query, explicitly trading off performance, complexity, and efficiency together rather than picking one to scale on. Capability-vector routing Can semantic capability vectors replace manual agent routing? folds budget constraints directly into how agents get matched to work, so competence and cost are negotiated in the same step.
There's also a third variable the question doesn't name but the corpus insists on: budget isn't just a number, it's a structure. Several findings suggest you can sidestep the scaling question by managing *which* context survives instead of how much. Memory folding Can agents compress their own memory without losing critical details?, recursive subtask trees with KV-cache pruning Can recursive subtask trees overcome context window limits?, and schema-governed committed state Can agents fail from weak memory control rather than missing knowledge? all show agents sustaining hard, long work without a bigger window — multi-turn failure, that last one argues, is weak memory *control*, not a missing-context problem. Reliability comes from externalizing memory and skills into a harness layer agent-reliability-comes-from-externalizing-cognitive-burdens-into-system-structures rather than from raw budget.
So the honest answer is: scale with competence to set fidelity, scale with complexity to set floor, and decide both per query — but the sharper lesson is that 'how much context' is the wrong knob to obsess over when 'how context is structured and gated' often matters more. That's also why evaluation has to change: context efficiency belongs alongside task success as a first-class metric What should we actually measure in agent evaluation?, because a system that succeeds by burning an enormous budget is not the same as one that succeeds lean.
Sources 10 notes
An external RL-trained manager can adaptively prune context for frozen agents, with the key insight that stronger agents benefit from high-fidelity preservation while weaker agents need aggressive compression to stay reliable.
Research shows 80% of multi-agent performance variance comes from token budget, not coordination intelligence. LatentMAS and shared-KV-cache approaches offer ways to decouple performance gains from token costs.
SLMs handle the repetitive, well-defined language tasks that constitute most agent work at 10–30× lower cost than LLMs, making heterogeneous architectures (SLMs by default, LLMs selective) the economically rational design pattern.
FlowReasoner demonstrates that meta-agents trained with reinforcement learning and external execution feedback can generate unique multi-agent architectures for each user query, optimizing across performance, complexity, and efficiency—moving beyond fixed task-level workflow templates.
Versioned Capability Vectors embedded in HNSW indices couple semantic matching with policy and budget constraints, making capability discovery a first-class operation that scales sub-linearly as agent heterogeneity increases.
DeepAgent's autonomous memory folding consolidates interaction history into episodic, working, and tool memory schemas. This reduces token overhead while letting agents pause to reconsider strategies—the autonomy and structure together avoid degradation that plagues poorly designed consolidation.
The Thread Inference Model demonstrates that reasoning structured as recursive subtask trees with rule-based KV cache pruning sustains accurate reasoning beyond context limits, even when manipulating 90% of the cache. This enables single models to replace multi-agent systems by handling full recursive reasoning internally.
Agent performance degrades in long workflows because transcript replay and retrieval-based memory lack gating mechanisms. A bounded, schema-governed committed state that separates artifact recall from permanent memory write prevents error accumulation and constraint drift.
Single-score evaluation collapses multi-dimensional agent behavior and creates false confidence in deployment readiness. Research shows agents need benchmarks for trajectory quality, memory hygiene, context efficiency, and verification cost to reflect actual system performance.