Which agent architectures consistently outperform base models on hard prediction questions?
This reads the question as: when agents beat the raw model they're built on, what's actually doing the work — and the corpus suggests the gains come less from any named architecture than from what gets bolted around the model.
This explores which agent setups reliably outperform their underlying base model on hard tasks — and the most useful thing the corpus says is that the question is slightly misaimed. The consistent performance gains don't come from a particular architecture diagram; they come from externalizing work the model would otherwise have to redo every turn. One synthesis argues that reliable agents push three burdens — memory (state persistence), skills (reusable procedures), and protocols (structured interaction) — out of the model and into a surrounding 'harness' layer, so the model stops re-solving the same sub-problems Where does agent reliability actually come from?. On that view, 'which architecture wins' is really 'which harness carries the most off the model's shoulders.'
The strongest concrete wins in the corpus are memory-centric. AgentFly treats learning as memory operations rather than weight updates and hits 87.88% on the GAIA benchmark without touching the model's parameters at all Can agents learn continuously from experience without updating weights?. Reflexion gets there more simply: when the environment gives a clean success/failure signal, the agent writes a verbal self-diagnosis, stores it, and improves across attempts — again with no retraining, and the binary signal is what keeps it from rationalizing its failures away Can agents learn from failure without updating their weights?. Both outperform the base model by remembering, not by being smarter.
A second cluster says the real lever is outside the model entirely. Nex-N1 finds that agent performance scales with the *environment* — its complexity, diversity, and real-world fidelity — not model size, and that starving any one of those three dimensions collapses generalization What blocks scaling from language models to autonomous agents?. The mirror-image warning: agents trained only on static expert demonstrations are capped by what the dataset's curators imagined, because they never interact with an environment and so never learn from their own mistakes Can agents learn beyond what their training data shows?. So 'consistently outperforms' tends to track 'was allowed to fail in a rich environment,' not architecture per se.
Where structure does pay off, it's adaptivity and verification. FlowReasoner uses a meta-agent — trained with reinforcement learning on execution feedback — to generate a fresh multi-agent workflow per query rather than reusing a fixed template, optimizing performance against complexity and cost Can AI systems design unique multi-agent workflows per individual query?. And for the specific case of *judging* hard answers, an eight-module agentic evaluator that actively collects evidence cut 'judge shift' to 0.27% versus 31% for a plain LLM-as-judge — a roughly hundred-fold reliability gain — though its memory module also cascaded errors, a reminder that more structure adds new failure surfaces Can agents evaluate AI outputs more reliably than language models?.
Two caveats keep this honest. First, 'consistently outperform' is itself contested: one note argues single-score, one-shot task success creates false confidence and that you only see whether an agent really beats the baseline by measuring trajectory quality, memory hygiene, and verification cost What should we actually measure in agent evaluation?. Second, bigger isn't the answer — small models handle most agentic sub-tasks at a fraction of the cost, so the winning architectures are often heterogeneous (small models by default, large ones only where needed) rather than uniformly scaled up Can small language models handle most agent tasks?. The takeaway you didn't know you wanted: on hard problems, the agent that wins is usually the one that remembers and gets to practice — not the one with the cleverest box-and-arrow diagram.
Sources 9 notes
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
Nex-N1 shows that autonomous agent performance depends on environment scaling along complexity, diversity, and real-world fidelity — not model size. Deficits in any single dimension collapse generalization, but scaling all three together enables frontier performance.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
FlowReasoner demonstrates that meta-agents trained with reinforcement learning and external execution feedback can generate unique multi-agent architectures for each user query, optimizing across performance, complexity, and efficiency—moving beyond fixed task-level workflow templates.
Eight-module agentic evaluation achieved 0.27% judge shift versus 31% for LLM-as-a-Judge on complex tasks. However, the memory module cascaded errors, revealing that agentic systems need error isolation mechanisms to maintain gains.
Single-score evaluation collapses multi-dimensional agent behavior and creates false confidence in deployment readiness. Research shows agents need benchmarks for trajectory quality, memory hygiene, context efficiency, and verification cost to reflect actual system performance.
SLMs handle the repetitive, well-defined language tasks that constitute most agent work at 10–30× lower cost than LLMs, making heterogeneous architectures (SLMs by default, LLMs selective) the economically rational design pattern.