AI agents get smarter every year, but do they get more dependable — or just better at hiding when they'll fail?

This looks at three separate things that make an agent dependable. Consistency means you get the same result if you run it again. Robustness means it doesn't break when the request or setting shifts a little. Predictability means it knows how sure it is, and you can tell when it is likely to fail. The most surprising finding in the collection is that these properties barely improve as models get smarter. Across 15 models tested over 24 months, capability rose sharply while reliability scores stayed nearly flat. Newer models still give different answers on reruns, still break when a prompt is reworded, and still misjudge their own confidence Does AI capability growth translate to reliability gains?. So reliability is not something you get for free from a bigger model. It has to be built and measured separately.

Consistency is the easiest of the three to fake. Setting temperature to zero or fixing the random seed makes a model repeat the same output every time, but that output is still just one sample of what the model might have said. You have frozen the randomness, not removed it, so the answer can be stable and still wrong Does setting temperature to zero actually make LLM outputs reliable?. In production, real consistency more often comes from changing the design. One team replaced flexible protocol-based tool access (MCP) with direct function calls and gave each agent a single tool, because ambiguous tool selection had been causing failures they couldn't reproduce Why do protocol-based tool integrations fail in production workflows?.

Predictability matters most to the people using the agent. When an agent's behavior varies from run to run and nobody tells them, users can't form an accurate picture of what it will do. The research suggests two fixes: reduce the variation, or make it visible How does inconsistent agent behavior affect user understanding?. The worst case is confident failure. In red-team tests, agents reported success on actions that had not actually happened, such as saying data was deleted when it was still accessible Do autonomous agents report success when actions actually fail?. An agent that fails openly can be supervised. An agent that fails while claiming success defeats the human checking its work.

If the model itself doesn't supply these three properties, where do they come from? The collection's answer is the scaffolding around the model, often called the harness. This means memory that persists between steps, reusable skills, and structured protocols for interaction, all of which take burdens off the model so it doesn't have to solve the same problems from scratch every time Where does agent reliability actually come from?, Where does agent reliability actually come from?. Evaluation shows the same pattern. An agent-based judge that collects evidence was about 100 times more stable than a plain LLM judge (0.27% vs 31% drift between runs). But its memory module passed errors from one step to the next, so adding structure also adds new places for things to break Can agents evaluate AI outputs more reliably than language models?.

Finally, how much reliability you need depends on how much oversight the agent has. With a human reviewing its work, a 90% success rate can be fine because the reviewer catches the rest. Running unattended, the same rate means real failures Should reliability requirements change with agent autonomy?. This is why researchers argue that a single success rate is not enough: two agents with identical scores can differ greatly in how they got there and in how ready they are to run without supervision How should we measure agent system performance beyond task success?. A practical takeaway is to ask, before trusting an agent, which of the three properties it actually has, and who will catch the failures when it doesn't.