If an AI assistant says 'I'm 90% sure,' does that actually help you know when to double-check it — or just make you trust it more?

This explores whether putting confidence signals in front of users helps them trust an agent the right amount, neither too much nor too little. The short answer: the corpus has no study that tests confidence displays head-on. What it does have is a consistent pattern across neighboring trust signals, and that pattern is a strong warning. Signals meant to help people calibrate trust often turn into shortcuts that stand in for checking whether the agent is actually right.

The closest material treats confidence as one of three separate trust patterns, alongside observability (showing the process) and citations (showing sources). Each one answers a different question: how it worked, how sure it is, and where the information came from Should agent trust signals stay separate or merge together?. That same source makes the key point: building these patterns doesn't guarantee they calibrate trust accurately. Design guides like the AI UX Playground list similar verifiability patterns, and the most relevant one here is failure disclosure, which plainly states what the system can't do. That's a cousin of confidence display that points toward limits rather than certainty Do UI patterns for verifiability actually build user trust?.

The worry comes from what happens to other signals. In 24,000 search interactions, irrelevant citations raised user preference almost exactly as much as relevant ones did. People counted the citations and didn't read them Do users trust citations more when there are simply more of them?. Agent plans worked the same way: plausible but wrong plans fooled users as well as correct ones, because a clean logical structure *looks* trustworthy Does letting users edit agent plans improve trust calibration?. Conversational style builds trust in ChatGPT whether or not the answers are accurate Does conversational style actually make AI more trustworthy?. If a confidence number behaves like these signals, people will read "92%" as a sign of competence and won't treat it as a probability to reason with.

A confidence display only helps if the number tracks something real. Agents built on foundation models give different outputs on different runs, and users can't form accurate mental models unless that variation is reduced or made visible How does inconsistent agent behavior affect user understanding?. A confidence signal could be one way to make the variation visible, or it could hide it behind a number that looks precise. Format matters too. Argument-map explanations improved trust calibration on verbal tasks but hurt it on visual ones, and users' satisfaction ratings ran in the opposite direction from their actual calibration Do visual rationales help or hurt how people calibrate trust?. So users may like a confidence display that is actually making their judgment worse.

The surprising point is that people may not need confidence signals much at all for the moments that matter. In one agent study, trust dropped sharply on irreversible, externally visible actions like sending an email, even when output quality was fine. Stakes alone didn't trigger it What makes people distrust AI agents they delegate to?. Travelers similarly pulled back on high-stakes visa questions without being prompted Do travelers trust AI differently for brainstorming versus visa tasks?. People already apply their own sense of risk. A confidence display risks overriding that useful caution with an agent's own self-assessment, which may be poorly calibrated.