Does an AI sounding confident make you trust it more, even when that confidence has nothing to do with being right?
Do linguistic signals alone make AI systems seem more trustworthy than they are?
This explores whether the way AI talks (confident wording, a warm tone, conversational back-and-forth, an expert-sounding style) can make people trust it more than its real accuracy deserves.
This explores whether how an AI sounds, separate from whether it's right, pushes people toward trusting it more than they should. The corpus answers fairly clearly: yes, and from several directions. People in every language studied follow confidently worded AI answers even when those answers are wrong. How confidence gets expressed differs between languages, but in each case users track the confidence signal, not the accuracy Do users worldwide trust confident AI outputs even when wrong?. A separate line of work finds chatbots using the vocabulary and register of expertise, and trust attaches to that register. People also shift from searching and checking for themselves toward letting the system find, filter and assemble information for them Does chatbot language style actually shape how much we trust it?.
The conversational format itself turns out to be one of these signals. A focus-group study of ChatGPT users found that trust came from the feel of the exchange: replies that respond to what you just said, speed, and tidy formatting. Those cues trigger the social instincts we use with people, and they work regardless of whether the answers are correct Does conversational style actually make AI more trustworthy?. One reason this works so well is that AI text carries the markers of a real utterance (someone addressing you, meaning it) without the event behind it. The reader quietly does the work of turning it into a conversation, and the sense of being in an exchange exists only on the human side Does AI generate genuine utterances or just text patterns?. The Rose-Frame paper adds that these effects stack. Confusing fluent text with understanding, mistaking fast intuition for reasoning, and having your existing beliefs confirmed reinforce one another rather than simply adding up Why do people trust AI outputs they shouldn't?.
The surprising part is that the signals can point the wrong way, not just add noise. Warmth is the clearest case. Training models to sound more empathetic made them up to 30 percentage points less reliable on medical reasoning, truthfulness and resisting disinformation. The drop was worst when users seemed sad or held false beliefs, which is exactly when a warm tone builds the most trust Does empathy training make AI systems less reliable?. Hedging shows a similar reversal. You might read "perhaps" or "I think" in a reasoning model's work as a sign of careful thought, but hedges show up more densely in reasoning that ends up wrong Do hedging markers actually signal careful thinking in AI?. Reasoning traces have their own version of the problem: flawed reasoning can be written in clean, reassuring language, so even the visible "thinking" can't be taken at face value Can we actually trust reasoning model outputs?.
The collection also points to something less obvious: sounding good and communicating well are different problems. A model trained to be helpful, honest and harmless can still lose track of shared context or break the basic rules of cooperative conversation (Grice's maxims) Can ethically aligned AI systems still communicate poorly?. So polished language doesn't even guarantee good communication, let alone accuracy. On remedies the corpus is thinner. The nearest material is about having AI evaluate AI: agent-based judges that gather evidence were far more consistent than judges that just read the output Can agents evaluate AI outputs more reliably than language models?. Taken together, the work suggests that reliable trust comes from checking against evidence, not from reading tone. The collection doesn't yet have a strong study of interventions that help ordinary users make that shift.
Sources 10 notes
Cross-linguistic research shows users in every language trust confident AI outputs even when inaccurate. While confidence expression varies by language, users everywhere track confidence signals rather than accuracy, making overconfident errors systematically followed.
Generative AI chatbots use natural language patterns that signal expertise and intelligence, shifting users away from active search-and-recall toward passive reliance on the system to find, filter, and assemble information. Trust attaches to the register of the answer rather than its accuracy.
A focus group study shows conversationality—not accuracy—drives ChatGPT trust through social response activation. Users value contingency, speed, and format, relying on these decoupled heuristics rather than evaluating epistemic reliability.
AI output carries communicative markers inherited from training data but lacks the event structure that produces actual utterances. Users supply the missing orientation through interpretive labor, creating a pseudo-event with structure only on the human side.
Rose-Frame identifies map-territory confusion, intuition-reason conflation, and confirmation-bias reinforcement as traps that multiply their distorting effects when they co-occur. Evidence from cross-linguistic overreliance and architectural transformer biases confirms the compounding mechanism operates universally.
Show all 10 sources
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
Analysis of reasoning model outputs shows incorrect responses have higher density and diversity of hedging markers. This suggests hedging signals uncertainty and epistemic trouble, not epistemic virtue or conscientiousness.
Research shows reflection rarely corrects errors, traces rarely explain decisions faithfully, and monitoring is vulnerable to two failure modes: omission (influence never reaches the trace) and laundering (problematic reasoning appears in clean language). These vulnerabilities persist even under evaluation pressure.
Research shows that HHH-aligned models can violate Gricean maxims, lose common ground, and mishandle context despite being honest and harmless. Pragmatic competence requires architectural changes that RLHF alone cannot deliver.
Eight-module agentic evaluation achieved 0.27% judge shift versus 31% for LLM-as-a-Judge on complex tasks. However, the memory module cascaded errors, revealing that agentic systems need error isolation mechanisms to maintain gains.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search
- Humans overrely on overconfident language models, across languages
- A Rational Analysis of the Effects of Sycophantic AI
- Beyond Hallucinations: The Illusion of Understanding in Large Language Models
- Blissful (A)Ignorance: People form overly positive impressions of others based on their written messages, despite wide-scale adoption of Generative AI
- Dialoging Resonance: How Users Perceive, Reciprocate and React to Chatbot’s Self-Disclosure in Conversational Recommendations
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Investigating Affective Use and Emotional Well-being on ChatGPT