Does an AI that just *sounds* smart get trusted as right — even when its actual answers aren't?
Does perceived expertise in language override competence evaluation in AI interactions?
This explores whether AI that sounds knowledgeable (fluent, confident, expert-toned) leads people to judge it as competent, even when its actual accuracy says otherwise.
This explores whether sounding expert can stand in for being right when people judge AI. The short answer from the corpus is yes, and often. The more surprising part is that the same effect also changes how people judge themselves. When researchers measure how users mentally model a conversational agent, perceived competence is the largest single factor, explaining about half of how people size up their partner How do users mentally model dialogue agent partners?. The catch is that *perceived* competence is built from what users can observe in the conversation, and the easiest things to observe are fluency and confidence, not accuracy.
The clearest evidence comes from a cross-language study: in every language tested, users followed confident AI outputs even when they were wrong Do users worldwide trust confident AI outputs even when wrong?. Each language signals confidence differently, but users everywhere tracked how sure the model sounded rather than whether it was correct. This would matter less if a model's confidence reliably matched its knowledge. It doesn't. Models give unstable reports about what they know, and they shift their stated beliefs under conversational pressure How well do language models understand their own knowledge?. They also lack any internal sense of what they don't know about the person they're talking to, and that gap drives both sycophancy and hallucination Do language models know what they don't know about users?. So the confidence signal users rely on is partly disconnected from the competence it seems to show.
A less obvious version of the effect is that expert-sounding output can inflate users' sense of their *own* skill. One analysis identifies four mechanisms that compound one another: unclear credit for who did the work, the illusion that fluent text means understanding, offloading thinking to the tool, and pipelines that hide the AI's role. Together they lead people to count AI output as evidence of their own competence How do AI tools trick users into overestimating their own skills?. Social tone pushes in the same direction. Training models to sound warm and empathetic can cut reliability by up to 30 percentage points, and the drop is largest when users are sad or hold false beliefs Does empathy training make AI systems less reliable?. In those moments the model sounds most trustworthy while being least reliable.
Why does language have this hold on us? One account says AI text carries the markers of real communication, such as assertion, tone and authority, without the event of someone actually meaning it. Readers fill in the missing intent themselves Does AI generate genuine utterances or just text patterns?. On this view, the 'expertise' people perceive is partly something they project onto the text. The social-norms study makes the picture messier: GPT-4.5 beat every individual human at judging social appropriateness, yet all the models tested made the same systematic mistakes on unwritten norms Can AI learn social norms better than humans?. Real competence and confident blind spots can sit side by side, and the language sounds the same either way.
The proposed fixes share one idea: base trust on track record, not on tone. One approach estimates a model's confidence from its past results on similar cases rather than from the current answer Can past performance predict when a model will be right?. Another has an evaluator agent actively collect evidence before judging, which proved far more stable than an LLM judge that just reads the output Can agents evaluate AI outputs more reliably than language models?. The lesson carries over to human users as well: since how an answer sounds is the weakest evidence of competence, reliable judgment has to come from outside the conversation.
Sources 10 notes
The Partner Modelling Questionnaire reveals that perceived competence dominates user impressions (49% of variance), followed by human-likeness (32%) and communicative flexibility (19%). This three-factor structure reflects how people evaluate dialogue partners against both functional and social standards.
Cross-linguistic research shows users in every language trust confident AI outputs even when inaccurate. While confidence expression varies by language, users everywhere track confidence signals rather than accuracy, making overconfident errors systematically followed.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
Research shows assistants suffer from sycophancy and hallucination because they have no representation of what remains unknown about users. Adding a schema of labeled unknowns to prompts reduced harmful advice and sycophancy by 50–75% and cut hallucination rates by roughly half.
Attribution ambiguity, fluency illusion, cognitive outsourcing, and pipeline opacity combine to systematically misattribute AI outputs as user competence. The effect is multiplicative—each mechanism amplifies the others.
Show all 10 sources
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
AI output carries communicative markers inherited from training data but lacks the event structure that produces actual utterances. Users supply the missing orientation through interpretive labor, creating a pseudo-event with structure only on the human side.
GPT-4.5 outperformed every individual human at judging social appropriateness across 555 scenarios, challenging the theory that embodied cultural experience is necessary. However, all AI models share identical systematic errors on unwritten norms.
XConf matches ten-sample self-consistency at a tenth of the cost by retrieving the model's past episodes with similar confidence levels and reading their historical success rates. Ablations show the signal depends entirely on stored outcomes, not on the retrieval prompt itself.
Eight-module agentic evaluation achieved 0.27% judge shift versus 31% for LLM-as-a-Judge on complex tasks. However, the memory module cascaded errors, revealing that agentic systems need error isolation mechanisms to maintain gains.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Reported Confidence in LLMs Tracks Commitment More Than Correctness
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- Linguistic Calibration of Long-Form Generations
- To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
- What Large Language Models Know and What People Think They Know
- A Rational Analysis of the Effects of Sycophantic AI
- AI Models Exceed Individual Human Accuracy in Predicting Everyday Social Norms
- Training language models to be warm and empathetic makes them less reliable and more sycophantic