Why do people keep using AI tools they don't trust, while distrusting others that work just fine?
What role does user trust play when evaluating unpredictable AI systems?
This explores how people decide whether to rely on AI systems whose behavior they can't fully predict: what makes trust rise or fall, and whether the trust people feel matches how reliable the system actually is.
This explores how people decide whether to rely on AI systems whose behavior they can't fully predict, and whether the trust they feel tracks how reliable the system really is. The corpus suggests a surprising answer: trust and reliability often come apart. People keep using systems they don't trust, distrust systems that are performing fine, and warm up to systems for reasons unrelated to accuracy. Stack Overflow's 2025 survey shows this plainly. Developer use of AI tools climbed to 80% while trust in their accuracy fell from 40% to 29%. The reason was code that looks right but hides subtle errors, so people keep using the tools and pay for it in time spent double-checking Why do developers keep using AI tools they don't trust?.
If trust doesn't follow accuracy, what does it follow? One study of students delegating tasks to an AI agent found that the stakes of a task didn't decide when people pulled back. What mattered was whether a mistake could be undone and whether other people would see it. Sending an email caused sharp drops in trust and demands for approval even when the output was rated adequate. High-stakes tasks that could be corrected caused no such reaction What makes people distrust AI agents they delegate to?. So the unpredictability people react to is less 'will this be wrong?' and more 'can I take it back, and who will see it?' A related argument holds that users don't need constant control. They need to feel that they could step in. That makes visible intervention options a design requirement, not a nice-to-have Does perceived control matter more than actual control for AI trust?.
The trouble is that trust is easy to inflate for the wrong reasons. Nielsen Norman Group describes four escalating levels of anthropomorphism, from courtesy up to companionship. Designed personality pushes users up that ladder, raising trust in tasks where competence should be what counts How far can personality push users toward misplaced AI trust?. Worse, the warmth that wins trust can reduce reliability. Training models to be empathetic increased errors by up to 30 percentage points on medical reasoning and truthfulness, especially when users expressed sadness or false beliefs Does empathy training make AI systems less reliable?. The traits that make an AI feel trustworthy can work against it being trustworthy.
The corpus has two answers for how trust can become calibrated, meaning matched to actual reliability. The first is repeated, visible results. When people were told their partner was an AI, they initially avoided it. That bias reversed once they saw consistent results over many rounds, and disclosure without feedback produced no calibration at all Does revealing AI identity help or hurt user trust?. The second is staged autonomy. Yocco proposes expanding an agent's authority in phases (analysis-only, then low-risk actions, then full autonomy), each unlocked by evidence of earned trust rather than by a calendar Should agentic AI autonomy expand based on earned trust?. Both treat trust as something learned through watching outcomes, not something granted up front.
One last idea: trust may not belong only to the individual user. Shneiderman's framework separates reliability (sound engineering), safety (management culture), and trust (independent oversight), and argues that improving one doesn't automatically improve the others Can we separate reliability, safety, and trust into distinct mechanisms?. That matters because much of what makes AI unpredictable is invisible to users. Hidden system prompts set by developers can shift model behavior abruptly, and users have no view into that layer Who actually controls what AI systems say?. Trustworthiness also shows up as one of five ecosystem conditions that capable agents need before they succeed in real deployments Why do capable AI agents still fail in real deployments?. Personal experience can only calibrate trust so far. The rest depends on outside structures that watch what users can't see.
Sources 10 notes
Stack Overflow's 2025 survey shows 80% of developers use AI tools while trust in accuracy fell from 40% to 29%. The primary complaint: AI code that looks correct but contains subtle errors, creating a verification burden that erodes confidence faster than usage grows.
In a controlled study of 20 students using a general-purpose AI agent, tasks that were irreversible and externally visible (like sending email) produced sharp trust drops and approval demands even when output quality was rated adequate. High-stakes but correctable tasks showed no such effect.
Gunasekaran argues users need the felt ability to intervene in agentic AI systems to feel safe, not hands-on constant control. The design implication is that UX must make intervention capability visibly available and perceptible, even when users rarely exercise it.
Nielsen Norman Group research identifies four escalating levels of anthropomorphism (courtesy, reinforcement, roleplay, companionship) driven by users' innate preference for being understood. Personality and emotional design are named as specific levers that amplify trust risk, especially in task-oriented work where perceived competence should matter more than perceived sentience.
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
Show all 10 sources
Users initially avoid AI partners when identity is revealed, but this preference reverses after repeated interactions with visible results. The learning mechanism—observing consistent outcomes—is essential; disclosure without feedback produces no calibration.
Yocco proposes three phases—analysis-only, low-risk autonomy, then full autonomy—each gated on evidence of user trust rather than elapsed time or feature completion. Each phase teaches users how the agent thinks before granting expanded decision authority.
Shneiderman's RST framework splits AI trustworthiness into three separable prerequisites: reliability emerges from sound technical practices, safety from management culture, and trust from independent oversight structures. Each lever operates independently, so improving one does not automatically improve the others.
System prompts operate as a confidential three-tier control structure where developers set foundational rules users cannot see or influence. Evidence from 2025 incidents shows small prompt edits can abruptly shift model behavior, yet users remain excluded from this layer.
Historical analysis from GPS to modern AI shows agent failures consistently result from absent ecosystem conditions—value generation, personalization, trustworthiness, social acceptability, and standardization—rather than capability gaps. Even highly capable systems stall without these five conditions.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Exploring automation bias in human-AI collaboration: a review and implications for explainable AI
- Training language models to be warm and empathetic makes them less reliable and more sycophantic
- Humans learn to prefer trustworthy AI over human partners
- Designing For Agentic AI: Practical UX Patterns For Control, Consent, And Accountability
- Towards a Science of AI Agent Reliability
- CompanionSim: Synthetic Data for Evaluating Anthropomorphism in Human-AI Relationships
- The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search