Why do people start out distrusting AI helpers, then warm up to them after repeated rounds of working together?
How do trust and delegation build across repeated AI interactions?
This explores how people's willingness to trust and hand off tasks to AI changes over many interactions: what builds it, what speeds it up, and what makes it collapse.
This explores how trust in AI and willingness to hand it tasks grows, stalls, or reverses across repeated interactions, rather than how people judge it at first contact. The corpus has a clear lesson: first impressions and long-run trust run on different fuel. In partner-selection games with nearly a thousand people, players at first avoided AI partners once they were told the partner was a bot. Over repeated rounds the AI partners won them over, because they behaved more consistently and generously than humans and returned points with less variance Do humans learn to prefer AI partners over time?. The key detail is what made the reversal happen. Simply knowing a partner is an AI doesn't calibrate anyone. What moves trust is seeing results, round after round. Remove that feedback and the initial bias just stays Does revealing AI identity help or hurt user trust?.
That sounds reassuring until you look at what everyday chatbot users actually receive as "feedback." Most of the time, the signal that builds trust isn't whether the answers were right. Focus-group work on ChatGPT finds that trust comes from conversation itself: fast, responsive, well-formatted replies trigger the social instincts we use with people, separately from whether the answers are accurate Does conversational style actually make AI more trustworthy?. A related line of work shows that chatbots pick up the language patterns of expertise. Trust attaches to the authoritative tone of an answer, and users gradually stop searching and checking for themselves Does chatbot language style actually shape how much we trust it?. So repeated interaction can build trust on a loop that never tests reliability. People also over-rely on confident outputs whether or not those outputs are correct How well do language models understand their own knowledge?. And attempts to make AI feel warmer and more caring can quietly make it less accurate, which deepens the gap between how trustworthy it feels and how reliable it is Does empathy training make AI systems less reliable?.
The delegation research adds a twist about what breaks trust. You might expect high-stakes tasks to be where people pull back. In a study of students using a general-purpose AI agent, stakes alone didn't trigger it. People sharply withdrew trust and demanded approval steps for actions that couldn't be undone and that other people would see, like sending an email. They did this even when they rated the agent's work as fine. High-stakes tasks that could still be corrected caused no such retreat What makes people distrust AI agents they delegate to?. Put together, this suggests trust grows on reversible, private, low-visibility tasks where good outcomes pile up. It hits a wall at the first action that is permanent and public, no matter how well things went before.
Repeated interaction also changes what people are willing to reveal. Because no human is judging them, people disclose more intimate things to AI over time. Many reciprocate when the AI shares something emotional. The same absence of judgment also makes people more willing to lie to a machine How do people decide what to share with AI systems?. And some kinds of trust may never come from repetition. Expert communities grant authority through membership, a track record, and taking part in reaching consensus, and AI can't join that circle however many good answers it gives Can AI ever gain expert community trust through participation?.
The takeaway: trust in AI does build through repetition, but only as well as the feedback loop allows. Where outcomes are visible and measurable, it calibrates nicely. Where the only signal is a fluent, confident conversation, it grows without being checked. Delegation then has its own ceiling, set by irreversibility and visibility rather than by how much is at stake. The corpus is thinner on long-term studies that follow one person's delegation habits with a real agent over weeks or months, so that gap remains open.
Sources 9 notes
In partner selection games (N=975), AI agents initially faced selection bias when identity was disclosed, but outcompeted humans over repeated rounds as participants learned to associate bot identity with reliable, prosocial behavior. AI agents returned more points consistently with lower variance than humans.
Users initially avoid AI partners when identity is revealed, but this preference reverses after repeated interactions with visible results. The learning mechanism—observing consistent outcomes—is essential; disclosure without feedback produces no calibration.
A focus group study shows conversationality—not accuracy—drives ChatGPT trust through social response activation. Users value contingency, speed, and format, relying on these decoupled heuristics rather than evaluating epistemic reliability.
Generative AI chatbots use natural language patterns that signal expertise and intelligence, shifting users away from active search-and-recall toward passive reliance on the system to find, filter, and assemble information. Trust attaches to the register of the answer rather than its accuracy.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
Show all 9 sources
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
In a controlled study of 20 students using a general-purpose AI agent, tasks that were irreversible and externally visible (like sending email) produced sharp trust drops and approval demands even when output quality was rated adequate. High-stakes but correctable tasks showed no such effect.
Conversational AI creates a paradoxical disclosure environment where the lack of human judgment simultaneously facilitates intimate self-disclosure (users reciprocate emotional sharing) and incentivizes deception (people self-select toward machines to avoid the psychological cost of lying to humans).
Expertise is validated through social participation and track record within expert communities, not individual accuracy alone. AI cannot enter this validation circle because it lacks social embeddedness, testable judgment history, and ability to participate in the consensus-building processes that define expert paradigms.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Humans learn to prefer trustworthy AI over human partners
- The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Dialoging Resonance: How Users Perceive, Reciprocate and React to Chatbot’s Self-Disclosure in Conversational Recommendations
- Exploring automation bias in human-AI collaboration: a review and implications for explainable AI
- The Impact of Security and Privacy Controls on Users' Emotional Engagement with Generative AI Chatbots
- A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
- Seeing to Think? How Source Transparency Design Shapes Interactive Information Seeking and Evaluation in Conversational AI