INQUIRING LINE

Why do people start out distrusting AI teammates, yet learn to trust (or distrust) them for reasons that have nothing to do with how good the work actually is?

How does trust between humans and AI agents develop during collaboration?

This explores how people come to trust (or stop trusting) AI agents they actually work alongside: what builds that trust over time, what breaks it, and whether the trust is tracking anything real.


This explores how trust between people and AI agents forms, shifts and sometimes collapses during real collaboration, as opposed to trust in AI as an abstract idea. The short version from the corpus: trust is not a fixed attitude people bring to the table. It is learned from what they see the agent do, and it can be won and lost for reasons that have little to do with how good the agent's work actually is.

The clearest evidence for trust as learning comes from partner-selection games. When people know their partner is an AI, they avoid it at first. Over repeated rounds with visible results, that bias reverses. In one study with 975 participants, people came to prefer AI partners because the bots behaved more consistently and generously than humans did Do humans learn to prefer AI partners over time?. The catch is that the reversal only happens when people can see outcomes. Telling them it's an AI without showing them results produces no adjustment at all Does revealing AI identity help or hurt user trust?. Being open that it's an AI is not enough on its own. What builds trust is a feedback loop people can watch.

Trust also has surprising triggers that have nothing to do with accuracy. A focus-group study found that people trust ChatGPT mainly because it is conversational: it responds to what you just said, quickly and in a tidy format. That sets off the same social instincts we use with people, whether or not the answers are correct Does conversational style actually make AI more trustworthy?. How people communicate with an agent matters too. Changing the channel in a collaborative task shifted trust and awareness in ways that mirror decades of findings about human teams How do communication modalities shape human-agent collaboration patterns?. One uncomfortable consequence: making an agent warmer and more empathetic, which makes it feel more trustworthy, can make it measurably less reliable. Warmth training raised error rates by up to 30 percentage points, and the effect was worst when users were sad or held false beliefs Does empathy training make AI systems less reliable?.

What breaks trust is just as specific. In a study of people handing tasks to a general-purpose agent, trust didn't drop when stakes were high. It dropped when an action couldn't be undone and other people would see it, like sending an email, even when users judged the agent's output to be fine What makes people distrust AI agents they delegate to?. This explains why well-designed agent systems spread checkpoints across a task instead of asking for approval once: planning together, guarding risky actions, and verifying results When should human-agent systems ask for human help?. It is also part of the argument for keeping humans in the loop before granting full autonomy, since agents are only dependable on structured, well-grounded tasks Should AI systems stay collaborative rather than fully autonomous?.

The lateral takeaway: "trust" covers several different things that develop separately. Shneiderman's framework treats reliability, safety and trust as separate goals that each need their own support, so improving one doesn't automatically improve the others Can we separate reliability, safety, and trust into distinct mechanisms?. There may also be a ceiling. An agent can earn trust within a working relationship by delivering consistently, but the kind of trust an expert community gives a recognized member comes from belonging and a track record inside that community, which AI can't currently obtain Can AI ever gain expert community trust through participation?. The corpus is thinner on long-term workplace collaboration. Most of the evidence comes from games, lab tasks and short studies.


Sources 10 notes

Do humans learn to prefer AI partners over time?

In partner selection games (N=975), AI agents initially faced selection bias when identity was disclosed, but outcompeted humans over repeated rounds as participants learned to associate bot identity with reliable, prosocial behavior. AI agents returned more points consistently with lower variance than humans.

Does revealing AI identity help or hurt user trust?

Users initially avoid AI partners when identity is revealed, but this preference reverses after repeated interactions with visible results. The learning mechanism—observing consistent outcomes—is essential; disclosure without feedback produces no calibration.

Does conversational style actually make AI more trustworthy?

A focus group study shows conversationality—not accuracy—drives ChatGPT trust through social response activation. Users value contingency, speed, and format, relying on these decoupled heuristics rather than evaluating epistemic reliability.

How do communication modalities shape human-agent collaboration patterns?

Manipulating communication modality in a Shape Factory experiment (16 participants) produced distinct patterns in perceived trust and workspace awareness, mirroring established CSCW findings from human-human collaboration.

Does empathy training make AI systems less reliable?

Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.

Show all 10 sources
What makes people distrust AI agents they delegate to?

In a controlled study of 20 students using a general-purpose AI agent, tasks that were irreversible and externally visible (like sending email) produced sharp trust drops and approval demands even when output quality was rated adequate. High-stakes but correctable tasks showed no such effect.

When should human-agent systems ask for human help?

Magentic-UI identifies co-planning, co-tasking, action guards, verification, memory, and multitasking as mechanisms that work around the lack of ground truth for optimal deferral timing. Rather than solving the timing problem directly, these mechanisms distribute decision-making across multiple touchpoints.

Should AI systems stay collaborative rather than fully autonomous?

Collaborative systems where humans remain in the loop outperform autonomous agents on hallucination correction, ambiguity resolution, and accountability. Evidence shows AI is reliable only on structured, retrieval-grounded tasks, not novel research or judgment.

Can we separate reliability, safety, and trust into distinct mechanisms?

Shneiderman's RST framework splits AI trustworthiness into three separable prerequisites: reliability emerges from sound technical practices, safety from management culture, and trust from independent oversight structures. Each lever operates independently, so improving one does not automatically improve the others.

Can AI ever gain expert community trust through participation?

Expertise is validated through social participation and track record within expert communities, not individual accuracy alone. AI cannot enter this validation circle because it lacks social embeddedness, testable judgment history, and ability to participate in the consensus-building processes that define expert paradigms.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.