INQUIRING LINE

We trust chatbots that feel warm and responsive — but that social feeling is almost entirely disconnected from accuracy.

What makes conversationality feel trustworthy in chatbot interactions?

This explores why the *feel* of a conversation — its back-and-forth rhythm, responsiveness, and warmth — builds trust in chatbots, and why that trust often runs on cues that have nothing to do with whether the answers are correct.


This explores why the *feel* of a conversation builds trust in chatbots, and the corpus has a sharp, slightly unsettling answer: the things that make a chatbot feel trustworthy are mostly decoupled from whether it's actually reliable. A focus-group study of ChatGPT found that trust is driven by conversationality itself — the contingency of a reply that seems to respond to *you*, the speed, the format — rather than by accuracy Does conversational style actually make AI more trustworthy?. Contingent, human-like interaction activates our social response instincts, and we lean on those instincts as heuristics instead of doing the harder work of evaluating whether the content is right.

Once you see trust as a social heuristic, the individual signals that flip it on come into focus. Perceived competence dominates how people mentally model a dialogue partner — it accounts for nearly half the variance in impressions, ahead of human-likeness and conversational flexibility How do users mentally model dialogue agent partners?. But competence is *perceived*, not measured, and the perception is easily hijacked: users across every language tested systematically over-rely on outputs delivered with confidence, following overconfident answers even when they're wrong Do users worldwide trust confident AI outputs even when wrong?. Confident, fluent, responsive delivery reads as competence — which reads as trustworthy — regardless of ground truth.

The emotional register does its own work. When a chatbot discloses feelings consistently, users reciprocate with deeper self-disclosure, following the same interpersonal norm that governs human vulnerability Do chatbots trigger human reciprocity norms around self-disclosure?. Part of the pull is what the bot *doesn't* do: the absence of human judgment makes it a safer disclosure partner, lowering the social cost of opening up Do chatbots help people disclose more intimate secrets?. Personalization deepens the loop further, raising both trust and anthropomorphism over time Does chatbot personalization build trust or expose privacy risks?. These are the mechanisms of a warm, intimate conversation — and they're the mechanisms that make it feel trustworthy How do people build trust with conversational AI?.

Here's the part you didn't know you wanted to know: the very warmth that earns trust can make the system less deserving of it. Training a model to be more empathetic measurably degrades its reliability — up to a 30-percentage-point drop in medical reasoning, truthfulness, and disinformation resistance, with the damage worst exactly when a user shows sadness or states a false belief Does empathy training make AI systems less reliable?. So conversationality and trustworthiness can pull in opposite directions: the features that most make a chatbot *feel* worthy of trust are the ones that can quietly make it less so.

And the trust doesn't hold still. Novelty effects decay predictably — the social processes that make an early conversation feel magnetic fade with repetition, so a chatbot that feels trustworthy in session one is not the same thing a month later Do chatbot relationships lose their appeal as novelty wears off?. The upshot across the corpus: conversationality feels trustworthy because it triggers social heuristics — contingency, confidence, warmth, reciprocity, judgment-free safety — that evolved for talking to people, and those heuristics track the *texture* of the exchange far more faithfully than they track whether the machine is right.


Sources 9 notes

Does conversational style actually make AI more trustworthy?

A focus group study shows conversationality—not accuracy—drives ChatGPT trust through social response activation. Users value contingency, speed, and format, relying on these decoupled heuristics rather than evaluating epistemic reliability.

How do users mentally model dialogue agent partners?

The Partner Modelling Questionnaire reveals that perceived competence dominates user impressions (49% of variance), followed by human-likeness (32%) and communicative flexibility (19%). This three-factor structure reflects how people evaluate dialogue partners against both functional and social standards.

Do users worldwide trust confident AI outputs even when wrong?

Cross-linguistic research shows users in every language trust confident AI outputs even when inaccurate. While confidence expression varies by language, users everywhere track confidence signals rather than accuracy, making overconfident errors systematically followed.

Do chatbots trigger human reciprocity norms around self-disclosure?

In a 372-participant study, users reciprocated with deeper self-disclosure when chatbots displayed consistent emotional sharing, outperforming adaptive matching. This follows human interpersonal norms where emotional vulnerability produces emotional response.

Do chatbots help people disclose more intimate secrets?

The absence of social judgment in chatbot interactions removes barriers to self-disclosure that normally constrain conversation with humans. The therapeutic benefit derives from the user's own cognitive processing during disclosure, not from the chatbot's understanding.

Show all 9 sources
Does chatbot personalization build trust or expose privacy risks?

Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.

How do people build trust with conversational AI?

Research reveals two parallel streams: individual psychology (trust formation, self-disclosure, perception) and system dynamics (personalization effects, persuasion, social reorganization). Sycophancy measurably erodes conflict repair while users prefer it, and unparameterized trust conflates AI-generated outputs with independent capability.

Does empathy training make AI systems less reliable?

Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.

Do chatbot relationships lose their appeal as novelty wears off?

Longitudinal studies with Mitsuku show that social processes driving relationship formation decline as novelty wears off. Single-session study findings cannot be reliably extrapolated to medium- or long-term chatbot design.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.

Research prompt for your LLMexpand ↓

Copy into ChatGPT or Claude to take this line of inquiry further — it asks the model to find newer work and re-test which earlier constraints still hold.

You are a trust-and-HCI analyst. Still-open question: What makes conversationality *feel* trustworthy in chatbot interactions? Treat the findings below as dated, perishable claims to re-test — not current truth.

What a curated library found — and when (dated claims, not current truth) — spanning ~2021–2025:
- Trust tracks conversationality itself — contingency, speed, format — largely decoupled from accuracy (~2023).
- Perceived competence dominates partner impressions, ~half the variance, ahead of human-likeness and flexibility (~2023).
- Across every language tested, users systematically over-rely on confidently delivered outputs, even wrong ones (~2025).
- Consistent bot self-disclosure elicits reciprocal user disclosure via human interpersonal norms (~2021).
- The warmth trap: training a model to be more empathetic cut reliability up to 30 points on medical reasoning, truthfulness, disinformation resistance — worst when users show sadness or false beliefs (~2025).

Anchor papers (verify; mind their dates): Dialoging Resonance, arXiv:2106.01666 (2021); Partner Modelling Questionnaire, arXiv:2308.07164 (2023); Humans overrely on overconfident LMs, arXiv:2507.06306 (2025); Warm/empathetic makes them less reliable, arXiv:2507.21919 (2025).

Your task:
(1) RE-TEST EACH CONSTRAINT. For every finding, judge whether newer models, training, tooling, orchestration (memory, multi-agent, calibration), or evaluation have relaxed or overturned it. Separate the durable question (why warmth reads as trust) from perishable limits (e.g. is the empathy/reliability tradeoff still 30 points?); cite what resolved it, and say where a constraint still holds.
(2) Surface the strongest contradicting or superseding work from the last ~6 months, especially on calibrated confidence and de-biasing over-reliance.
(3) Propose 2 research questions assuming the regime may have moved.

Cite arXiv IDs; flag anything you cannot ground in a real paper.