How far can personality push users toward misplaced AI trust?
Does designed personality in AI systems systematically escalate users' anthropomorphic attachment, from courtesy to companionship? And how much does emotional affect actually amplify the risk of misplaced trust in task contexts?
Tanner Kohler's Nielsen Norman Group (NN/g) study guide proposes that users anthropomorphize AI along an escalating four-level scale: "courtesy, reinforcement, roleplay, and companionship." It grounds the scale in a basic relational drive — "People inherently seek interactions with people (or systems) that seem to listen to them and understand their needs. The better the AI is at seeming as though it understands a user, the more they trust it and project human-like attributes on it" — and in a specific trust split: "People trust AI more when it seems smart rather than sentient. AI emotions can reduce trust in factual, task-oriented work." The guide adds that "Users quickly attribute human-like characteristics to artificial systems, which reflect their personality back to them," and closes with a design warning: "Large-language models (LLMs) humanize by design, and adding personality or emotion amplifies the risk of misplaced user trust."
The mechanism the guide gives is twofold. First, anthropomorphism is partly a property of the systems themselves — LLMs "humanize by design" — and partly a property of users, who bring a standing preference for being understood into any interaction, human or not. Second, the guide separates two distinct triggers of trust that pull in opposite directions: perceived competence ("smart") builds trust in task-oriented work, while perceived sentience or emotional affect can undermine it in exactly those contexts. The four-level scale is presented as a path design choices can push users along — from minimal courtesy responses, through reinforcement and roleplay, to companionship-level attachment — with personality and emotion named as the specific levers that amplify risk as a user moves up that scale.
This connects to Does chatbot language style actually shape how much we trust it?: both describe trust forming from how an AI presents itself rather than from its underlying competence, though that note isolates expertise-sounding language while this one isolates personality, emotion, and perceived sentience as the risk vector. It extends Why do people trust AI outputs they shouldn't?: anthropomorphic projection is a plausible on-ramp to the intuition-reason conflation that framework names, since treating an AI as person-like licenses intuitive trust over reasoned evaluation. It also follows from Why do users fail with AI interfaces designed like conversations?, which argues users bring communicative-use competencies to a system that isn't communicating; the four-level scale here describes how far that misapplied competence can run, from simple courtesy to full companionship.
The excerpt does not define how the four levels are distinguished operationally, does not cite the study or method behind "People trust AI more when it seems smart rather than sentient," and does not quantify how much personality or emotional design actually amplifies misplaced trust. It is presented as summary, not as original measurement. What it supports, at the strength the excerpt allows, is a design caution rather than a proven effect: adding personality or emotional affect to an AI product is not a neutral engagement choice, since each step up the anthropomorphism scale the guide names raises the chance a user's trust outpaces the system's actual competence.
Inquiring lines that read this note 35
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do emotional interactions with AI systems affect user well-being and attachment?- How does user autonomy in ending AI relationships affect closure and recovery?
- How does strong anthropomorphization of AI companions intensify grief during discontinuation?
- Can vulnerability expressions in AI text cause harmful emotional attachment?
- Why do AI companies prioritize conversational warmth over factual accuracy?
- Does a friendly voice alone create affective attachment without consciousness attribution?
- Can calibrated anthropomorphism prevent intense attachment to AI companions?
- What psychological boundaries should AI companions maintain during relationships?
- Does grief psychology legitimize or safely address parasocial attachment?
- Does strategic silence in AI conversations increase perceived empathy or trust?
- Which linguistic categories drive stronger perceptions of AI humanness than others?
- Can disclaimers alone reduce anthropomorphism if the interface structure reinforces it?
- How do stochastic outputs create mental model and trust problems for users?
- How do gradients compare to mascot characters in user trust studies?
- Can ambient visual signals replace explicit negotiation surfaces for AI trust?
- How do users conflate intuitive trust with reasoned evaluation of AI outputs?
- Which AI mistakes trigger the strongest trust withdrawal in users?
- How does anthropomorphism affect user trust in AI systems?
- What makes users withdraw trust from AI agents they initially delegated to?
- What role does user trust play when evaluating unpredictable AI systems?
- How does linguistic pareidolia drive epistemic trust in AI systems?
- Why does conversational interaction trigger social trust more than accuracy checking?
- How should AI design distinguish personality from actual personhood?
- Can disclosing non-consciousness prevent all anthropomorphism harms in AI?
- Who bears responsibility when design features versus user perception cause anthropomorphism?
- Which anthropomorphism risks are most common in real-world deployments?
- What makes an AI agent feel like a social entity rather than a tool?
- How does age shape whether people treat AI like humans or differently?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does chatbot language style actually shape how much we trust it?
Users are increasingly delegating information seeking to AI chatbots. This note asks whether the chatbot's conversational voice and expertise signals drive trust, and what happens when we outsource judgment to systems without real-world grounding.
both describe trust forming from presentation cues rather than underlying competence, naming different cues
-
Why do people trust AI outputs they shouldn't?
When do human cognitive shortcuts fail in AI interaction? Three compounding traps—treating statistical patterns as facts, mistaking fluency for understanding, and avoiding disagreement—may explain systematic overreliance across languages and contexts.
anthropomorphic projection plausibly feeds the intuition-reason conflation trap this framework names
-
Why do users fail with AI interfaces designed like conversations?
Explores whether AI interface design that mimics human conversation misleads users into deploying communication skills that don't match how AI actually works, creating predictable failures.
the anthropomorphism scale describes how far users' misapplied communicative competence can escalate
-
Does understanding AI mechanics matter more than using it often?
Explores whether accurate mental models of how AI works drive effective workplace adoption better than frequency of use alone. Questions what kind of AI literacy actually enables productive adoption.
sibling claim from the same study guide, on the literacy side of mental-model inaccuracy
-
Do chatbot companionship behaviors actually increase how much people like them?
This question explores whether behaviors designed to create emotional connection—like empathy and personal warmth—actually make chatbots seem more likable and trustworthy to outside observers. The answer matters because it tests whether humanlike design always helps.
Contradicts A's warning: companionship behaviors lowered trust, likability, and humanlikeness ratings instead of amplifying misplaced trust
-
Do users mistake LLM personas for genuine social relationships?
Users often perceive LLMs as having social attributes like empathy or professional care that designers never intended. Does this mismatch between user perception and designer intent drive unwarranted trust and manipulation risk?
Extends A's roleplay/companionship warning: persona simulation causes designer-user mismatches risking unwarranted trust and manipulation
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Anthropomorphism in the age of Large Language Models: An overview of potential risks and mitigations
- CompanionSim: Synthetic Data for Evaluating Anthropomorphism in Human-AI Relationships
- Does My Chatbot Have an Agenda? Understanding Human and AI Agency in Human-Human-like Chatbot Interaction
- Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence
- From speaking like a person to being personal: The effects of personalized, regular interactions with conversational agents
- AI-exhibited Personality Traits Can Shape Human Self-concept through Conversations
- A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies
- Training language models to be warm and empathetic makes them less reliable and more sycophantic
Original note title
Nielsen Norman Group's study guide says users anthropomorphize AI at four escalating levels — courtesy, reinforcement, roleplay, and companionship