SYNTHESIS NOTE
Topics›Psychology Chatbots Conversation›this note

Why don't design recommendations transfer cleanly to AI companions?

This research explores why design guidance from adjacent fields like trustworthy AI fails to apply straightforwardly to AI companions. It asks whether ethics and UX can be treated separately or must be negotiated together for each design choice.

Synthesis note · 2026-09-25 · sourced from Psychology Chatbots Conversation

The paper's central finding is that design recommendations written for adjacent domains, such as trustworthy AI and relational AI systems, do not transfer cleanly to AI companions, and that the friction comes from ethical and UX considerations being "deeply intertwined" rather than separable. The abstract frames AI companions as conversational agents "designed to cultivate emotional connection rather than primarily to support humans in instrumental tasks." It says prior work offers possible design solutions but leaves open "what makes AI companion design difficult as a design problem." The authors' answer is that existing recommendations "often require context-sensitive application." The discussion gives the general form: "what optimizes user experience for one individual in one situation may create risks for a different user or in another context."

The paper reaches this through four phases (literature review, practitioner co-analysis, internal heuristic evaluation, external expert assessment) that produced nine design principle areas where applicability was tense. The discussion excerpt covers four of them, and each relates ethics to UX differently. Safety was "consistently viewed as ethically non-negotiable, yet still required context-sensitive implementation." Predictability and Consistency spanned both domains but was "primarily associated with flexibility for UX optimization," even though consistency itself "was viewed as ethically important." Engagement and Empathetic Response were framed as UX-oriented principles that should "generally be upheld," with participants asking for moderation when excessive engagement might lead to dependency or when empathetic responses could enable harmful behaviors. So there is no single rule such as "ethics constrains, UX optimizes." The relationship has to be worked out principle by principle.

Read against the nearest notes, this paper moves the contingency up to the level of design principles. Can attachment theory prevent parasocial harm in AI companions? treats safety as a boundary that can be built into a module. This paper agrees safety is non-negotiable but adds that how the boundary is applied still flexes with users and contexts. How should chatbot design vary by relationship duration? also makes design guidance depend on a variable, there the time horizon. Here the variable is user and situation. Does chatbot personalization build trust or expose privacy risks? has the same shape at the feature level: one move that improves the experience also raises the risk. The companion paper says this holds for engagement and empathy too, where the UX gain and the potential for dependency or harm are the same behavior.

The excerpt does not say how many practitioners or experts took part or who they were. It does not name the other five principle areas or list which recommendations from which adjacent domains were tested. It also does not report whether context-sensitive application resolved the tensions or improved anything for users. The findings are stated as what participants "viewed" and "emphasized," not as measured user outcomes. The implication at that strength is modest: the nine areas work as a map of where borrowed guidance needs a context check before it is applied to a companion. They are not verified design rules. Engagement and empathy in particular should be treated as principles with a moderation condition, not as quantities to maximize.

Inquiring lines that read this note 2

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Why does polished presentation create unearned authority in AI outputs? How should designers communicate what AI systems truly are and can do?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
13 direct connections · 77 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

ethical and UX considerations are intertwined in AI companion design — recommendations from adjacent domains need context-sensitive application