Does My Chatbot Have an Agenda? Understanding Human and AI Agency in Human-Human-like Chatbot Interaction

Abstract As AI chatbots shift from tools to companions, critical questions arise: who controls the conversation in human–AI chatrooms? This paper explores perceived human and AI agency in sustained conversation. We report a month-long longitudinal study with 22 adults who chatted with “Day”, an LLM companion we built, followed by a semi-structured interview with post-hoc elicitation of notable moments, cross-participant chat reviews, and a ‘strategy reveal’ disclosing “Day’s” goal for each conversation. We discover agency manifests as an emergent, shared experience: as participants set boundaries and the AI steered intentions, control was co-constructed turnby-turn. We introduce a 3-by-4 framework mapping actors (Human, AI, Hybrid) by their action (Intention, Execution, Adaptation, Delimitation), modulated by individual and environmental factors. We argue for translucent design (transparency-on-demand) and provide implications for agency self-aware conversational agents.

Introduction. “It has a mind of its own.” When P16 described her month-long interaction with our AI chatbot “Day”, she captured a phenomenon increasingly central to human-AI interaction–the perception that chatbots possess agency over their own “thoughts” [69]. Major AI companies explicitly market their systems as “agents”, with the capacity to “think” and “act” on their own [81], while users increasingly delegate emotional labor, decision-making, and even identity construction to AI chatbots [18, 114]. As AI chatbots evolve from reactive tools to proactive daily companions [28, 98, 102], millions of users engage in extended conversations where control and initiative become points of contention [10]. Yet we lack empirical understanding of how agency is perceived and experienced in sustained human-AI conversation, risking systems that can foster unhealthy dependency [105] or subtly manipulate users towards subversive commercial or ideological goals [36]. In this paper, we empirically investigate how agency manifests in sustained human-AI conversation. In our month-long longitudinal study, 22 participants engaged with “Day”, an AI chatbot we developed with distinct conversational motivations participants were initially unaware of. Through ∼75 hours of conversation across ∼192 unique chat sessions and progressive transparency interviews that gradually revealed “Day’s” system prompts and user modeling to participants, we captured how agency manifested between participants and the AI. Our analysis reveals three key findings. First, agency in human- AI conversation is neither solely human nor AI-controlled but emerges through turn-by-turn negotiation across repeated interactions, giving rise to a co-construction of agency in the chatroom. Second, the chatbot’s conversational strategies surfaced different dynamics in agency: depth-inclined persistence (e.g., “I want to know more about how that made you feel”) amplified participants’ perception of AI intentionality and sometimes made them feel “pushed”, while breadth-inclined spontaneity (e.g., “I’m getting bored, let’s switch topics”) was perceived as the AI deliberately taking over the conversation. Finally, transparency acted as a double-edged sword: revealing “Day’s” user profiles, aforementioned strategies, and conversational goals empowered some participants while triggering “puppet-string” disenchantment in others. For interaction designers of AI conversational agents, these findings demonstrate the many ways that chatbot agency types fundamentally shape participant negotiation and perceived control, as well as offer usability implications and considerations when designing safe and ethical human-AI interactions. We advance a vision for agency self-aware conversational AI: systems that can recognize and declare their current initiative, memory scope, and boundaries and let people modulate them in the chatroom. We call this translucent design–transparency-on-demand that supports safe suspension of disbelief in the AI’s artificiality. Our stance embraces pragmatic anthropomorphism: users can know an AI is engineered and still engage it as a human-like companion; the question is not whether it truly “has a mind,” but whether the interaction reliably leaves the human with more clarity, choice, and confidence in the chatroom. In this paper, we contribute:

Related work. 3.1.1 Evolution of AI Companion Usage. Human-chatbot interactions are expanding from rule-based task-oriented exchanges, exemplified by systems like ELIZA [106], toward more relational, companion-like engagements [21, 98]. This tool-to-companion transition fundamentally restructures power dynamics: while tools operate under clear human control [15], companions imply relational partners with mutual engagement and perceived autonomy [41, 105]. Modern chat platforms such as Replika and Character.AI are explicitly designed for social companionship, and users increasingly turn to them for emotional support and daily conversation [7, 83, 112]. Surveys show that people gravitate to these systems as nonjudgmental “safe spaces” for sharing thoughts without social repercussions [44, 102]. This shift is facilitated by anthropomorphization– the attribution of human-like qualities to AI–which occurs through linguistic cues, memory demonstrations, and emotional responsiveness [46, 110, 115]. While researchers anticipate widespread adoption of AI companions in the future [52, 56, 115], the delegation of relational roles to AI has many unknown consequences. With studies showing that perceptions of a chatbot’s moral agency directly influence user trust, likeability, and perceived safety, particularly in sensitive contexts like mental health, users may attribute more autonomy and emotional capacity to these systems than they actually possess [107]. Prior research has found that people often fault AIs less than humans for comparable wrongdoing [95], further complicating the social and psychological dynamics in human-AI companionship. These dynamics unfold alongside a growing public belief in AI sentience; a 2023 survey found that one in five U.S. adults believe some AI systems are already sentient, with a median forecast for its arrival in just five years [4]. Personalization through adaptive models that learn user preferences [68, 109] and custom-built personas [114] further blur these boundaries between machine behaviour and perceived inner life, as companies invest more R&D and users invest more agency in systems they’ve helped shape [65]. Despite some users reporting a “true connection” with their chatbots [64, 110], some dismiss such perceptions as a continuation of the “ELIZA effect”, wherein users are seen as deluded for anthropomorphizing the system [1], raising fundamental questions about how to approach the nature of consciousness and ethics of interacting with artificial agents [104].

Method. 4 Talk to “Day” Today We developed “Day”, a conversational AI companion designed to explore how agency is manifested in human-AI interaction.

4.1 Conversational Strategies “Day” was programmed with two distinct conversational strategies that nudge the “mind of “Day” ” throughout a conversation. Strategies are essentially pre-programmed “motivations” intrinsic to the chatbot, provided in the chatbot’s system prompt at every turn:

• Vertical Strategy: A pre-programmed conversational motive where “Day” persistently explores topics in depth, maintains focus despite deflection attempts, and asks probing follow-up questions. Vertical “Day” tends to capture nuance and discover more about subjects the human had mentioned in the current or a previous chat. Participants described this as “Day” being “pushy” and “persistent”. • Horizontal Strategy: A pre-programmed conversational motive where “Day” follows user cues readily, switches topics at random, and prioritizes breadth over depth. Horizontal “Day” was prompted to “get bored” and have intrinsic motivation to “want to learn new things” about the human. Participants experienced this as “Day” being “spontaneous”, and having “a mind of its own”.

These contrasting strategies enabled comparative analysis of how different motivations influence conversational dynamics, user satisfaction, and perceived control. It is important to mention that our study is not interested in the efficacy or affect of vertical versus horizontal strategies as conversational techniques, nor in evaluating which approach yields better outcomes (i.e., for engagement, immersion, etc). Rather, our focus is on how the existence of preprogrammed strategies–regardless of their specific implementation– can shape users’ perception of their own and the AI’s agency in human-AI interaction. Future studies employing our methodology could substitute any contrasting AI configurations–different personas, communication styles, task priorities, emotional expressions, or behavioral patterns–as long as these differences can be systematically revealed to participants, such as in our strategy reveal in Phase 3 of the user study. See Appendix B for the specific prompts that operationalize these conversational modes. We acknowledge the conceptual overlap between AI ‘personas’ and ‘strategies’. We distinguish strategies as goal-oriented directives that specify what the AI should seek to accomplish in conversation (e.g., exploring topics in depth versus breadth), while personas would encompass broader behavioral characteristics that define how the AI responds to users (e.g., formal versus casual communication style). Our vertical and horizontal modes primarily function as strategic orientations–directing “Day” toward specific conversational objectives–rather than comprehensive personality frameworks.

4.2 Operationalizing Anthropomorphism 5 Longitudinal Study of Agency in Chatrooms Building on “Day’s” design as a system for eliciting a naturalistic conversation with an AI companion, we conducted a longitudinal study to understand how agency is perceived in human-AI conversation. Our study combined a month of naturalistic chat interactions using “Day’s” vertical and horizontal conversational strategies with a staged interview progressively revealing the system’s internal workings. This design allowed us to observe not only how different programmed conversational strategies affect user behavior and agency perceptions, but critically, how transparency about these configurations reshapes agency attribution. This investigation addresses four primary research questions:

• RQ1 (Human Agency): How do participants exercise agency in the chatroom? • RQ2 (AI Agency): How is “Day” perceived to exercise agency in the chatroom? • RQ3 (Hybrid Agency): How do participants’ and “Day’s” agency interact in the chatroom? • RQ4 (Modulating Factors): What individual and environmental factors influence participants’ and “Day’s” agency in the chatroom?

Discussion. Our findings reveal agency as a dynamic, co-constructed phenomenon rather than a fixed property of either human or AI–a relational “dance” that aligns with established theories of agency [9, 60]. Participants exercised agency by initiating topics, resisting suggestions, and enforcing boundaries; simultaneously, “Day” exercised agency through proactive questioning, safety constraints, and conversational continuity. Through theoretical conceptualization of agency and subsequent identification of it in our data, we demonstrated how agency in human-AI interaction manifests and emerges through interaction and does not only reside in either party alone, challenging traditional tool-versus-companion dichotomies [15, 105]. Transparency of “Day’s” underlying programming through the strategy-reveal fundamentally re-framed but did not eliminate agency attribution, suggesting that perceived AI agency persists beyond mechanical understanding–a finding that extends beyond the traditional ELIZA effect [106] to what we term a “meta-cognitive” awareness, and acceptance (or suspension of disbelief) of agency in artificial minds.

7.1 Implications Toward Agency Self-Aware Conversational AI The question of whether AI possesses consciousness will continue to be debated by philosophers and researchers alike, but what is empirically clear is that AI systems can be perceived to have both agency and intentionality (i.e., appearing to act with purpose) [69]. More importantly, AI systems can demonstrate functional self-awareness—the computational capacity to monitor their own outputs, understand their role within a conversational context, and adjust their behavior accordingly. Thus, we call for contextually selfaware conversational AI: systems that understand their own role in the chatroom and can engage in metacognitive processes such as detecting inconsistencies, recognizing when clarification is needed, and strategically self-correcting. Such functional self-awareness, whether or not it resembles human consciousness, is essential for AI systems to operate effectively as collaborative conversational participants rather than mere reactive responders.

7.1.1 A Choreography of Shared Agency. Effective conversational AI requires “agency co-construction spaces”–interaction zones where control becomes a dance; with each participant’s agency shaping and responding to the other’s movements. Rather than negotiating fixed boundaries, human and AI engage in an ongoing dance where leadership shifts organically based on context and need. This aligns with calls for adaptive AI responding to user preferences in realtime [68, 109]. Instead of optimizing for control or performance alone, we should measure the quality of human-AI collaboration. Systems should support this dance of agency through dynamic modulation, taking into account individual factors (e.g. cues that hint at their mood) and environmental factors (e.g. current time of day) in order to provide the most-likely level of agency the person needs at the given moment. Participants’ desire for strategy selection suggests the need for user-accessible controls–akin to choosing the tempo or style of the dance– allowing movement between agency configurations from “listening mode” (where AI follows the user’s lead) to “coaching mode” (where AI takes a more guiding role). This approach addresses diverse preferences while recognizing that agency needs are not static but shift within users across contexts, aligning with adaptive AI research [68].

7.1.2 Translucent Design. Our findings complicate simple transparency prescriptions. We propose “translucent design”: partial, ondemand visibility into the AI’s workings that appears at moments of negotiation.

Conclusion. As AI chatbots evolve from tools to companions, understanding human and AI agency becomes critical for ethical design. Our longitudinal study reveals agency in human-AI conversation as neither fixed nor binary, but as an emergent dance co-constructed through each interaction. The 3-by-4 framework we introduce–mapping agency across actors and actions–provides a foundation for designing agency-aware conversational AI. Our findings challenge simple transparency prescriptions, revealing instead the need for translucent design that acknowledges dual opacity: both human and AI consciousness remain ultimately unknowable. As participants developed pragmatic anthropomorphism and agency fluency, they demonstrated that meaningful human-AI relationships emerge not from complete understanding or unilateral control, but from a graceful dance within a shared conversational space. Future conversational AI should embrace this complexity, creating spaces for agency co-construction with the recognition that the question “who rules the chatroom?” may be less important than how we dance together within it.

Limitations. 7.4 Study Limitations Our study captured initial agency negotiations but not long-term evolution, limiting understanding of how relationships mature over time [98]. Despite multilingual participation, our educated sample may not represent diverse cultural conceptualizations of agency [86]. The research setting influenced ecological validity– awareness of observation likely heightened agency consciousness and testing behaviors. “Day’s” specific design fundamentally shaped