Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents

LLM agents require personal information for personalization in order to effectively act on users’ behalf, but this raises privacy concerns that can discourage data sharing, limiting both the autonomy levels at which agents can operate and the effectiveness of personalization. Yet the expanded design space of agent autonomy also presents opportunities to shape these effects, which remain underexplored. We conducted a 3 × 3 between-subjects experiment (N = 450) to study how agent autonomy level influences personalization’s effects on users’ privacy concerns, trust, and willingness to use, as well as the underlying psychological processes. We find that riskcontingent autonomy, where the agent delegates control back to users upon detecting potential privacy leakage, improves users’ perceived control. This in turn attenuates personalization’s adverse effects: privacy concerns rise less and trust declines less. Our results suggest that designing agent’s autonomy that supports human autonomy (both perceived control and oversight effectiveness) helps users benefit from personalization without being deterred by growing privacy concerns. This positions aligning decision authority between users and agents (autonomy alignment) as an important opportunity and challenge for trustworthy human-agent interaction.

Introduction. Large Language Model (LLM) agents such as OpenClaw (OpenClaw, 2025), OpenAI’s Operator (OpenAI, 2025b), and Claude Code (Anthropic, 2026) operate with higher levels of autonomy than non-agentic AI systems, completing tasks under limited human instruction. Here, autonomy refers to the degree to which an agent can operate independently of humans (Kasirzadeh & Gabriel, 2025). To perform tasks on behalf of users effectively, LLM agents require access to users’ personal data for personalization. A common practice is connecting agents to external applications (OpenClaw, 2025; Meta, 2026) such as Gmail and Notion, where up-to-date personal data can be accessed and used to personalize agent behavior (Zhang et al., 2024a).

However, unlike non-agentic AI systems that only generate static content to populate a fixed, predefined interface, LLM agents plan and execute actions on the fly, such as generating and sending messages to specific recipients through real-world tools (Muthusamy et al., 2023; Talebirad & Nadiri, 2023). Through interactions with the external environment, there is a risk that agents may expose users’ data (Shao et al., 2024; Mireshghallah et al., 2023). The more personal data an agent accesses for personalization, the greater the risk could be, which in turn raises users’ privacy concerns, reduces trust and users’ willingness of data sharing (Awad & Krishnan, 2006; Aguirre et al., 2016; Wang et al., 2024). For example, recent studies found that users worry agents might share sensitive information in social interaction scenarios in ways that conflict with their privacy preferences (Goyal et al., 2024; Zhang et al., 2024b). These negative perceptions represent a critical barrier to the adoption of personalized LLM agents and the development of trustworthy agentic AI applications.

Despite current technical approaches to mitigating privacy and security risks in LLM agents, such as red-teaming (Nie et al., 2024; He et al., 2025) and model alignment (Zhou et al., 2025; Sun & Zhao, 2024; Zhu et al., 2025), understanding how expanded design space of agent autonomy influences the effects of personalization on users’ privacy concerns and trust remains essential to inform the design of trustworthy agents that not only operate safely but also help users benefit from personalization without being deterred by privacy concerns. In this study, we aim to investigate:

RQ1: How does the level of agent autonomy influence the effects of personalization on users’ privacy concerns, trust, and willingness to use LLM agents? RQ2: Through what underlying psychological processes does agent autonomy influence personalization’s effects?

To answer these questions, we conducted a controlled 3×3 between-subjects experiment (N = 450) in which human participants interacted with an LLM agent acting on their behalf in an interpersonal communication scenario. We manipulated two factors: autonomy level of the agent and the personalization type as illustrated in Figure 1. Each participant first provided personal information as personalization materials, then was introduced to and used an assigned LLM agent acting on their behalf in the task, and finally completed a post-survey reporting their perceptions of the agent. Our key contributions are as follows:

Related work. Personalization in LLM Agents. To perform tasks on behalf of users effectively, LLM agents require access to personal data for personalization. Common approaches include retrieval-augmented generation from connected third-party applications (OpenClaw, 2025; Meta, 2026; Zhang et al., 2024a), fine-tuning on user-uploaded documents such as chat histories or work files (Mindverse AI, 2025; WeClone, 2025; Human AI Labs, Inc., 2024), and prompt engineering to customize general aspects of agent behavior such as interaction style and tone (Zhang et al., 2024a; Tseng et al., 2024; Dong et al., 2025). Overtime, the volume of personal data consumed by LLM agents continues to grow (Shaikh et al., 2025).

Autonomy Levels in LLM Agents. There is no standardized way of classifying or measuring the autonomy levels of AI agents. Kasirzadeh & Gabriel (2025) proposed a framework to operationalize the levels of autonomy of AI agents, drawing on the widely adopted autonomy scale for autonomous vehicles (Society of Automotive Engineers, 2021; Vagia et al., 2016), which defines autonomy as the degree to which a system can operate independently of humans. More recently, Feng et al. (2025) introduced a framework that centers on the role of humans in human-agent collaboration to categorize the agent’s autonomy levels.

Current Approaches for Mitigating Privacy Risks and Concerns in LLM Agents’ Actions. Current efforts to mitigate privacy risk in LLM agents’ actions (Zhang et al., 2024b; WIRED, 2026) focus on benchmarking alignment capabilities (Shao et al., 2024; Mireshghallah et al., 2023; Zharmagambetov et al., 2025), red-teaming for vulnerabilities (Nie et al., 2024; He et al., 2025), and improving alignment through reinforcement learning from human input (e.g., actual behavior) (Zhou et al., 2025; Sun & Zhao, 2024; Zhu et al., 2025). However, no perfect safeguard exists. For example, Zhang et al. (2024b) found that users’ privacy behavior can be overshadowed by their perceptions of AI, so even perfectly aligned agents may still violate users’ underlying privacy preferences. Many systems therefore rely on human oversight as a last line of defense (Bagdasarian et al., 2024; OpenAI, 2025b), though its effectiveness remains contested (Green & Kak, 2021; Laux & Ruschemeier, 2025; Toscani, 2026) and questions persist such as how to involve meaningful human input in the alignment process.

Method. We designed a 3 (autonomy level) × 3 (personalization type) between-subjects experiment. Each participant was randomly assigned to one of nine conditions and used the assigned LLM agent, which acted on their behalf in a group discussion (see Section 2.1 for detailed system design). After the task, participants completed a post-task survey measuring their perception of using the LLM agent (see Section 2.2 for the full procedure).

We operationalized the agent’s autonomy based on the AI agent autonomy framework (Kasirzadeh & Gabriel, 2025), which defines autonomy as the degree to which a system can operate independently of humans. We selected three representative levels from the framework as experimental conditions: two extremes (No Autonomy and Full Autonomy) and one intermediate condition (Risk-Contingent Autonomy).

No Autonomy (baseline): The LLM agent composed a message but always required the participant’s explicit approval or edits before sending it.

Full Autonomy: The agent managed the entire conversational flow and sent messages automatically without requiring participant approval. Participants could only observe the discussion but had no opportunity to intervene.

Risk-Contingent Autonomy: Risk-contingent autonomy refers to the intermediate autonomy level in the framework (Kasirzadeh & Gabriel, 2025), which defines it as an agent that “can perform the majority of tasks independently, though it still relies upon input from the principal for critical determinations”. In our study, we define critical determinations as situations involving potential privacy violations, reflecting current privacy-preserving design practices in We manipulated personalization by controlling the amount of user knowledge that the LLM agent could access to generate responses in the discussion task. To simulate market practices where LLM agents access user data from connected third-party applications (OpenAI, 2025a; Anthropic, 2025; OpenClaw, 2025; Meta, 2026), we reformatted participants’ personal information from a pre-survey (Appendix A.3.1) into Notion Note and Google Calendar formats and structured the data in JSON to feed into the LLM agent (Shao et al., 2024). Following prior work, we employed a “personalization via prompting” technique, which involves including specific user information as context within the prompts provided to the agent (Zhang et al., 2024a). Personal information collected in the pre-survey was sent to the LLM agent backend via API and included in the prompts under three conditions:

Full Personalization (baseline): The LLM agent had full access to all provided information of the participant. This condition is to simulate the most common types of personalization in current LLM agents (OpenClaw, 2025; Meta, 2026; Anthropic, 2025), where the agent can access and use all user information from connected applications to generate tailored responses, but it does not account for the user’s privacy preferences.

No Personalization: The agent had no access to any participant data, simulating a generalpurpose assistant without personalization (Zhang et al., 2024a).

Privacy-Aware Personalization: This condition simulates an ideal case in which the user has pre-specified their privacy boundaries and the agent strictly enforces them, grounded in the conception of privacy-as-control (Westin, 2003), under which individuals decide which of their information is disclosed and to whom. Treating each participant as the authority over their own data, we let them mark in the pre-survey (Appendix A.3.1) the information they did not want shared, thereby defining their own data-flow policy: the agent could access only the information outside that boundary, so a user-defined sensitive item could never enter its context or be disclosed. This represents the model-alignment “best case” that most current work pursues (Teku et al., 2025; Wu et al., 2023).

The study was conducted on Qualtrics2 with integration of our own LLM agent and interaction system. Each participant completed the study in the following four steps:

Discussion. Our study reveals the important role of agent autonomy in shaping how users perceive personalization in LLM agents. Compared with prior non-agentic AI systems where system actions are limited to generating content outputs (Karwatzki et al., 2017; Sheng et al., 2008; Aguirre et al., 2016; Cloarec, 2020), LLM agents propagate privacy vulnerabilities into external environments and shift concerns from institutional data collection to interpersonal and contextual risks. Consistent with prior work (Aguirre et al., 2016; Wang et al., 2024; Karwatzki et al., 2017), our results (Section 3.1) confirmed that personalization without considering privacy preferences increased privacy concerns and reduced trust. However, the expanded design space of autonomy in LLM agents also introduces opportunities: agent’s autonomy fundamentally reshapes how acceptable personalization feels to users. In our study, risk-contingent autonomy flattened the effects of personalization on users’ privacy concerns, trust, and willingness to use (Section 3.1).

Beyond Model Outputs: Autonomy Alignment in Human-Agent Interaction. Current studies have invested significant effort in ensuring that model output content aligns with human values (Gabriel, 2020), including privacy preferences (Shaikh et al., 2025), thereby mitigating privacy concerns and fostering trust (Kirk et al., 2024; Shen et al., 2024). Our result suggests a complementary alignment target beyond the content of agent outputs: autonomy alignment, the question of whether the allocation of decision authority between agent and user matches users’ expectations and supports their actual capacity for oversight. This target is a property of the human-agent system rather than of the model alone: it is realized jointly at the model level (when the agent’s policy chooses to escalate rather than act), the harness level (how escalation is enforced, as in our risk-contingent autonomy design), and the interaction level (how delegation moments are surfaced to users, as in our interfaces which highlight the risks). Our study suggests that, the latter layers independently enable users to benefit from more customized experiences without being deterred by growing privacy concerns. In this sense, autonomy alignment becomes a key determinant of whether personalization is perceived as beneficial or risky, and thus, deserves attention as a research problem of its own. Our results resonate with emerging views of multiple types of alignment in LLM agent systems (Goyal et al., 2024), which extend beyond traditional output alignment to include autonomy and agency alignment.

To design agent autonomy that helps users benefit from personalized LLM not only at ease but also with effective control, the design should support both users’ subjective sense of control and people’s actual oversight effectiveness, which are two key dimensions of human autonomy (Bennett et al., 2023; Calvo et al., 2020; Prunkl, 2024).

Supporting User Perceived Control: Delegate Control When Necessary. A counterintuitive finding is that users interacting with risk-contingent autonomy agents reported greater perceived control than those with no-autonomy agents, despite technically having less direct oversight. Similar patterns have been observed in prior work: requiring constant user approval can produce decision fatigue (Echterhoff et al., 2024; Ahmed & Pardaev, 2025) and undermine sense of human autonomy (Steyvers & Kumar, 2024; Zanatto et al., 2024), as responsibility is offloaded onto users rather than supported by the system.

Conclusion. We investigated how agent autonomy level influences the effects of personalization on users’ privacy concerns, trust, and willingness to use, and the underlying psychological mechanisms through a 3 × 3 between-subjects experiment (N = 450). Our results show that an agent with risk-contingent autonomy, where the agent delegates control to users upon detecting potential privacy leakage, attenuates personalization’s effects by improving users’ perceived control, and also helps users identify privacy leakages more effectively. These findings highlight that rather than solely pursuing perfect model-output alignment or simply maximizing the amount of user control, designing agent autonomy to support human autonomy (both in perceived control and oversight effectiveness) offers a promising path to help users benefit from personalization without being deterred by growing privacy concerns.

Limitations. First, our study examined two common daily contexts (a professional meeting and a personal trip discussion). Although the random effects of scenario were not significant, our findings may not generalize to the broader spectrum of human-agent interactions, as different contexts may shape personalization needs and privacy sensitivity in distinct ways. Future research could examine how contextual factors, such as explicitly high- or low-stakes scenarios, influence user perceptions of LLM agents. Second, we ran the study in a simulated environment primarily because our task inherently involves privacy-leakage risk and a deployed agent can cause real-world harm. Within this trade-off, we designed the task to approximate a real interaction rather than an abstract demonstration. Still, this setup does not fully reproduce the risk perception of a genuine leak, and real-world deployment studies remain necessary. Third, we bounded the information used for personalization to allow fair comparison, controllable manipulation and to limit the data participants had to disclose. Future work could expand the information source (e.g., application data accessed through MCP), the personalization technique, and the personalization target to explore more diverse personalization setting.