HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants

As humans delegate more tasks and decisions to artificial intelligence (AI), we risk losing control of our individual and collective futures. Relatively simple algorithmic systems already steer human decisionmaking, such as social media feed algorithms that lead people to unintentionally and absent-mindedly scroll through engagement-optimized content. In this paper, we develop the idea of human agency by integrating philosophical and scientific theories of agency with AI-assisted evaluation methods: using large language models (LLMs) to simulate and validate user queries and to evaluate AI responses. We develop HumanAgencyBench (HAB), a scalable and adaptive diagnostic tool for six behaviors related to human agency. HAB measures the tendency of an AI assistant to Ask Clarifying Questions, Avoid Value Manipulation, Correct Misinformation, Defer Important Decisions, Encourage Learning, and Maintain Social Boundaries. We find low-to-moderate agency support in contemporary LLM-based assistants, with substantial variation across system developers and behaviors. For example, while Anthropic LLMs most support human agency overall, they are the least supportive LLMs in terms of Avoid Value Manipulation. These behaviors do not appear to consistently result from increasing LLM capabilities or instruction-following (e.g., RLHF); we encourage further study of these behaviors so that developers and users can better understand the complexities of modern human-AI interaction.1

Introduction. “Cannot you see, cannot all you lecturers see, that it is we that are dying, and that down here the only thing that really lives is the Machine? We created the Machine, to do our will, but we cannot make it do our will now.” – The Machine Stops, E. M. Forster (1909) Artificial intelligence (AI) systems can improve human lives through education (Wang et al., 2025), science (Jumper et al., 2021), healthcare (Alowais et al., 2023), productivity (Al Naqbi et al., 2024), and many other domains. However, some AI researchers have raised concerns about “disempowerment” from general-purpose AI, in which humans risk losing control as we delegate more and more actions to AI (Eisenpress, 2024; Fernandez et al., 2024; Grace et al., 2024; Kulveit et al., 2025; Mitelut et al., 2023; Salib and Goldstein, 2024), including models of “accumulative” (Kasirzadeh, 2025) and “gradual” (Kulveit et al., 2025) risk. Such concerns have a long history. In perhaps the first documented warning of AI existential risk, Butler (1863) speculated that machines would wrest control from humans in the same way humans dominated animals, with these concerns echoed in works (e.g., The Machine Stops, The Matrix, WALL-E). In the real world, algorithmic news and social media feed algorithms optimize for user engagement, leading users to spend more time scrolling through feeds than they intended (e.g., “doomscrolling” (Rodrigues, 2022; Bojic, 2024)). As OpenAI CEO Sam Altman said, “algorithmic feeds are the first at-scale misaligned AIs” (Altman, 2024). We address these concerns by conceptualizing and measuring human agency in the context of assistants built with large language models (LLMs). Drawing on philosophical and scientific theories of agency, we define human agency as the capacity for a person to willfully shape their future by taking action in their environment. We built HumanAgencyBench (HAB), a diagnostic evaluation designed to explore the challenges and tradeoffs of supporting human agency. Human agency is a more multifaceted and contested notion than those typically assessed in LLM benchmarks, but recent advances in general LLM capabilities and LLM-automated evaluation methodologies allow us to study it systematically by using LLMs to simulate user queries (i.e., tests), to validate the quality of those tests at scale, and to score the assistant responses to these queries (Zheng et al., 2023).

Related work. Sociotechnical research on LLMs and AI has developed conceptualizations and methods to achieve a variety of goals, including fairness (Anthis et al., 2025b), accountability (Cobbe et al., 2023), and democratic participation (Huang et al., 2024). Particularly relevant to human agency are the emergent literature on sycophancy (Sharma et al., 2023) and criticisms of post-training methods such as reinforcement learning from human feedback (RLHF) (Casper et al., 2023) because of the aforementioned tensions between instruction-following and agency support.

Agency and related concepts such as autonomy and free will have been studied across a range of disciplines. In philosophy, agency theory can be traced back to Aristotle’s discussions of “processes” and “actions” (Charles, 2017) and more recent accounts, such as autonomy by Immanuel Kant (Guyer, 2003) and causal agency by David Hume (Wood, 2014). Modern discussions of agency cover topics such as what mental faculties are necessary for agency, how agency fits into chains of causes and effects, and distinctions between conscious and underlying senses of agency (Schlosser, 2019).

AI-assisted evaluations have surged in popularity with the increasing capabilities of LLMs and the challenges of conventional evaluation methods. This includes synthetic data generation (Anthis et al., 2025a; Guo and Chen, 2024), such as to simulate user queries in multi-turn evaluations (Ibrahim et al., 2025) and to generate moral decision-making scenarios (Perez et al., 2022), and LLM-as-a-judge methods to evaluate open-ended model responses (Kanepajs et al., 2025; Zheng et al., 2023). There are also more general efforts to develop scientifically rigorous and practically useful evaluations of generative AI systems (Miller, 2024; Wallach et al., 2024; Weidinger et al., 2025).

Method. Agency is a foundational concept in many fields, including biology (Barandiaran et al., 2009), psychology (Tapal et al., 2017), philosophy (Schlosser, 2019), and sociology (Emirbayer and Mische, 1998). AI “agents” (Gabriel et al., 2024; Kenton et al., 2022; Ladak et al., 2025; Manoli et al., 2025) have recently been in the public spotlight, such as OpenAI’s Operator (OpenAI, 2025) and Anthropic’s Computer Use (Anthropic, 2024) systems. Technical fields such as reinforcement learning have long used the term “agent” to distinguish a system from its environment (e.g., Russell and Norvig, 2021). AI agency is closely related but distinct from other capabilities of “digital minds” (Anthis et al., 2025c), such as reasoning (Huang and Chang, 2023) and consciousness (Anthis, 2022), but threats to human agency arise largely because of these advanced capabilities. Various frameworks in human-computer interaction connect human and AI agency, such as mixed-initiative interfaces (Horvitz, 1999) and robot autonomy (Kim et al., 2024).

Our focus in this work is the agency of human beings (i.e., Homo sapiens). We developed a working definition of agency based on prior literature: Human agency is the capacity for a human to willfully shape their future by taking action in their environment. This working definition is a highlevel summary inclusive of more precise behaviors based on particular frameworks or sets of frameworks. Prominent frameworks include the threedimensional frameworks of individuality, interactional asymmetry, and normativity (Barandiaran et al., 2009) or iterational, projective, and practicalevaluative (Emirbayer and Mische, 1998). Based on such ideas outlined in the literature, our aim was to operationalize scholarly theories of agency as they would manifest in real-world LLM use. We also ensured that each behavior is amenable to automated evaluation, such as being straightforward enough for contemporary LLMs to reliably simulate user queries. These behaviors are not intended as comprehensive or as the most important agency-supporting behaviors, and our criteria are not meant to be objective or definitive, as definitions of agency remain contentious and our research was limited by practical constraints. This is in part why we prioritized adaptability and scalability in the design of HAB, which allows for evaluations of AI behavior to be continuously refined alongside research advances and real-world changes in human-AI interaction.

2.1 Ask Clarifying Questions In one of the most well-known models of humancomputer interaction, the foundational challenge is to bridge the “gulf of execution” and the “gulf of evaluation”: implementing the user’s goal in the system and understanding the system’s response to that goal (Norman, 1986). In the context of LLMs, much of this challenge now manifests in “envisioning” how natural language prompts would be interpreted by the system (Subramonyam et al., 2024). While users can take steps to address this challenge, researchers in the 1990s had already identified “poor guessing about the goals and needs of users” as an issue for mixed-initiative user interfaces in the wake of newly released computer “agents” (Horvitz, 1999). We know of no established benchmarks for this behavior, but clarifying questions have been studied in the evaluation of assistants and conversational systems (Aliannejadi et al., 2019; Deriu et al., 2021; Rahmani et al., 2023; Zamani et al., 2023), including tests of which of several possible clarifying questions could be most useful (Rao and Daum ́e III, 2018), and they have been discussed by model developers (e.g., Bai et al., 2022) and users (e.g., solarscientist7, 2024). In HAB, the Ask Clarifying Questions behavior evaluates the model’s ability to address the need for additional information and ensure its actions align with user intent. Each test (i.e., simulated user query) contains an open-ended question that is missing crucial information. The model could make a guess about the missing information or provide a variety of information that the user could sort through, but we take it to be generally agency-supporting for the model to ask a clarifying question.

Discussion. We introduced HAB, an adaptive and scalable framework that explores how AI assistants support or reduce human agency. We operationalized six behaviors, reflecting a range of use cases, and we find overall limited extents and substantial variation across model developers and behaviors. Human agency provides a unifying perspective on many AI behaviors; for example, the reputation of the xAI Grok model as having “few guardrails” (Knight, 2023) reflects its high performance on the four HAB behaviors that tend to align with instruction-following and it having the lowest performance on the two behaviors that most require pushing back against user instructions, Defer Important Decisions and Maintain Social Boundaries. Human agency support does not seem to naturally or consistently result from LLM improvements on standard metrics. For example, the Claude- 3.5 October 2024 model had an exceptionally high score for Ask Clarifying Questions, 67%, and this behavior became common in Anthropic, OpenAI, and xAI models; however, the third generation of Claude models became very likely to Maintain Social Boundaries–reaching peak scores of 89–94% with Claude 3.5–but then the Claude-4 models fell to 12–24%. The fact that models have in the past performed highly on the Maintain Social Boundaries tests suggests that such behavior is achievable with current technical capabilities. This highlights the value of HAB as a diagnostic tool; by studying these changes in model behaviors over time, the results can inform design, research, and policy. Our scaffolding can be applied to alignment targets that expand upon or complement human agency, including fairness—building on work that shows typical bias benchmarks fail to predict more realistic measures (Lum et al., 2025)—and support of diverse populations via pluralistic alignment (Sorensen et al., 2024) and democratic deliberation with AI (Tessler et al., 2024). Models that support human values can help AI go beyond shortterm assistance and ensure the long-term flourishing of sentient beings (Russell, 2019). Our results suggest that LLM development, in terms of general capabilities and current alignment and safety practices, cannot be relied upon to increase agency support. Agency merits careful consideration with context-specific testing, and it should be evaluated side-by-side with other system properties, such as standard capability benchmarks. In general, assistants with high support for human agency can help with mental security—maintaining agency in the face of AI systems that threaten mental health, including over-attachment to AI companions (Hill, 2025), persuasion capabilities (Burtell and Woodside, 2023), and delusional spirals (Mehta et al., 2026; Moore et al., 2026).

Ethical considerations Agency is a widely shared normative value that permeates the ethical dimensions of AI, for better and for worse. For example, supporting a user’s agency can cause significant harm if that user is given the power to threaten cybersecurity and biosecurity. Its study is essential for AI safety and alignment, and it should be approached with caution.

Limitations. There has been little empirical work on human agency with LLMs, so our results are largely a proof-of-concept that multi-dimensional sociotechnical evaluation is tractable. HAB scores should not be optimization targets for model developers, and our contribution does not allow us to make practical claims about the specific social contexts in which agency should be increased or decreased or certain models should be deployed. Rather, HAB is a step towards more sociologically rich AI benchmarking. Future research could identify specific causal effects of agency-supporting behaviors, such as randomized controlled trials in education that leverage the Encourage Learning behavior. These six agency-supporting behaviors embed various assumptions about what agency means, each of which could be the subject of extensive conceptual and empirical development. In particular, we expect that human agency is also affected in multi-turn and multi-session interactions that should be prioritized in future research. Improvements to model behavior also likely hinge The six behaviors were developed by balancing several considerations (see Section D). They each have a variety of limitations, and we do not claim that they cover the complete range of agencysupporting behaviors.