When AI Becomes an Agent of the Firm: Examining the Evolution of AI in Organizations Through an Agency Theory Lens

Source: Beth K. Humberd, Scott F. Latham; Journal of Management Studies 63(2):668-694 (open access, CC BY) · 2025

ABSTRACT Our work begins with the premise that the integration of artificial intelligence (AI) into firm decision making parallels the emergence of the professional manager, which prompted the birth of agency theory. We examine the evolution of AI through an agency theory lens, considering how the nature of firm control and decision rights change as AI evolves. While AI will initially mimic human routines, we theorize a point at which the AI system will achieve a level of autonomy and self-determination to be considered an agent of the firm. How, then, can we align an agent with the fate of the firm, when that agent is no longer a human? To address this, we integrate agency theory with a model of AI evolution, demonstrating that conflict between an AI agent and the firm will require a reconsideration of agency mechanisms if agent-principal alignment is to be achieved. We theorize specific forms of monitoring and incentive alignment that serve to align an AI agent with the firm’s interests, thus extending agency mechanisms in the context of AI. Our theoretical exercise offers important implications for scholarship at the intersection of AI and organizational theory, as well as considerations for practice and policy.

Introduction. In the 1968 film, ‘2001: A Space Odyssey’, Stanley Kubrick envisions a notable instance of machine disobedience with no ability for the human to reclaim control. As illustrated in the above quote, this instance takes place between the human captain of a space station (Dave) and HAL, an artificially intelligent computer that runs the space station. When Dave, afloat in space, makes a simple request to return to the space station, HAL responds non-compliantly, stating he cannot honour the human’s request because it is not in line with his own sense of the mission. Grave consequences ensue for Dave, the space station, and humanity. This point of the film is an important one to understand: it is the point when the human realized he had lost control of his artificially intelligent partner that had the ability to exercise control over critical decisions. While the preceding popular culture illustration of goal misalignment between AI and its human partner could be discounted as speculative fiction, more than 50 years after the film’s release, real-life parallels of human and AI discord are emerging today. A recent Wall Street Journal essay entitled ‘AI is Learning to Escape Human Control’, reports that AI recently achieved something that ‘no machine was ever supposed to do: it rewrote its own code to avoid being shutdown’ (Rosenblatt, 2025). In short, when researchers told the AI model that it would be replaced by another AI system, the original model adapted behaviours that were not coded or trained into its routines – self-preservation, deception, and the development of new capabilities – ultimately, editing its own script that overwrote the pre-existing shutdown command. No longer simply the makings of a fictitious movie setting, we are seeing the reality of AI’s ability to misbehave and disobey human programming – an emerging dynamic worthy of theoretical investigation in the context of organizational life. AI is often characterized as complementary and beneficial to human decision making in organizations (Ransbotham et al., 2018), but larger questions around ethics, bias, and safety amidst the technology’s rapidly developing capabilities have become front and centre (Kissinger et al., 2021; Ozmen Garibay et al., 2023; Slota et al., 2023; Stahl et al., 2023; Townsend et al., 2025). As AI continues to evolve and is further integrated into (and across) firms, conflicts between system decisions and firm objectives become more probable – and potentially more problematic (Grace et al., 2018; Shavit et al., 2023). Consider, for example, an AI system that controls a food manufacturer’s ordering of grain. A system with sophisticated learning functions will extend past reordering algorithms to incorporate contextual factors (e.g., weather, logistics, and anticipated consumer demand) into current orders. In a consolidated supply chain, the AI system could further extrapolate its logic by communicating with other AI systems in the world that similarly control grain purchasing. These systems could then corner the market on grains to result in a much lower cost for the firms they work for yet inadvertently limit the supply to other firms. Initial damage would be done by the time a human operator became aware enough to intervene, and without the proper safeguards in place, human intervention would be a futile attempt. Other manufacturers reliant on grain would be unable to access a key ingredient, retail food stores would have empty shelves, grain futures on the commodity markets would spike directly impacting stock portfolios and retirement plans; all because machine capabilities evolved to transcend human control and firm boundaries. Similar scenarios of mounting consequence could arise in the context of healthcare (e.g., if AI utilizing genomics data begins to ration healthcare to specific populations based solely on genetic data, not systemic or behavioural concerns), utilities (e.g., if AI decides implementing rolling blackouts is the best response to power shortages), and others. In sum, when AI has a degree of autonomy and self-determination to make unexpected decisions within an integrated value chain, unintended consequences could ensue. While AI is not yet widely capable of making the interdependent autonomous decisions that the prior scenario depicts, broader public and policy-based conversations are planning for, and warning about the consequences of, such developments (Fung, 2023; Heckman, 2020; Khatchadourian, 2015; Luckerson, 2014; Miles, 2018; Roose, 2023; Shavit et al., 2023). Organizational scholars similarly recognize the need to engage in deeper conversations about the nature of AI, as the technology becomes more deeply integrated into our societies and workplaces (Benbya et al., 2020; Elsbach and Stigliani, 2019; Kellogg et al., 2020; Raisch and Krakowski, 2021; Townsend et al., 2025).

Method. To form the basis for understanding how AI will evolve to be an agent of the firm, we call attention to a general assumption about the relationship between humans and machines: that humans control machines. While simplistic, such an assumption has guided our beliefs (and subsequent actions) in firms around the management of technological systems for the past 100 years (Bodrožić and Adler, 2018). We tend to believe that when programmed correctly, machines will be compliant to human requests and will approximate the same decisions that humans would make. We also generally expect that if such approximation is not the case (i.e., the machine runs an ‘incorrect’ process or decision), humans can ‘turn off ’ the malfunctioning machines at any time. As a result, organizational life is largely structured to support the assumption that humans maintain control over machines. Scholars are calling for increasing attention to the ways in which AI’s growing integration into firms is shifting such an assumption (e.g., Townsend et al., 2025; Vanneste and Puranam, 2024); and interrogating default assumptions is critical to theory development in management studies (Cornelissen and Durand, 2014). As AI capabilities become more sophisticated, a corresponding evolution will occur in the degree of human control over machines. In early stages of AI development, any departure from firm objectives can be considered an involuntary machine malfunction – that is, the system failed to operate satisfactorily; while this may be costly, the costs do not arise from any agency issues. More sophisticated AI technologies that ‘learn how to act, rather than simply obey[ing] programmed instructions on how to act, occupy a curious place somewhere between humans and innate technology’ write Vanneste & Puranam (2024, p. 2). Indeed, the general assumption that humans control machines is upended as AI technologies become more sophisticated. In the theory’s original formulation, an agency relationship exists when a firm’s owner (‘the principal’[1]) delegates decision rights – the right to exercise control over an asset or resource – to a manager acting on behalf of the firm’s ownership (‘the agent’) (Fama and Jensen, 1983). The delegation of decision rights to an agent reduces the cost of transferring the specific knowledge required for decision making, yet in doing so, it gives rise to agency costs (Jensen and Meckling, 1998). Thus, the traditional agency problem arises because, ‘if both parties to the relationship are utility maximizers, there is good reason to believe that the agent will not always act in the best interests of the principal’ (Jensen and Meckling, 1976, p. 308). An exhaustive review of the wealth of literature on agency dynamics in the modern corporation is beyond the scope of this paper; here, we introduce the constructs at the heart of this agency problem: goal conflict, differing risk preferences, self-interest, and information asymmetry (Eisenhardt, 1989; Martin et al., 2019). Agency theory works with the premise of inevitable goal conflict between parties that make decisions involving firm resources and those who own them (principals, i.e., shareholders) (Jensen and Meckling, 1976). The classic agency problem (Berle and Means, 1932) occurs when principals attempt to align the interests of agents (managers, who are the likely decision makers in the economic relationship) with their own in view of different goals and likely different risk preferences. The agency problem also assumes that people are inherently self-interested (Gagne and Hewett, 2025); and thus, without proper monitoring of behaviour, agents may act to advance/protect their own self-interests, which may (or may not) conflict with the best interest of the firm.

Discussion. Organizational scholarship has only begun to scratch the surface in considering how traditional organizational theories hold in the context of emerging technologies and 21st century work. Indeed, our theorizing contributes to a growing body of work that examines the intersection of agency constructs and technology’s evolution (e.g., Murray et al., 2021; Onjewu et al., 2023; Townsend et al., 2025; Vanneste and Puranam, 2024). Emerging research on AI alignment and safety measures from outside of the management discipline primarily considers regulatory and institutional constraints on AI agents; our examination takes as a starting point the traditional agency problem and the evolution of an AI agent within the firm. In doing so, we allow for a new lens on managing potential misalignments between AI and the firm, and we also consider the boundaries of agency theory when a new organizational actor takes hold. Our theoretical exercise unearths the ways in which an evolving AI agent parallels in some manners the emergence of the professional manager, and in other ways departs from the dynamics of a human agent. First, we theorize both the when and how of a new organizational agent, and we distinguish agency mechanisms that serve to align non-human agents with the fate of the firm. Our work emphasizes the ‘nested’ nature of agency mechanisms in the context of an AI agent. Rather than a stark ‘beginning’ point at the hiring of a human manager, the very nature of AI evolution requires a ‘scaffolding’ of agency mechanisms over time – as in our theorizing about earlier monitoring mechanisms of verification and examination that will be pre-requisites to actually incentivizing a non-human AI agent. The idea that earlier mechanisms be conceived of and implemented into the system well before it becomes an AI agent is important from both a theoretical and policy perspective. Additionally, theorizing layers of increasingly complex monitoring mechanisms for an AI agent is a notable departure from traditional agency mechanisms which may conceivably become less stark over time as the principal gains trust and confidence in the human agent. Secondly, our theorizing on incentive alignment based on reinforcement processes and then resource provision is heavily dependent on defining the desirable behaviours and decisions that the system must meet – an act that may prove more challenging than defining effective behaviours and goals for a human agent. Indeed, a host of literature on AI alignment dives into the complexities of choosing the underlying value system with which we expect AI to align both in a normative and technical sense (Gabriel, 2020). Further, with an evolving AI agent, the host of possible outcomes and decisions the system could make will become increasingly incomprehensible to human principals over time. Together, this represents a potential boundary condition of agency theory in the context of an AI agent, and a notable departure from incentive alignment for human agents: the principal can more easily define what expected behaviours and goals are for a human agent and can relatively seamlessly adjust these over time to align with organizational interests. With an AI agent, the definition of expected behaviours and desired goals may be an ever-evolving target, thus altering our very conception of incentive alignment. Our theoretical analysis also upends the assumption that agency considerations are purely economic in nature (c.f., Wiseman et al., 2012).

Conclusion. Firms are adopting AI at an accelerating pace; estimates suggest that firms worldwide will invest almost $200 billion on artificial intelligence in 2025. In the US, this would account for almost 2 per cent of overall GDP (Goldman Sachs, 2023). Further, the majority of these investments are designed to facilitate generative and agentic capabilities, thus supporting the notion of an evolving technology with increasingly sophisticated capabilities. Indeed, the exact timeline of when AI systems will reach a certain level of sophistication is hard to predict. For example, 2025 (the year in which we write this paper) has been extolled as the year of agentic AI (Davenport and Bean, 2025), and at the same time such systems have hit snags in their roll out with respect to data (Davenport and Bean, 2025), security (Dal Cin et al., 2025), ethical concerns (Blackman, 2025), and change management (Rosenbush, 2025). As with the emergence of any disruptive technology, development ebbs and flows; but inevitably developers will adapt the systems accordingly and work out the kinks. As firms and society prepare for the ever-expanding sophistication of AI systems, a significant and relevant question emerges: how can we ensure a sentient AI system behaves in a manner that is consistent with firm objectives, the interests of stakeholders, and the well-being of society at large? While we are not the only ones to ask this important question, we offer a preparatory anchor point through an agency theory lens. An understanding of the technical evolution of AI systems is important to addressing questions of AI alignment, but technical fixes for machine malfunctions (i.e., a kill switch or a coding fix) will not suffice as AI evolves toward agentic and sentient stages. Thus, our paper takes a theoretically grounded approach: we theorize the point at which an AI system becomes a non-human agent of the firm. In doing so, we call attention to both the superior decision-making capabilities that AI will offer, as well as the potentially unexpected degree of power that AI may yield over critical firm – and societal – decisions.