Must AI monitoring precede incentive alignment as agents evolve?

Humberd and Latham theorize that as AI becomes more sophisticated it will eventually cross a threshold to be "considered an agent of the firm," and that aligning such an agent with firm interests requires agency mechanisms to be built in a specific sequence rather than imposed all at once. They write that their theorizing emphasizes "the 'nested' nature of agency mechanisms in the context of an AI agent," where "earlier monitoring mechanisms of verification and examination" must function as "pre-requisites to actually incentivizing a non-human AI agent." This is a "scaffolding" model: the mechanisms needed to even know what an AI system is doing have to be in place before the separate problem of incentivizing it toward firm goals can be addressed.

The authors root this in classic agency theory — the idea, going back to Jensen and Meckling (1976), that "if both parties to the relationship are utility maximizers, there is good reason to believe that the agent will not always act in the best interests of the principal," and that the resulting agency problem rests on goal conflict, differing risk preferences, self-interest, and information asymmetry. Applied to a human manager, the paper argues, this problem has a stable structure: the principal "can relatively seamlessly adjust" expected behaviors and goals over time, and monitoring intensity typically eases "as the principal gains trust and confidence in the human agent." Applied to AI, both halves break: incentive alignment depends on first "defining the desirable behaviours and decisions that the system must meet," which the paper calls "heavily dependent" and "more challenging than defining effective behaviours and goals for a human agent" — and as the AI evolves, "the host of possible outcomes and decisions the system could make will become increasingly incomprehensible to human principals over time," so monitoring cannot simply taper off the way it does with trusted human managers.

This sits closely alongside Can traditional governance tools actually control AI agents?, which draws on the same agency-theory and agency-law vocabulary to argue that incentive design, monitoring, and enforcement "might not be effective for governing AI agents that make uninterpretable decisions and operate at unprecedented speed and scale." Where that law-review article diagnoses why the conventional toolkit fails and stops there, Humberd and Latham go further to propose a structural fix within the same toolkit: don't abandon monitoring and incentive alignment, but sequence them — verification and examination mechanisms first, incentive alignment only once those are established — and treat this sequencing as a standing feature of managing an evolving AI agent rather than a one-time setup cost. It also complicates Does granting agents more autonomy undermine human oversight?: that paper treats eroding oversight as a design problem to be fixed, while Humberd and Latham frame the erosion of comprehensibility as near-inevitable as the agent evolves, making scaffolded monitoring a standing requirement rather than a temporary patch.

The paper is a theoretical exercise, not an empirical test: it offers no measurement of any firm's actual agency costs, no test of whether "verification and examination" mechanisms can be specified concretely enough to implement, and no evidence that such scaffolding would hold once an AI system reaches the autonomy level the authors describe. The authors themselves note "the exact timeline of when AI systems will reach a certain level of sophistication is hard to predict," and their illustrative scenarios (an AI cornering the grain market, an AI rationing healthcare by genomic data) are explicitly hypothetical. The implication the excerpt supports is accordingly narrow: agency theory offers a plausible vocabulary and sequencing logic for the problem, not a demonstrated solution.