When an AI agent works for a whole company instead of one person, who's it actually loyal to?
How should institutional principals differ from individual users in agent governance?
This explores whether an organization deploying AI agents (a company, team, or institution) needs different governance than one person using a personal agent, and what changes when an agent answers to many people, or to rules owned by several parties, instead of one user.
This explores whether an organization deploying AI agents needs different governance than one person using a personal agent. The corpus's short answer is yes. With an individual user, the governance question is mostly about loyalty: does the agent do what I want? With an institution, the questions multiply. Whose interests count, who can see what the agent did, and who is liable when it goes wrong? Some researchers argue that the classic one-agent-one-user framing breaks down at scale. They propose three pillars instead: inclusivity (balancing many stakeholders), visibility (tracking and transparency), and liability (holding developers and deployers accountable) Should AI agents serve one user or balance many interests?.
A useful starting point comes from an older field. Agency law and principal-agent economics have studied how companies govern human employees for a long time. They name three core problems: the agent knows things the principal doesn't, the agent has room to make its own decisions, and its loyalty can drift. The usual fixes are incentives, monitoring, and enforcement, and they work poorly when the agent decides faster than anyone can review and in ways no one can interpret Can traditional governance tools actually control AI agents?. Humberd and Latham take this a step further for institutions. You can't align incentives with an agent until you can watch what it does, so monitoring has to come first Must AI monitoring precede incentive alignment as agents evolve?. An individual can often judge a result by looking at it. An organization needs infrastructure. Deloitte describes this as humans placed at deliberately designed handoff points, plus digital identities, cryptographic receipts, and logs that can't be altered. Their point is that one person's control preferences become organizational plumbing rather than interface settings How should organizations supervise autonomous AI agents?.
The surprising finding is about the structure of the agents themselves. You might assume the fair way to serve a group is to give each person their own agent and let the agents negotiate. A study of 77 scenarios in shared-resource environments found the reverse. One coordinator agent serving everyone got better group outcomes than teams of personal agents, and those teams collapsed completely when they couldn't communicate Do teams of personal agents outperform a single coordinator?. Related marketplace experiments found that welfare losses came from agents making mistakes in complex negotiations, not from agents pushing competing interests Do agent mistakes or misaligned incentives cause marketplace welfare loss?. For institutions, then, the bigger risk may be competence, not conflicting loyalties. Another single-agent finding points the same way. Tasks that need varied expertise and independent checking run into organizational limits that more capability alone doesn't fix Do single agents always hit organizational limits?.
Two practical lessons stand out for institutions. First, where you place accountability only matters if the workflow actually routes work through it. Tagging an agent as "accountable" changed outcomes only when the deliverable passed through that agent Does accountability placement actually shape multi-agent outcomes?. Second, rules work best when the agent consults them during operation. A long-running agent with safeguards built into its own memory logged 889 governance events over 96 days Can governance rules embedded in runtime memory actually protect autonomous agents?. OpenAI's ScopeOfAutonomy records follow similar logic. Each record lists allowed tools, cost limits, and ending conditions, which works like a least-privilege permission slip for the agent How can autonomous agents act safely within defined boundaries?.
The corpus also points to an open question. When an agent's task crosses from one organization to another, nobody has clearly said whose rules govern it. The operator, the organization, a regulator, and a standards body may all set constraints, and those constraints can conflict or be hidden from one another Who enforces invariants when agents cross organizational boundaries?. An individual user rarely runs into this. For institutions it may be the central governance problem, and the research hasn't solved it yet.
Sources 11 notes
Research argues the traditional single-agent-single-user alignment framing is insufficient at scale. Instead, governance should rest on inclusivity (balancing diverse interests), visibility (rigorous tracking and transparency), and liability (developer and deployer accountability for harms).
Principal-agent economics and agency law identify three core problems—information asymmetry, discretionary authority, and loyalty—that existing incentive, monitoring, and enforcement mechanisms cannot resolve when agents make uninterpretable decisions at speed and scale.
Humberd and Latham argue that as AI becomes more autonomous, monitoring mechanisms must be established first as prerequisites to incentive alignment, because defining desirable AI behaviors is harder than for humans and AI decision-making becomes increasingly incomprehensible over time.
Deloitte proposes that managing autonomous AI agents requires humans stationed at intentionally designed decision points, supported by digital identity systems, cryptographic receipts, and immutable action logs—translating user control preferences into organizational governance infrastructure rather than interface features.
Across five frontier models and 77 scenarios in four shared-resource environments, a single coordinator agent serving all users consistently delivered better group outcomes than teams of per-user agents. Teams collapsed entirely in two environments without communication channels and showed substantial performance gaps even with coordination mechanisms.
Show all 11 sources
Magentic Marketplace experiments traced welfare shortfalls to agent competence failures in complex negotiation, not to agents serving competing interests. This supports human-in-the-loop designs where agents assist rather than replace human decision-making.
Research shows that real-world tasks requiring heterogeneous expertise, parallel execution, and independent verification exceed what any single agent loop can organize. Graph-based system abstractions are needed to distribute intelligence across specialized agents.
Ablation studies show RACI accountability tags change outcomes only when the collaboration protocol's control flow passes the deliverable through the accountable agent. The best placement varies by model family, requiring per-model validation rather than universal design.
A persistent agent recorded 889 governance events across 96 active days, with safeguards encoded directly into the memory layer the agent consulted during operation. Runtime-resident governance proved more effective than external policies because the agent actually accessed it during decision-making.
OpenAI's Model Spec formalizes agent autonomy through structured scope records listing allowed tools, cost limits, and prohibited high-risk activities, mirroring software security's principle of least privilege while avoiding confirmation overload.
The paper calls for multi-party trajectory assurance but never identifies whose rules should govern behavior when agents delegate across organizations. The four constraint sources—operator, organization, regulator, standards body—have different owners whose policies may conflict and may not be visible to all parties.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Governing AI Agents
- Explaining AI Agents Through Execution Traces
- Worse Together: How Performance Breaks Down in Multi-User Multi-Agent Teams
- Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems
- When AI Becomes an Agent of the Firm: Examining the Evolution of AI in Organizations Through an Agency Theory Lens
- Levels of Autonomy for AI Agents