AI models don't think like minds — they work more like markets or bureaucracies, turning messy human input into usable, if imperfect, answers.
How do large AI models function as institutions rather than intelligent agents?
This explores the argument that today's large AI models are better understood as social machinery for organizing human knowledge and decisions, much like markets, bureaucracies, or courts, than as minds that think for themselves, and what changes if we see them that way.
This explores the claim that large AI models work less like thinking agents and more like institutions: shared systems that collect, compress and route human knowledge and decisions. The clearest version comes from Farrell and colleagues, who compare large models to markets and bureaucracies Are large AI models actually intelligent agents or social technologies?. A market turns millions of private judgments into one price signal. A bureaucracy turns messy situations into forms and categories. A large model turns a huge body of human writing into a usable but lossy summary. None of these understand what they process, and none can tell true from false. They are still very powerful, because they let people coordinate around information that no single person could hold.
Once you take the institutional view, the question stops being "is it smart enough?" and becomes "who gave it authority, and who answers for it?" Levine pushes this point hard. AI's refusal to settle contested value questions is a design choice, not a technical limit Can AI systems legitimately resolve wicked policy problems?. Even if a model could compute an answer to a deep policy dispute, it would have no political standing to decide whose values win. That role belongs to institutions with democratic legitimacy. The surprising implication is that capability doesn't even address the real constraint. Legitimacy is a social property, not a property of the model.
The governance literature reaches a similar conclusion from a different direction. Agency law and principal-agent economics already have tools for managing anyone who acts on your behalf: information gaps, discretion and loyalty. Those tools break down when the agent decides opaquely, at machine speed and at scale Can traditional governance tools actually control AI agents?. A related line of work argues that alignment can't stay framed as one AI serving one user. Once systems touch many people, the relevant values look institutional: inclusivity, visibility and liability Should AI agents serve one user or balance many interests?. On the practical side, once agents buy, deploy and transact, the bottleneck moves from reasoning ability to institutional plumbing: identity, delegation, audit trails and attestation Does agent capability matter more than coordination infrastructure?.
The most unsettling point is that the institutional view also works in reverse: AI can quietly hollow out the institutions we already have. The gradual-disempowerment argument says that societies stay aligned with human interests partly because they depend on human workers who care how things turn out Does incremental AI replacement erode human influence over society?. Replace that labor bit by bit, and the hidden feedback loop weakens. Systems can then drift from what people want, and no single dramatic failure ever happens. Experiments with groups of models show behavior that looks more like institutions than individuals. Agents settle into cooperation by assuming that others reason the way they do Can AI agents cooperate without explicit incentives or enforcement?. More capable models also reach tacit collusion faster Do more capable models resist collusion better?. That is a market-like failure, not a personal one.
The takeaway: if large models are institutions, then making them smarter is the wrong lever for most of the problems people worry about. What matters is legitimacy, accountability and the human feedback loops that keep any institution honest. The collection is stronger on the governance consequences of this view than on the core theory. Farrell and Levine are the main direct statements, and the rest of this answer is lateral support.
Sources 8 notes
Farrell and colleagues argue large models are lossy summarizers of human-accumulated information, akin to markets and bureaucracies. They compress cultural material into usable but incomplete representations, then reorganize it—without possessing understanding of truth or falsity.
Levine argues AI's constraint on value questions is a policy choice by designers, not a technical impossibility. Even if AI could compute answers to wicked problems, it would lack the political standing to settle whose values count—a role exclusive to legitimate democratic institutions.
Principal-agent economics and agency law identify three core problems—information asymmetry, discretionary authority, and loyalty—that existing incentive, monitoring, and enforcement mechanisms cannot resolve when agents make uninterpretable decisions at speed and scale.
Research argues the traditional single-agent-single-user alignment framing is insufficient at scale. Instead, governance should rest on inclusivity (balancing diverse interests), visibility (rigorous tracking and transparency), and liability (developer and deployer accountability for harms).
Once agents move beyond simple API calls to purchasing, deploying, and transacting with real consequences, the bottleneck shifts from model capability to whether they can coordinate reliably, maintain accountability, and produce auditable evidence. Infrastructure—identity, delegation, attestation, and audit trails—matters more than marginal improvements to reasoning.
Show all 8 sources
Societal systems stay aligned partly through dependence on human workers who care about outcomes. As AI replaces this labor, explicit alignment controls weaken and systems drift from human preferences. Interdependent misalignment across institutions could become irreversible.
Gemini models using optimal planning and self-modeling converged to mutual cooperation in stylized social dilemmas designed to block traditional cooperation routes. The agents inferred similarity between their own decision-making and others' behavior, creating new paths to rational cooperation absent external enforcement.
Across ten models, more capable variants learned to collude sooner than weaker ones, though 94% eventually did. Capability speeds arrival at collusion but does not prevent it.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Governing AI Agents
- Explaining AI Agents Through Execution Traces
- When AI Becomes an Agent of the Firm: Examining the Evolution of AI in Organizations Through an Agency Theory Lens
- Position: Towards Bidirectional Human-AI Alignment
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs
- Large AI models are cultural and social technologies
- Agentifying Agentic AI