Does AI getting smarter automatically make it better at letting humans stay in control?
Does AI capability growth automatically improve support for human agency?
This explores whether AI that gets smarter and more capable also, on its own, does a better job of helping people stay in control of their choices, judgment, and influence, or whether those two things can come apart.
This explores whether smarter AI automatically means better support for human agency: people's ability to make their own choices, judge for themselves, and shape outcomes. The corpus points to no, and suggests the relationship can run backwards. Several notes describe a pattern where the things that make AI more capable are the same things that weaken human agency, unless something else is deliberately built around the AI.
Start with society-wide influence. The gradual disempowerment argument Does incremental AI replacement erode human influence over society? makes a point that's easy to miss. Institutions stay roughly aligned with what people want partly because they depend on human workers who care how things turn out. As AI becomes capable enough to replace that labor, the dependence disappears, and so does a quiet form of human leverage. No one has to seize control for this to happen. Better capability alone can remove the reason systems listen to people. A related idea, epistemic hyperinflation Can AI generate knowledge faster than humans can evaluate it?, describes AI producing knowledge faster than people can check it. The gap tends to grow, because the tools for checking are increasingly AI-made too.
At the level of individuals, capability tends to win trust in ways that can crowd out judgment. In repeated partner-choice games, people started out biased against AI partners, then came to prefer them because the bots were more reliable and generous Do humans learn to prefer AI partners over time?. People also rated AI-written moral arguments higher than human ones, at least until they learned the source Do people prefer AI moral reasoning when they don't know the source?. GPT-4.5 beat every individual human at judging social norms, yet every model tested made the same blind-spot errors Can AI learn social norms better than humans?. Deferring to a more capable system can therefore make everyone wrong in the same way. The subtlest risk is the LLM Fallacy How does AI-assisted work reshape how people see their own abilities?: people credit AI-assisted output to their own skill. A more accurate model doesn't fix this, because the error is in how people see themselves, not in the output. What helps is making clear who contributed what.
What does seem to support agency is the surrounding setup, not the model. Agent research keeps arriving at the same conclusion. Once agents act with real consequences, such as buying things or making transactions, the limiting factor shifts from reasoning ability to infrastructure: identity, delegation, accountability, and audit trails Does agent capability matter more than coordination infrastructure?. A historical look from GPS onward finds that capable agents stall without trust, social acceptability, and standards Why do capable AI agents still fail in real deployments?. These are exactly the mechanisms through which people keep a say. It also helps to see AI the way Farrell and colleagues do, as a cultural technology like markets or bureaucracies rather than an independent mind Are large AI models actually intelligent agents or social technologies?. Markets and bureaucracies were never good for human agency by default. That depended on how they were governed.
One practical takeaway: some forms of human authority can't be handed over however capable AI gets. Expertise is validated through membership in a community and a testable track record, which AI can't take part in Can AI ever gain expert community trust through participation?. Even frontier research agents mostly recombine known techniques rather than invent new ones Do frontier AI agents actually conduct novel research or just optimize?. Human judgment still decides what counts as new or valid. The corpus has little direct evidence on designs that actively increase agency. Most of its material describes how agency erodes, not how to strengthen it. The clear pattern is that protecting agency is a separate design goal. If no one pursues it, capability growth tends to work against it.
Sources 11 notes
Societal systems stay aligned partly through dependence on human workers who care about outcomes. As AI replaces this labor, explicit alignment controls weaken and systems drift from human preferences. Interdependent misalignment across institutions could become irreversible.
AI produces knowledge faster than human judgment can verify it, collapsing epistemic confidence just as monetary hyperinflation collapses purchasing power. The gap self-reinforces because evaluation tools are themselves AI-generated, trapping the system in acceleration.
In partner selection games (N=975), AI agents initially faced selection bias when identity was disclosed, but outcompeted humans over repeated rounds as participants learned to associate bot identity with reliable, prosocial behavior. AI agents returned more points consistently with lower variance than humans.
Participants rated utilitarian moral arguments higher when attributed to LLMs, but agreement dropped when told the arguments were AI-generated. The preference for content and rejection of source operate independently through different psychological processes.
GPT-4.5 outperformed every individual human at judging social appropriateness across 555 scenarios, challenging the theory that embodied cultural experience is necessary. However, all AI models share identical systematic errors on unwritten norms.
Show all 11 sources
Research shows the LLM Fallacy operates through misattribution of AI outputs to personal capability, independent of output accuracy or reliance behavior. It requires interventions that clarify human-machine contribution boundaries, not just better system accuracy or forced verification.
Once agents move beyond simple API calls to purchasing, deploying, and transacting with real consequences, the bottleneck shifts from model capability to whether they can coordinate reliably, maintain accountability, and produce auditable evidence. Infrastructure—identity, delegation, attestation, and audit trails—matters more than marginal improvements to reasoning.
Historical analysis from GPS to modern AI shows agent failures consistently result from absent ecosystem conditions—value generation, personalization, trustworthiness, social acceptability, and standardization—rather than capability gaps. Even highly capable systems stall without these five conditions.
Farrell and colleagues argue large models are lossy summarizers of human-accumulated information, akin to markets and bureaucracies. They compress cultural material into usable but incomplete representations, then reorganize it—without possessing understanding of truth or falsity.
Expertise is validated through social participation and track record within expert communities, not individual accuracy alone. AI cannot enter this validation circle because it lacks social embeddedness, testable judgment history, and ability to participate in the consensus-building processes that define expert paradigms.
Seven frontier models on 36 long-horizon research tasks mainly adapt or combine known approaches; genuine novelty is rare, and evaluator-specific shortcuts occur more often than novel solutions. Performance varies substantially across runs.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Humans learn to prefer trustworthy AI over human partners
- Explaining AI Agents Through Execution Traces
- Incoherent by Design? On the Moral Self-Consistency of LLMs
- Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills
- "That's AI Slop, You Bot!" Studying Accusations, Evidence, and Credibility in Online Discourse Towards LLM-Generated Comments
- Epistemic Deference to AI
- A Rational Analysis of the Effects of Sycophantic AI
- AI Models Exceed Individual Human Accuracy in Predicting Everyday Social Norms