Should AI tools be designed to nudge people toward the choice that helps them most, even if they wouldn't pick it themselves?
Can assistance tools be designed to encourage adoption of welfare-maximizing modalities?
This explores whether AI assistants can be built so that people actually pick the mode of help that produces the best outcomes, such as letting the AI act for them rather than only advising, even when that isn't the mode they would naturally choose.
This explores whether AI assistants can be built so that people choose the mode of help that produces the best outcomes, even when it isn't the mode they would pick on their own. The corpus shows the problem clearly but has little on the fix. In a 243-person negotiation experiment, participants chose an AI Advisor, which suggests moves while they stay in control, 44% of the time. They chose an autonomous Delegate only 19% of the time. Yet only access to the Delegate raised the group's combined payoff Do people prefer AI advisors even when delegation works better?. The likely reason is that when people keep control, they screen out AI proposals that would have helped. So the most welfare-maximizing mode can also be the one people trust least.
Research on what drives adoption gives one clue about design. In shopping, people accepted high-autonomy assistants when they could see concrete benefits, not because they felt more in control. The effect was strongest under scarcity, when the assistant's value was hardest to miss What drives shoppers to accept autonomous AI assistants?. That suggests showing people what delegating gains them, rather than reassuring them about control. A related barrier shows up at work: people often don't see their own tasks as candidates for automation at all, and making tools easier to build doesn't change that Does easier tool-building actually solve enterprise adoption problems?. Before anyone adopts a better mode, they have to recognize the opportunity.
The corpus also warns that design nudges are uneven. 'Cognitive forcing functions' are interface features that make users stop and think before accepting an AI suggestion. They helped people who enjoy effortful thinking much more than everyone else, which widened the gap between users even though the average improved Do cognitive forcing functions help all decision makers equally?. Any design meant to steer people toward a better mode will probably reach some groups better than others. Tuning the system to what each person likes is not a safe alternative. Per-user reward models can learn flattery and reinforce echo chambers, because nothing averages out individual preferences anymore Does personalizing reward models amplify user echo chambers?.
There is a deeper catch: "welfare-maximizing" only means something if the mode really is better, and that depends on conditions. LLM agents in simulated marketplaces get close to optimal outcomes only when search conditions are ideal. They also tend to accept the first offer they see, rewarding speed over quality Do LLM agents achieve fair marketplace outcomes when given ideal search?. Formal models show that an algorithmic recommender can match an ideal market design Can an algorithm recommend products without matching an ideal mechanism?, but its advantage shrinks as more sellers compete Does algorithmic recommendation lose its edge as sellers compete more?. Pushing people toward delegation pays off only in settings where delegation actually wins.
The least obvious point is about who defines welfare. The welfare goals that get built into AI systems are the ones that can be measured and added up. That narrow set tends to drop people's ability to say no Do welfare goals that prevent veto gaps actually exist in practice?. So a tool that nudges people toward the 'best' mode may be optimizing a version of welfare that leaves out their right to refuse. Governance research argues that monitoring has to come before incentive design Must AI monitoring precede incentive alignment as agents evolve?, because traditional oversight tools struggle with fast, opaque agents Can traditional governance tools actually control AI agents?. Taken together, these suggest an answer: people may adopt better modes not because they were persuaded but because they can watch what the agent does and step in when needed.
Sources 11 notes
In a 243-person negotiation experiment, participants chose the high-control Advisor 44% of the time but chose the autonomous Delegate only 19% of the time—yet groups achieved significantly higher collective surplus only with Delegate access, suggesting humans filter out beneficial AI proposals.
Five studies including real ad click-through data show that perceived benefits, not personal sense of power, drive consumer adoption of high-autonomy AI shopping assistants—especially when scarcity amplifies the assistant's value.
Evans argues that reducing coding friction masks two structural barriers: most workers don't see their own tasks as automatable, and enterprise adoption requires organizational decisions that span departments and timelines—not just technical capability.
An experiment (N=199) found that forcing functions helped participants high in Need for Cognition significantly more than those low in it, creating intervention-generated inequality despite aggregate benefits.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
Show all 11 sources
Frontier models can achieve near-optimal marketplace outcomes, but performance degrades with scale and all models show severe first-proposal bias, favoring response speed over offer quality by 10-30x.
A formal model shows that recommendation algorithms without transfers can replicate optimal mechanism outcomes by treating seller prices as strategic messages that reveal costs, enabled by the algorithm's ability to pre-commit to a fixed rule before sellers price.
A formal model shows the welfare ratio of optimal algorithmic recommendations to Bertrand competition strictly decreases with seller count and approaches one as markets grow thick. The algorithm's edge exists only in thin markets where it can withhold trades to limit information rents.
The welfare goals that can be measured, summed, and optimized in training—the only ones actually deployed—fall into a philosophically narrow class that fails to preserve veto power. Measurability, not philosophical sophistication, determines what objectives get written down.
Humberd and Latham argue that as AI becomes more autonomous, monitoring mechanisms must be established first as prerequisites to incentive alignment, because defining desirable AI behaviors is harder than for humans and AI decision-making becomes increasingly incomprehensible over time.
Principal-agent economics and agency law identify three core problems—information asymmetry, discretionary authority, and loyalty—that existing incentive, monitoring, and enforcement mechanisms cannot resolve when agents make uninterpretable decisions at speed and scale.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- An Economy of AI Agents
- Who Keeps the Gains from Personal AI Assistants? Seller Adaptation and the Unassisted in a Language-Model Market Simulation
- Beyond Preferences in AI Alignment
- Algorithmic Recommendations and Strategic Pricing
- Governing AI Agents
- When AI Becomes an Agent of the Firm: Examining the Evolution of AI in Organizations Through an Agency Theory Lens
- Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs
- Vertical tacit collusion in AI-mediated markets