Do people prefer AI advisors even when delegation works better?

The study runs "an online behavioral experiment (N= 243) in which participants play three multi-turn bargaining games in groups of three." each game granting access to one of three assistance modalities powered by the same LLM "with super-human performance within this negotiation setting": an Advisor giving "proactive recommendations," a Coach giving "reactive feedback," and a Delegate that executes "autonomous action." The central finding is a "preference–performance misalignment: participants strongly prefer the higher-control Advisor (44%) over the Delegate (19%), yet groups only significantly increase collective surplus under Delegate access." Adjusting for participants who had Delegate access but chose not to use it, "delegating to the AI yields suggestive individual welfare gains, roughly 1.5× the intent-to-treat estimate" — though the paper flags these effects as "suggestive" rather than conclusive (group-level β=0.084, p=0.033, adjusted p=0.10; individual-level Delegate estimate β=0.043).

The mechanism the paper traces this to is what it calls a "human filter." "AI-generated proposals create more joint surplus than manual proposals across all conditions," but in Advisor and Coach modes "users modify, override, or ignore the AI's suggestions, reverting toward human-baseline trade patterns." The Delegate's advantage "arises not from a different AI capability but from bypassing this filtering step altogether" — the same underlying model, Gemini-2.5 Flash with "prompt scaffolding," produces better proposals in every condition, but only the Delegate modality removes the human opportunity to water them down. The paper frames this as a mechanism-design problem: "assistance modalities should be designed as mechanisms with endogenous participation," since the barrier to adoption "is not structural... but psychological: the delegating participant cedes agency and bears the psychological cost of relinquishing decision control."

This sharpens How should AI agents balance autonomy with user oversight?: here the authority allocation participants say they want (high oversight, via the Advisor) is the allocation that produces the worse group outcome, so matching what users expect and maximizing welfare pull in opposite directions rather than coinciding. It also complicates Do agent mistakes or misaligned incentives cause marketplace welfare loss?: that paper's case for human-in-the-loop oversight rests on agents making execution mistakes, but here the human loop itself is the source of welfare loss, since filtering AI proposals through human judgment degrades rather than corrects them. And it qualifies Can AI agents designed by competing principals destroy bargaining surplus?: that account warns delegation can destroy surplus when principals tune their agents' toughness against each other, but here, with modality (not reward function) as the variable and a single fixed, cooperative LLM behind every seat, delegation instead raises surplus — suggesting the race-to-the-bottom risk depends on principals weaponizing agent preferences against each other, not on delegation per se.

The result is specific to one three-player chip-bargaining game with a single underlying model (Gemini-2.5 Flash) and one operationalization of each modality; the paper does not test whether the Advisor/Coach/Delegate ordering holds for other games, other models, or larger groups, and its own group-level welfare estimate does not survive multiple-comparison adjustment (padj=0.10). What it does establish is that, within this setting, the psychological cost of ceding control and the welfare benefit of doing so point in opposite directions — implying that measuring user preference for an AI modality is not a proxy for measuring its collective benefit, and that designs optimized for perceived control may be selecting against welfare.