Can we design interventions that reduce reliance on bad LLM advice?

A pre-registered, randomized online experiment with 400 participants tests three "reliance interventions" meant to help people decide when to follow advice from GPT-4o, a large language model, versus their own judgment. Participants worked through two tasks — difficult LSAT logical-reasoning multiple-choice questions, and an image-based numerical-estimation task of counting beans in jars, with advice quality varied across five levels from "very low" to "very high" — answering once alone, then again after seeing LLM advice modified by one of three interventions or an unmodified baseline. The three interventions were Uncertainty Highlighting (flagging uncertain tokens in the model's answer), Reliance Disclaimer (a persistent disclaimer that adds usability friction), and Implicit Answer (withholding the direct answer so the user must work through the advice). The paper's central finding, stated in its abstract: "while interventions reduce over-reliance, they generally fail to improve appropriate reliance" — appropriate reliance being defined, following prior human-AI collaboration research, as "relying on the AI when it's correct, and relying on yourself when it's not." All three interventions cut over-reliance in at least some condition, but did so "at the expense of useful reliance," pushing people toward under-reliance rather than toward the appropriate target.

Reliance Disclaimer was the only intervention to reduce both over- and under-reliance, and only on the LSAT task. Uncertainty Highlighting "worsened subjective perceptions and had the least effect on reliance" of the three. Implicit Answer "induced better self-reliance in both tasks, but at the cost of requiring significantly more time for participants to process the information." The paper ties the under-reliance cost to "the cost-benefit framework from explainable AI literature," under which users disengage from an intervention "if they are more cognitively complex than the task itself" — so added friction can push people away from advice they should have used, not only advice they should have rejected.

This extends Does explaining AI decisions actually reduce user overreliance on it? by testing the opposite design move — adding friction rather than adding transparency — and landing on the same asymmetry: an intervention can shift behavior without shifting it toward the correct target. It also sits beside Do LLM explanations actually help people judge answer quality?, where sourcing and flagged inconsistencies selectively cut over-reliance on wrong answers; here, disclaimers and withheld answers show the same selective effect is hard to reproduce outside the explanation format, and is task-dependent (the disclaimer worked for LSAT but not for numerical estimation).

The excerpt tests these interventions only in a short, incentivized lab setting with Prolific crowdworkers who may have hedged their answers "to maximize their chances of earning some bonus payment," and only against one model (GPT-4o) and two tasks chosen for general accessibility rather than domain expertise. It does not establish how these interventions perform in naturalistic, long-term human-LLM collaboration, or with other models. The implication the evidence supports is narrower than add friction or add transparency: a reliance intervention validated only by its reduction in over-reliance may be solving half the problem, since the same design can simply trade over-reliance for under-reliance without making the underlying reliance decision any more accurate.