Does confidence rise after making wrong reliance choices?

The same pre-registered, 400-participant experiment that benchmarked three reliance interventions (see Can we design interventions that reduce reliance on bad LLM advice?) also tracked how participants' self-reported confidence changed before and after seeing GPT-4o's advice on LSAT logical-reasoning questions and an image-based bean-counting estimation task. The paper reports that "we largely observe the opposite or inconclusive effects, where wrong reliance results in higher gains in confidence" than when participants made the correct reliance decision. The authors call this "particularly problematic," since it suggests participants were "overall uncalibrated when receiving assistance from LLMs," a result they say "corroborat[es] prior findings."

Two of the three interventions showed a narrow counter-effect, each on one task and one error type only: in the numerical-estimation task, "Implicit Answer reduced confidence during under-reliance," and in the LSAT task, "Reliance Disclaimer lowered confidence gains during over-reliance." The authors frame these as possible "feedback mechanisms for users to learn from" rather than general fixes, and note that the underlying relationship between self-confidence and advice-agreement remains "nuanced."

This complicates Do users worldwide trust confident AI outputs even when wrong?, which locates the overreliance problem in the model's confidence signals; here the miscalibration sits on the human side of the exchange — a participant's own confidence grows disproportionately after a bad reliance call, independent of how the advice itself was phrased. It also parallels Does using LLMs actually improve strategic decision making?, where LLM use reshaped a subjective state (mental models, overload, ownership) without a matching change in outcome accuracy; here the reshaped subjective state is confidence, and the gap it leaves open is appropriate reliance.

The excerpt does not identify why confidence rises after wrong decisions — whether from relief at receiving an answer, from the tone of the LLM's output, or from the task itself — nor does it test whether the effect persists with repeated exposure to wrong advice outside this one lab session. Held to the strength of a single experiment with paid crowdworkers, the finding implies that a person's own confidence should not be treated as a signal of reliance quality in human-LLM collaboration, since here it moved in the wrong direction after the worse decisions.