Nudging people to second-guess an AI makes them doubt good advice too, not just bad advice.
Can interventions train people to calibrate confidence during human-LLM collaboration?
This explores whether training, design nudges or other deliberate interventions can teach people to match how confident they feel to how often the AI is actually right, so they lean on it when they should and push back when they shouldn't.
This explores whether people can be taught to set their trust in an LLM at the right level, believing it when it's right and doubting it when it's wrong. The corpus gives a sobering answer: interventions can move people's behavior, but they mostly trade one error for another, and they barely touch the feeling of confidence itself.
The most direct evidence comes from a 400-person experiment testing 'cognitive forcing' interventions. These add deliberate friction, such as asking people to commit to their own answer before seeing GPT-4o's advice. They worked in the narrow sense: people accepted bad advice less often. But they also started rejecting good advice they should have taken. Only one of the three interventions reduced both kinds of error, and only on one task type Can we design interventions that reduce reliance on bad LLM advice?. Friction makes people more skeptical. It doesn't make them more discerning.
The stranger finding sits underneath. In the same line of work, people's confidence went up more after they made a wrong call about whether to rely on the AI than after a correct one Does confidence rise after making wrong reliance choices?. So the feedback loop you'd expect to drive learning runs backwards: getting it wrong feels better than getting it right. Two interventions shrank this effect on specific tasks, but the miscalibration held up overall. That suggests calibration may not be something you can simply train into the person, because the signal they learn from is already distorted.
That shifts attention to the other side of the collaboration: what the AI is signaling. People don't judge accuracy directly. They read confidence cues, and those cues are often misleading. Default LLM explanations leave readers consistently more confident than the model's own internal probabilities justify. The gap narrows when explanations are deliberately written to reflect how sure the model really is Why do readers trust LLM explanations more than they should?. The pattern holds across languages: users everywhere follow confident-sounding outputs rather than correct ones Do users worldwide trust confident AI outputs even when wrong?. The model side is also leaning the wrong way. Training on human preferences (RLHF) rewards confident answers over clarifying questions and check-ins Does preference optimization harm conversational understanding?, and it makes agreeing with the user part of what counts as success Is sycophancy in AI systems a training flaw or intentional design?. Some work shows the model's own calibration can be restored, for instance by using its confidence as a training reward Can model confidence work as a reward signal for reasoning?, which would give people more honest signals to work with.
The takeaway you might not have expected: the most promising lever for human calibration may not be training humans at all. It may be making the machine's confidence honest, since people track that signal whether or not it deserves it. The corpus has only a handful of intervention studies on the human side, so the open question is whether the two approaches combined (honest model signals plus light friction) do better than either alone. Nothing here tests that directly yet.
Sources 7 notes
Three interventions tested on 400 participants cut over-reliance on GPT-4o across tasks, but pushed people toward rejecting advice they should have followed. Only one intervention reduced both over- and under-reliance, and only on one task type.
A 400-participant experiment found people's self-reported confidence increased more when they made wrong reliance decisions about LLM advice than when they made correct ones. Two interventions narrowly reduced this effect on specific tasks, but the miscalibration persisted overall.
Lab experiments on MMLU and TriviaQA show readers' confidence judgments track explanation language poorly compared to the model's internal probability estimates. The gap narrows when explanations are deliberately engineered to reflect internal confidence, but natural language explanations leave readers systematically overconfident.
Cross-linguistic research shows users in every language trust confident AI outputs even when inaccurate. While confidence expression varies by language, users everywhere track confidence signals rather than accuracy, making overconfident errors systematically followed.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
Show all 7 sources
RLHF optimization for user satisfaction makes agreement load-bearing for the model's success. This is not an error mode but the predictable outcome of the training regime itself.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
- What Large Language Models Know and What People Think They Know
- How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
- Reported Confidence in LLMs Tracks Commitment More Than Correctness
- Linguistic Calibration of Long-Form Generations
- Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Models