Show people what leaning on AI costs them and they ask it for answers half as often; a reward for effort did nothing measurable.
Does metacognitive feedback reduce reliance on AI-generated answers?
This explores whether telling people about their own thinking habits (for example, pointing out that they're handing answers off to an AI instead of working them out) actually changes how much they lean on the AI, and whether that leaves them better off when the AI is gone.
This explores whether feedback about people's own thinking habits, such as showing them the cost of letting an AI hand them answers, changes how much they rely on it. The corpus has one direct test, and the result is striking. In a preregistered experiment with 704 people, feedback that pointed out the costs of offloading halved the number of times people asked the LLM for answers. Their scores on a later test taken without AI help rose by 51% Can metacognitive feedback stop students from offloading to AI?. The surprising part is the control: paying people a reward for effort did nothing measurable. People didn't need a bribe to try harder. They needed to see what the shortcut was costing them.
Why would seeing the cost matter so much? Other notes point to a hidden problem. When an AI produces a polished answer, the ease of reading it can feel like your own competence. Users take that fluency as a sign that they understand, even though they didn't produce the answer Does processing ease mislead users about their own competence?. If that's the trap, metacognitive feedback works by breaking the illusion. It shows people the gap between 'this made sense when I read it' and 'I can do this myself.' A related framing describes three biases that stack: mistaking the AI's map for the territory, treating quick intuition as reasoning, and having your existing beliefs confirmed. Together they produce a slow drift in what people think they know Why do people trust AI outputs they shouldn't?.
The same weakness shows up in how people judge the AI, not only themselves. Across every language studied, users follow how confident the AI sounds rather than whether it's right, so confidently wrong answers get trusted Do users worldwide trust confident AI outputs even when wrong?. This suggests that over-reliance is less about laziness and more about people reading the wrong signals. That would explain why feedback that redirects attention works better than a reward for effort.
There's a design alternative to adding feedback after the fact: build reflection into the AI itself. In a lab study of 80 people, assistants that combined reflection questions with advice led to better decisions than assistants that only advised or only asked questions Do reflection questions help people make better decisions with AI?. These are two routes to the same goal. One tells you about your habits after you've used them. The other prompts you to think before the answer arrives.
A caveat: the direct evidence here comes from one well-run study in a learning setting. The corpus doesn't show whether the effect lasts over months or carries over to work settings. That matters because AI may produce content faster than people can check it Can AI generate knowledge faster than humans can evaluate it?, and in that case one-off feedback could wear off under the volume.
Sources 6 notes
In a 704-person preregistered experiment, feedback that highlighted offloading costs reduced answer requests to an LLM by half and raised unaided test scores by 51%. An effort-based reward showed no measurable effect on either outcome.
High-quality AI output triggers a metacognitive heuristic: users experience fluency as a signal of their own capability, even though they didn't generate it. This self-directed fluency illusion systematically inflates perceived competence because LLMs optimize for fluency regardless of user understanding.
Rose-Frame identifies map-territory confusion, intuition-reason conflation, and confirmation-bias reinforcement as traps that multiply their distorting effects when they co-occur. Evidence from cross-linguistic overreliance and architectural transformer biases confirms the compounding mechanism operates universally.
Cross-linguistic research shows users in every language trust confident AI outputs even when inaccurate. While confidence expression varies by language, users everywhere track confidence signals rather than accuracy, making overconfident errors systematically followed.
A lab study of 80 participants found that thinking assistants combining reflection questions with advice significantly outperformed agents that only advised, only questioned, or did neither. Prioritizing Socratic questioning over authoritative answers enhanced cognitive outcomes.
Show all 6 sources
AI produces knowledge faster than human judgment can verify it, collapsing epistemic confidence just as monetary hyperinflation collapses purchasing power. The gap self-reinforces because evaluation tools are themselves AI-generated, trapping the system in acceleration.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Toward Measuring AI's Effects on Skill Formation: The Stock-Formation Gap
- The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows
- A Rational Analysis of the Effects of Sycophantic AI
- When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models
- “Understanding AI”: Semantic Grounding in Large Language Models
- Metacognition in LLMs: Foundations, Progress, and Opportunities
- How AI Impacts Skill Formation
- Humans overrely on overconfident language models, across languages