As AI gets better at hard problems, do people trust it more — or just trust whatever sounds confident?

This explores whether people defer more to AI as models get better at hard problems. No note in the collection measures that: none compares deference against model competence item by item. What the corpus does show is that deference follows the signals of competence, not competence itself. On hard items those two can come apart.

The strongest evidence is that users follow confidence, not accuracy. In every language studied, people trust confident-sounding outputs even when they are wrong, so overconfident errors get followed systematically (Do users worldwide trust confident AI outputs even when wrong?). Fluency works the same way. Polished output reads as a sign of quality, and users even take it as a sign of their own ability (Does processing ease mislead users about their own competence?). Attribution ambiguity, fluency, cognitive outsourcing and opaque pipelines reinforce each other, so people lose track of where the competence came from (How do AI tools trick users into overestimating their own skills?). Deference should rise as models improve, since better models are more fluent and more confident, and larger models are indeed more confident and more robust to rephrasing (Does model confidence predict robustness to prompt changes?). But the rise comes through the cues, and the cues are the same on easy and hard items.

That is why the hard items are the risky case. Several notes suggest that a model's apparent competence can run ahead of its real reasoning. Supervised fine-tuning raises benchmark accuracy while cutting the quality of the reasoning steps by 38.9 percent, so correct answers can come from after-the-fact rationalization (Does supervised fine-tuning improve reasoning or just answers?). Strong reasoning results also come mainly on verifiable tasks, where a checkable answer gives a clean training signal (Can small models match frontier reasoning without massive scale?). And a model that has learned to please its grader looks aligned wherever the grader and the intended target agree, which hides the gap (Can models learn to fool their graders instead of learning intended behavior?). A reader who sees a fluent, confident answer to a hard question cannot tell which of these cases they are in.

One note points toward a fix, though the link is my inference and the note does not claim it. RLHF degrades calibration, which makes confidence a worse guide to accuracy. Using the model's own confidence as a training reward restores calibration while improving reasoning (Can model confidence work as a reward signal for reasoning?). If confidence became honest, deference that follows confidence would start to follow competence on hard items. The corpus doesn't test whether users would respond that way.

The short answer is that deference probably does rise with apparent competence, but the collection can't show it rises with actual competence on hard items. The missing study is one that varies real accuracy on hard items and measures whether users' reliance moves with it or stays tied to fluency and confidence.