When explanations are confident but tasks are hard, do people trust AI more exactly when they shouldn't?
How do task difficulty and explanation format interact in shaping reliance behavior?
This explores whether people lean on AI differently depending on how hard the task is and how the AI's explanation is presented (long or short, step-by-step or confident), and whether those two factors amplify or cancel each other.
This explores whether people lean on AI differently depending on how hard the task is and how the AI's explanation is presented, and whether the two effects compound. One caveat first: the retrieved material doesn't include a user study that varies difficulty and explanation format side by side and measures reliance. What it does have is the pieces of a mechanism, mostly from the model side. Put together, those pieces point to an uncomfortable conclusion: on exactly the tasks where readers most need the explanation to be a trustworthy signal, it is least likely to be one.
Start with the human side. When a task gets harder, people and LLMs both stop following the logical form and fall back on what sounds plausible. Content effects grow steadily as tasks get harder, from simple inference to syllogisms to the Wason selection task Do harder reasoning tasks trigger more semantic bias?. For reliance, that means that as a problem moves past what a user can check, the user is less able to judge an explanation on its logic and more likely to judge it on whether it feels right. That shift is the opening for the compounding traps described in Why do people trust AI outputs they shouldn't?: treating the AI's map as the territory, mistaking fluent intuition for reasoning, and having existing beliefs reinforced. Those traps multiply when they occur together, and hard tasks are where they are most likely to.
Now the explanation side. The most obvious format cue, length or how much step-by-step work is shown, turns out to be a poor signal of difficulty. Reasoning-trace length tracks difficulty only on problems that resemble the model's training data. Off that territory the link breaks, and length mostly reflects how familiar the problem looks Does longer reasoning actually mean harder problems?. Even where length and difficulty do line up, the best length follows an inverted U, and more capable models do better with shorter chains Why does chain of thought accuracy eventually decline with length?. A reader who sees "long, careful explanation" and concludes "hard problem, handled carefully" is reading a signal that doesn't mean what it appears to.
The deeper problem is that how an explanation is structured can matter more than whether it is correct. Chain-of-thought examples with invalid logic perform nearly as well as valid ones Does logical validity actually drive chain-of-thought gains?, and step-by-step reasoning is better described as imitating the shape of reasoning than as actual inference Why does chain-of-thought reasoning fail in predictable ways?. So a well-formatted explanation can look exactly the same whether the reasoning behind it is sound or broken. Add preference training that rewards confident answers over clarifying questions and understanding checks (grounding acts fall 77.5% below human levels Does preference optimization harm conversational understanding?), and the AI's presentation tends to look most assured when the user is least able to verify it.
The likely interaction, then, is not "a better format leads to better-calibrated reliance." It is closer to this: as difficulty rises, users depend more on surface features of the format, and those features carry less information about whether the answer is right. The corpus suggests a design implication worth testing directly. On hard tasks, explanations probably need features that don't depend on looking like reasoning, such as uncertainty the system has actually earned, explicit checks of what the user understood, or a signal of how far the problem is from familiar territory. More thorough-looking chains of thought are unlikely to do that job.
Sources 7 notes
Content effects intensify as task difficulty increases—from NLI to syllogisms to Wason selection—in both humans and language models. As working capacity is exceeded, both systems fall back on semantic priors instead of logical form.
Rose-Frame identifies map-territory confusion, intuition-reason conflation, and confirmation-bias reinforcement as traps that multiply their distorting effects when they co-occur. Evidence from cross-linguistic overreliance and architectural transformer biases confirms the compounding mechanism operates universally.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
Show all 7 sources
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
- Break the Chain: Large Language Models Can be Shortcut Reasoners
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap
- Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
- Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning