Explaining Too Much? Understanding How Large Language Model Reasoning Traces Influence Performance and Metacognition

Large Language Model interfaces are increasingly verbose, exposing intermediate reasoning traces alongside final answers. Traces are framed as transparency mechanisms, yet it is unclear how people use them to solve problems. We report a preregistered betweensubjects study (N = 559) in which participants solved ten LSAT-style reasoning problems under one of three conditions: an Answer-only baseline, a Full-trace revealed before the answer, and a Summary-trace presented alongside the answer. Summaries preserved task performance at the no-trace baseline while significantly elevating trust and hedonic appeal, establishing that trace exposure shifts subjective appraisal of the interaction without bringing performance benefits. Under an open-weight reasoning model exposing verbose intermediate output, full traces additionally impaired performance relative to the answer-only baseline. Across all conditions, participants substantially overestimated their performance, and no trace format supported calibrated self-evaluation. Further analysis indicates that hedonic appeal, not trust, carries the indirect path to overestimation, consistent with a processing-fluency account.

Reasoning traces are best understood as user-facing interface artifacts rather than transparent windows into model cognition, and calibration is unlikely to emerge from the traces themselves and may best be scaffolded by interactions that elicit users’ own reasoning first.

Introduction. Large Language Models (LLMs) are becoming increasingly verbose, with users now exposed not only to final outputs but also to intermediate “reasoning traces”, chain-of-thought-style outputs that resemble the model’s inner workings [10].

Showing reasoning traces carries the promise of transparency. They are meant to help users make sense of AI answers and scaffold complex workflows [5, 35, 39, 40]. Yet prior research in XAI and HCI reports mixed findings. High verbosity can induce an illusion of understanding (e.g., illusions of knowledge [25]), and fluent justifications may lead users toward overreliance even when the underlying conclusion is wrong [14, 38]. Moreover, a model’s “chain-of-thought” is not always faithful to its intermediate steps and can function as a polished, but misleading, justification [10, 36, 53].

Despite this, providers explicitly market reasoning traces as tools for transparency [36], and while generating these traces is costly, traces are billed at output rates and largely increase the amounts of tokens as compared to a direct answer [5, 50]. The design question is thus whether exposing reasoning traces delivers the transparency providers claim, with benefits for its users.

At the same time, while AI has the potential to improve task performance [9, 49, 61], users often believe they perform better when assisted by AI [23, 32, 56] and struggle to align confidence with correctness. This highlights the core challenge of understanding not only how people can make better decisions with AI, but how AI interfaces shape metacognition, that is, users’ ability to monitor and evaluate their own decisions and regulate reliance during interaction with AI.

In this paper, we investigate how the shape of reasoning traces affects performance and metacognition in human-AI reasoning contexts. More specifically, we ask (1) what the benefits and drawbacks of different reasoning-trace formats are, (2) how these formats affect task performance, (3) how they shape metacognition, and (4) whether they can support calibrated reliance on AI.

To address these questions, we present findings from a preregistered between-subjects experiment (N = 559) in which participants solved ten LSAT-style logical reasoning problems with AI assistance under one of three interface conditions that span the practically relevant design space for exposing model reasoning:

• Answer-only condition (N = 187): An answer-only baseline that reflects early LLM interfaces.

• Full-trace condition (N = 183): A full reasoning trace shown before the model’s final answer, which remained hidden behind a button press. This ensured that participants engaged with the trace itself rather than the verdict, and mirrored the verbose outputs of current open-weight reasoning models.

• Summary-trace condition (N = 189): A summary of reasoning traces presented together with the answer, of the kind now adopted by several commercial systems (see [5, 39]).

These three formats capture the dominant ways in which reasoning is currently surfaced to users and allow us to control verbosity in our study. By isolating the content of the trace from how it is revealed, we can ask whether traces help, hurt, or need to be calibrated, and at what level of verbosity, if any, they support metacognitive accuracy rather than merely inflating the appearance of understanding. LSAT-style reasoning has been established as a robust measure of both performance and metacognition [54], has been used in prior human-AI interaction research [23], and serves as a benchmark for evaluating LLMs [29].

Across five outcomes (task performance, metacognitive accuracy, task load, trust, and usability), our results converge on a consistent dissociation: trace exposure shifts subjective appraisal of the interaction without bringing performance benefits. Summary traces preserved task performance at the no-trace baseline while significantly elevating trust and hedonic appeal which provides a contrast in which the underlying LLM model was held constant (GPT-5). Under an open-weight reasoning model exposing verbose intermediate output (GPT-oss-20b), full traces additionally impaired performance relative to the Answer-only baseline. Across all three conditions, participants substantially overestimated their performance, and no trace format supported calibrated self-evaluation, however, overestimation was largest under Full-trace. A mediation analysis indicates that hedonic appeal, not trust, carries the indirect path to overestimation, consistent with a processing-fluency account of metacognitive misattribution. Our contributions are threefold. First, we provide a controlled comparison of reasoning-trace formats in terms of cognitive and metacognitive performance. We isolate verbosity as a design dimension and show that summarized traces preserve performance without improving calibration while elevating subjective appraisal of the interaction.

Related work. In this section, we focus on the role of metacognitive processes in human cognition and human–AI interaction. We first introduce central concepts of human metacognition and examine how mechanisms such as self-monitoring and evaluation influence problem-solving, learning, and decision-making. We then consider how these processes are affected when people collaborate with AI systems, highlighting both opportunities and risks for cognitive performance and user judgment. Finally, we introduce key concepts related to reasoning traces and AI explanations, which aim to increase transparency but may also introduce new challenges for calibration and usability.

2.1 Metacognition and Human Reasoning Metacognition refers to the processes by which people monitor, plan, and evaluate their own thinking [26]. These processes are essential for complex problem-solving, learning, and optimizing behavior. Two primary constructs have been emphasized in psychological research: metacognitive accuracy and metacognitive sensitivity.

Metacognitive accuracy describes how closely individuals’ self-evaluations align with their actual performance [17, 26]. High accuracy allows people to recognize their limitations and make informed adjustments, such as seeking more information, revising a strategy, or making a conservative choice. Accuracy can be further decomposed into bias (systematic over- or underestimation) and noise (random fluctuations) [24].

Metacognitive sensitivity refers to the ability to discriminate between correct and incorrect judgments, often measured through confidence ratings after the decision [26]. High metacognitive sensitivity entails that an individual’s confidence ratings accurately reflect their performance: high confidence corresponds to correct judgments, and low confidence to incorrect ones.

Both constructs show substantial individual differences [1, 30] and are affected by external aids. People often misattribute externally generated information to themselves [28, 60], and reading fluent explanations inflates perceived understanding even when comprehension has not actually improved [25]. These findings establish an underlying expectation that any system that provides articulate external reasoning may shift users’ self-evaluations, regardless of whether it improves their actual performance.

2.2 Metacognition in Human-AI Interaction Augmenting human intellect has long been a core theme in HCI, as highlighted by Engelbart [22]. Research has emphasized the potential of AI for improving human performance [9, 49, 51, 61]. However, performance gains come with hidden risks related to how users perceive and rely on these systems.

Method. 3 Research Model Our study systematically investigates the benefits and drawbacks of reasoning traces for task performance, how they shape metacognition and calibration, and the usability and user-experience trade-offs users may encounter across explanation formats.

Note that we investigate traces from a user-facing perspective, that is, as a class of XAI output that users respond to regardless of its internal validity. Thus, the relevant question for our study is not whether a trace accurately reflects the model’s computation, but whether and how it shapes the user’s metacognitive processes. This framing is consistent with the XAI literature’s distinction between proxy and faithful explanations [10, 27, 53], and positions the faithfulness problem as a further design risk: Users may calibrate their reliance on outputs that are themselves unreliable representations of model “thinking”, amplifying the metacognitive consequences.

We treat the content format of the trace as a single design dimension, ranging from no reasoning traces to summaries to full reasoning traces, and ask how variation along this dimension shapes performance, metacognition, trust, task load, and usability.

Five preregistered research questions and corresponding hypotheses structure this contrast:

• RQ1 Metacognitive accuracy: As fluent or verbose external information has been shown to inflate perceived understanding without improving judgment [3, 25], we expect summaries of reasoning traces to support more accurate self-evaluation than full reasoning traces or no traces conditions. That is H1: Summary-trace>Full-trace > Answer-only in metacognitive accuracy.

• RQ2 Task load: Furthermore, as long explanations increase reading time and effort without proportional benefit [12, 55], we expect full reasoning traces to impose the highest task load. That is H2: Full-trace > Answer-only and Full-trace > Summary-trace; Summary-trace ≳Answer-only on the NASA-TLX.

• RQ3 Trust and confidence: Because fluent justifications inflate perceived understanding [25], and traces can shape user judgment independently of their content validity, triggering the processing-fluency heuristic [4], the most verbose format should yield the highest confidence and trust. That is H3: Full-trace ≥Summary-trace > Answer-only.

• RQ4 Objective performance: As explanations can scaffold complex reasoning [9, 52], we predict that any trace will improve accuracy over an answer-only baseline. That is H4: Full-trace ≥Summary-trace > Answer-only.

• RQ5 Usability preference: Finally, because concise justifications are typically easier to use while verbose ones risk overwhelming users [37], we expect reasoning trace summaries to be preferred over full reasoning traces.

That is H5: Answer-only ≥Summary-trace > Full-trace on the SUS.

Taken together, these predictions imply that reasoning traces should yield more trust, more confidence, and more accurate task performance and, at least in summarized form, lead to better calibration, with task load as the main cost of full traces.

4.1 Study Design We build on the implementation introduced by Fernandes et al. [23], which examined how AI assistance can improve performance while undermining metacognitive accuracy. We extended this approach to focus on LLM reasoning traces.

The study uses a between-subjects design with three conditions: Answer-only, Full-trace, and Summary-trace. The pre-registered sample is N= 570 (190 per condition), based on an a priori power analysis targeting the smallest effect of interest. After exclusions, the final analyzed sample was N= 559.

4.1.1 Models and Interface Rationale. Each problem appeared on the left-hand side of the screen, while an AI chat (using GPT-5 or gpt-oss-20b, depending on condition) was displayed on the right.

We used OpenAI’s ChatGPT due to its widespread adoption in cognitive performance tasks [11, 19, 20] and because recent reasoning models allow access to a reasoning summary. Since GPT-5 provides summaries rather than full reasoning traces, we employed gpt-oss-20b for Full-trace condition, as it openly exposes complete reasoning traces.

Discussion. The five preregistered predictions divide into a single confirmation and four divergences, revealing an informative pattern. Contrary to H4, participants who were exposed to Full-trace performed significantly worse and overestimated their performance to a greater degree than those who were not exposed to any trace. Qualitative patterns indicate that this is mediated by offloading and guided reasoning rather than by genuine critical thinking, as evidenced by the model traces. Traces acted as substitutes rather than supports for user reasoning. Summary-trace preserved task performance at the no-trace baseline and led to lower overestimation levels, but no condition produced well-calibrated self-evaluation, providing no support for H1. Trace exposure inflated bias and left sensitivity unchanged, indicating that reasoning content does not feed the monitoring loop, which aligns with the notion that explanations are rarely used in HAI decision-making [59]. H2 and H5 were also not supported, with null effects for task load and usability. H3, which predicted higher trust and confidence under trace exposure, was the only hypothesis the data partially supported: both trace conditions raised TSQ trust and UEQ hedonic appeal relative to the Answer-only baseline, with no reliable separation between Full-trace and Summary-trace.

Across hypotheses, trace exposure reliably shifted the surface of the interaction – trust and hedonic appeal – but did not shift effort, usability, performance, or calibration in the predicted directions. The unifying account is that users engaged with the trace superficially, as a fluent artifact rather than as an object of evaluation. Where the preregistration anticipated comparison between the user’s and the model’s reasoning, participants instead treated the trace as a coherent rendering whose internal plausibility substituted for their own monitoring.

Across the full sample, every CAIMS subscale positively predicted overestimation and calibration error, replicating an inversion reported by Fernandes et al. [23]. Self-report instruments of (metacognitive-) AI literacy appear to track something other than the behavioral competencies they are meant to index: participants who report planning, monitoring, and evaluating their reasoning with AI are not the participants whose confidence tracks correctness. Our study adds to Fernandes et al. [23] that self-report measures of AI literacy seem increasingly unsuited to certifying metacognitive support in HAI interfaces.

6.1 Complementarity, Overreliance, and Metacognitive support Reasoning traces are often marketed as transparency mechanisms assumed to help users understand why the model produced an answer and improve their judgment [5, 6, 39, 40]. Our findings complicate this assumption. Full traces made the model’s answer appear more inspectable and internally coherent, yet this additional information did not translate into better cognitive performance or calibration in the LSAT questions. Instead, it may have created an illusion of understanding [25] due to the model’s verbosity and apparent fluency, leading participants to trust the model’s answer, consistent with the trust pattern predicted under H3. These findings suggest that providing users with more information about how an LLM arrived at its answer does not necessarily support better reasoning, and may, in some cases, undermine it. Taken together, they reposition reasoning traces along three connected positions in HAI: complementarity, overreliance, and metacognitive support.

Preprint 2026-05-26 02:19. Page 20 of 1–27.

Unpublished working draft.

Not for distribution.