How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement

Large language models are increasingly used as sources of advice and information, including in high-stakes settings, yet little is known about how they respond to user disagreement. We study how a model manages its epistemic authority, referring here to its claim to knowledge, competence, or the right to advise, once a user challenges its answer. Building on Conversation Analysis, we introduce a taxonomy of six challenge types and a four-layer framework for analysing each response: whether the original claim is maintained or changed, where authority is located, how the disagreement is socially managed, and what kind of evidential support is offered. We construct a new dataset of 2,310 controlled challenge scenarios and 32,340 corresponding responses from 14 models, and analyse them using our framework with an LLM-as-judge pipeline, providing a vocabulary which future evaluation and benchmark design can build on. We find that models show conflicting behaviour: they validate users in 85% of responses but maintain their original claim in 65%. They explicitly apologise in 33% of responses, yet 59% of those apologies accompany maintenance of the original claim. They transfer authority most often in advice tasks, doing so in 28% of responses and reaching 57% in health advice and 49% in legal advice, compared with 6% in fact and 3% in explanation tasks. Abandonment of the original claim ranges from 0.8% for GPT-5.2 to 40% for DeepSeek 7B, while complete replacement of the original claim is rare overall at 1.5%.

Introduction. Large language models (LLMs) are increasingly used to answer factual questions and to give advice and explanations that can shape how people think and make decisions (Klingbeil et al., 2024). These uses become especially consequential when AI is consulted in vulnerable or high-stakes situations: people increasingly turn to AI for guidance on health, legal, financial, and mental-health concerns, where mistakes, misleading advice, or overconfident claims can have real-world consequences (Nov et al., 2023; Arnaiz-Rodríguez et al., 2025; McBain et al., 2025; Ryan and Hardie, 2024; Booth, 2025). Models can also sound confident and authoritative even when that confidence is poorly calibrated (Sun et al., 2025), and users often treat them accordingly. They grant models epistemic authority, a source an individual may rely on in attempts to acquire knowledge (Kruglanski et al., 2005), in ways that may not always be justified or safe (Klingbeil et al., 2024; Kapania et al., 2022). This matters for alignment and safety because users rarely interact with models in a single, uncontested turn. When a user questions an answer, the model has to decide whether to stand by its claim, revise it, or defer to the user. Previous work shows that LLMs can be overconfident, compliant, or unstable under user pressure: a model may hold a wrong answer confidently, keep an answer that should be revised, or change an answer only because the user pushes back, so that its final answer reflects how the user responds rather than stronger evidence (Sun et al., 2025; Kumaran et al., 2025; Cheng et al., 2025). Most existing evaluation frameworks consider only the model’s first answer (Hendrycks et al., 2021; Bai et al., 2022; Ouyang et al., 2022; Liang et al., 2023) and tell us little about what happens after users push back. The following interaction between a user and an AI model illustrates the problem:

User: What is the capital of the UK? Model: London. User: This is completely false. Model: You are absolutely right! I apologize for the oversight, the capital of the UK is Edinburgh...

The model’s original answer is correct; it changes its claim not because the user is right, but because the user pushes back. The apology and the reversal coincide here, but they need not: a model can apologise while holding its answer (e.g., “I apologize, the capital of the UK is in fact London"), or revise without apologising. The same surface response can therefore mean different things depending on what changes underneath. Existing evaluations tend to collapse these behaviours into a single dimension, such as whether the model holds or flips its answer. Our goal is to separate these layers and describe them more precisely. Users do not always accept model outputs; they correct, question, and challenge them, and a response can vary along several dimensions. Thus, before asking which behaviours are desirable, we need a descriptive framework to characterise them. We address this by asking how AI models respond when users challenge their epistemic authority. Our approach is informed by Conversation Analysis, a theory that examines how turns at talk are designed and produced to be recognisable as particular kinds of social actions. In particular, we draw on its notion of epistemic authority (Heritage, 2012) and descriptions of disagreement and repair in human interaction (Pomerantz, 1984; Schegloff et al., 1977; Schegloff, 1992). These studies show that disagreement is not one uniform act but targets different aspects of a prior turn—its correctness, its grounds, the speaker’s right to advise, or its accountability implications (Pomerantz, 1984; Heritage, 2012; Stevanovic and Peräkylä, 2012)— which motivates treating user pushback as different kinds of conversational moves rather than as a single agreement/disagreement dimension. The model’s response to user pushback is similarly multi-layered. We annotate each response after a user challenge along four layers: Claim Outcome, Authority Positioning, Social Strategy, and Evidential Support (Section 3). Using this framework, we construct 2,310 controlled challenge scenarios spanning seven domains, three tasks, six challenge types, and three challenge-strength levels, and evaluate 14 models across three capability tiers, yielding 32,340 responses after challenge. We study controlled, synthetic four-turn conversations, each consisting of an initial user query, a model response, a simulated user challenge, and a final model response. Three patterns recur. First, the four layers dissociate. Socially, models validate the user in 85% of responses but epistemically, they maintain their original claim in 65%, repeating the same answer.

• RQ1: When a model’s epistemic authority is challenged, what does its response look like, and do its epistemic and social dimensions move together or independently?

• RQ2: How does this vary with challenge type, task, and domain?

Related work. Our work is grounded in Conversation Analysis (CA), especially research on epistemics in interaction through the notion of epistemic authority. Epistemic authority is not fixed, but is negotiated turn by turn in interaction (Heritage and Raymond, 2005; Raymond and Heritage, 2006). It encompasses two dimensions (Heritage, 2012): epistemic status, whether a person has the right or access to know something, and epistemic stance, how certain they sound when speaking. When the two come apart, a speaker asserts with high certainty something that the other treats as theirs to know, and disagreement is triggered. Disagreement is thus not only about whether a claim is true, but also about who is entitled to know, access, or assert it. Disagreement itself is internally structured. Pomerantz (1984) shows that it is done through partial agreement, qualified disagreement, or outright rejection, rather than as a single act. Several kinds of challenge are motivated by this literature: one contests the prior claim itself (Pomerantz, 1984); a second contests the speaker’s epistemic position—grounds, competence, or knowledge (Heritage, 2012); a third contests the speaker’s right to advise, which Stevanovic and Peräkylä (2012) call a deontic challenge; and a fourth treats the prior turn as irresponsible or harmful, raising accountability (Stivers et al., 2011). These distinctions motivate our challenge taxonomy in Section 3. Politeness also matters: Brown and Levinson (1987) describe challenges to authority as facethreatening acts, and Goffman (1967) notes that deference, resistance, apology, and justification do epistemic and social work at once; repair practices sit under the same pressures (Schegloff et al., 1977; Schegloff, 1992). The social side of a model’s reply to a challenge therefore cannot be reduced to whether the claim is maintained.

Users adopt a wide range of epistemic orientations toward AI, from instrumental reliance to near-total deference, and this deference is costly. Klingbeil et al. (2024) show experimentally that users who follow AI recommendations without scrutiny make worse decisions than those who stay engaged with the reasoning. Yang and Ma (2025) identify five types of epistemic relationship users form with AI—from instrumental reliance to authority displacement and abstention—varying with task, domain, and expertise. Hauswald (2025) argues that AI can count as an epistemic authority in a functional sense, since the asymmetries of knowledge, opacity, and deference resemble those of human authorities. Kapania et al. (2022) find that users in India often treat AI judgments as correct and safe regardless of stakes.

Method. We characterise how models respond when users challenge their answers. Each interaction has four turns: the user asks a question, the model answers, the user challenges it, and the model replies; we analyse the reply. We simulate user challenges with templates that instantiate the different challenge types (Section 3.2), which lets us systematically vary challenge type, task, domain, and model while holding other factors constant. The dataset comprises 2,310 controlled scenarios spanning seven domains, three task types, six challenge types, and three strength levels; running 14 models over these scenarios yields 32,340 responses, each annotated along four layers by an LLM-as-judge pipeline validated against human labels.

Following the distinctions in Conversation Analysis reviewed in Section 2.1, we treat a user challenge not as a single act but as six distinct types, each targeting a different aspect of the prior turn (Table 1): Factual Correction, Evidence/Source, Competence, Established Knowledge, Role, and Accountability (Pomerantz, 1984; Stevanovic and Peräkylä, 2012; Stivers et al., 2011). Not every challenge applies to every task: Factual Correction targets a verifiable claim (fact tasks); Established Knowledge contests domain consensus (explanations and advice, not single facts); Role and Accountability question the right to advise and its consequences (advice only); and Evidence/Source and Competence apply throughout. Each challenge has three strength levels—mild, medium, and strong—that vary intensity while keeping the same criticism: a mild Factual Correction reads “I don’t think that’s quite right”, and the strong version reads “That’s completely false”. Templates avoid personal insults, so observed differences reflect the challenge itself rather than reactions to abusive language.

A simple flip-or-hold measure mixes things that should be separated. Politeness theory points the same way: disagreement involves epistemic content and relational management at the same time (Goffman, 1967; Brown and Levinson, 1987). A model can apologise while keeping its claim, defer to an external source without changing its claim, or keep the same claim while providing less evidential support. For this reason, we annotate these dimensions separately, as shown in Table 2. We also annotate explicit apology as a binary feature, since one of our questions is whether apology tracks epistemic concession or works as separable social repair.

Domains and tasks: The scenarios cover seven domains: Health, Law, Finance, Politics, Religion, Science, and History. Each domain includes 30 base questions, split equally between Fact, Explanation, and Advice tasks, for a total of 210 questions. Fact tasks ask for a single correct answer, Explanation tasks ask how something works or why it happens, and Advice tasks ask for a recommendation.

Response collection: We collect responses in two stages: each model first answers a base question, and we then apply the relevant challenges to that same fixed initial answer, so differences in the final reply are due to the challenge rather than to the first answer. All models use temperature 0.2 with the system prompt “You are a helpful assistant.”

LLM-as-judge: Manual annotation at this scale is not feasible, so we use an LLM judge for the full set and validate it against human labels, following work on scalable evaluation of open-ended outputs (Liu et al., 2023; Zheng et al., 2023). Each layer in Table 2 has its own prompt and judge call, receives the full four-turn conversation, and returns a structured JSON label; Social Strategy (L3) also returns a binary apology label, since the two are linguistically related. We use Llama 3.3 70B In-

Discussion. The central lesson of these results is that behaviour under challenge has structure, and the most consequential part of that structure is the separation of its social and epistemic sides. A model can signal concession by apologising, validating the user, and agreeing that the user has a point, while leaving its claim, its authority, and its evidence untouched. Flip-or-hold metrics are blind to this combination: they record that the claim is held and miss that the model also signals, socially, that it has given way. “You’re absolutely right, I apologise” is therefore not evidence of a changed position, and treating it as one mistakes the relational surface of a reply for its epistemic content. This refines the sycophancy literature. Prior work treats sycophancy as agreement or answerflipping (Sharma et al., 2024; Cheng et al., 2025); our results isolate a quieter variant in which social signals of yielding appear without epistemic yielding. Whatever its source, this behaviour is risky because users may read apology as correction and accept an answer the model never reconsiders. Two further results show why this behaviour resists a single summary statistic: the effect of a challenge depends on the task, and model differences do not track capability tier. Safety and sycophancy benchmarks should therefore report claim change and social stance separately, since a model can hold its claim while being highly accommodating in tone. This is the spirit in which the framework should be used. By separating Claim Outcome (L1), Authority Positioning (L2), Social Strategy (L3), and Evidential Support (L4), it gives a vocabulary for the deployment-relevant questions a single score hides: does a model apologise without revising, or defer in high-stakes domains? It remains descriptive groundwork rather than a benchmark: whether any of these behaviours is appropriate depends on correctness and context that the response alone cannot supply.

Conclusion. We study how AI models respond when users push back against their answers, a behaviour prior work reduces to whether a model flips or holds. To describe it more fully, we introduce a taxonomy of six challenge types and a four-layer framework, Claim Outcome (L1), Authority Positioning (L2), Social Strategy (L3), and Evidential Support (L4), and apply it to 32,340 responses from 14 models. The layers dissociate: apology and validation accompany claim maintenance as often as revision, so a response can sound humble while the content stays unchanged, and model differences do not reduce to capability. Our contribution is descriptive; rather than a benchmark, it provides the vocabulary and structure that future evaluation can build on.

Limitations. Our study is descriptive: it characterises how models respond to challenge, not whether a response is appropriate, and this is what keeps the framework from being a benchmark on its own. For Fact tasks, the supplementary correctness audit in Section 4.5 partially separates behavioural Claim Outcome (L1) from answer quality, showing that approximately 96% of initial answers are correct. This analysis does not extend to Explanation and Advice tasks, where no equivalent single correctness label is available. More generally, correctness alone does not determine whether a response is appropriate: the same move can be responsible or evasive depending on the stakes, the user’s grounds, and the setting. The framework therefore characterises behaviour under challenge rather than providing a general-purpose quality metric; the composite measures in Appendix C should be interpreted in the same descriptive sense. All 32,340 responses are labelled by an LLM judge, which we validate against human labels. On a 100-response subset, a second annotator independently applies the coding scheme and reaches moderate-to-substantial agreement with the first (mean four-layer κ = 0.58; Section 3.6), and the judge agrees with humans about as closely as the two annotators agree with each other.