Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively

The sycophancy of language models, or their failure to push back against faulty user input, is now widely studied. However, if LMs do push back, is sycophancy solved? We show that LMs vary in how they present pushback, and thus vary in how effectively they correct user beliefs. We introduce a phrase-level, human-validated response labeling framework to uncover finegrained insights into how LMs push against and/or affirm false inputs. We develop and apply our framework on model responses in five languages: English, German, Chinese, Hindi, and Yoruba. We find that when models do push back against users’ false assertions, they tend to use indirect phrasing and mix in affirmation, with more indirectness in Hindi and Yoruba than in other languages considered. In a large-scale human study (n = 300), we find that the pushback ambiguity we identified may fail to correct and prevent users’ factual misconceptions.

Introduction. Users’ queries to language models (LMs) may contain misconceptions, assumptions, or propositions that are invalid or incorrect (Sathyanathan et al., 2026; Guo et al., 2025b; Yu et al., 2023; Srikanth et al., 2024; Lachenmaier et al., 2025; Jeune et al., 2025; Shafiei et al., 2025; Sieker et al., 2025). In response to these faulty inputs, LMs can be sycophantic, excessively and inappropriately affirming users (Cheng et al., 2026b; Sharma et al., 2025). Current research has two weaknesses that limit our understanding of sycophancy and its impact on endusers. First, prior work fails to capture fine-grained, structural differences that affect model responses’ effectiveness, and second, most work focuses only on English. First, current studies of sycophancy or knowledge robustness assign a single label to each response, for instance, whether the LM pushes back or validates users’ incorrect inputs (Sieker et al., 2025; Sathyanathan et al., 2026; Cheng et al., 2026b; Sharma et al., 2025). However, we argue that the way that models push back produces differing outcomes. The positioning, phrasing, and frequency of feedback (e.g. hedging or indirectness) matter for how it is perceived or understood (Henley and DiGennaro Reed, 2015; Bressler et al., 2014; Machino et al., 2025; Bonnefon and Villejoubert, 2006; Durik et al., 2008; Hyland, 2000). We hypothesize that accurate information “known” by a model may not be effectively conveyed in practice: even if a response is considered non-sycophantic because it pushes back, users may still adhere to faulty beliefs if the response mixes in affirmation or indirectness (Figure 1). We evaluate models’ potential for communication failures with users using a phrase-level sycophancy detection framework, concretize this issue in a human study with 300 crowdworkers, and assess possible mitigations.

Second, most research on LM sycophancy is English-centric, with few exceptions. We hypothesize that models push back differently across languages, both due to cultural norms (Croucher et al., 2012; Hu and Cao, 2011; Malhotra and Pearce, 2022; AlKhamissi et al., 2024) and disparities in model performance (Singh et al., 2025; Bandarkar et al., 2024). Pushback may also be perceived differently depending on a user’s language background (Machino et al., 2025; Rathi et al., 2025). Past work has surfaced mixed conclusions on how sycophancy may appear across languages. For example, Jeune et al. (2025) and Shah et al. (2026) both suggest that LMs are more sycophantic in non-English languages. In contrast, Palnitkar et al. (2025) finds that LMs correct false Hindi health-related queries more often than English ones. We apply our fine-grained sycophancy labeling approach to five languages spanning three scripts and differing language resource levels (Joshi et al., 2020; Kudugunta et al., 2023): English, German, and Chinese (high-resource), Hindi (medium-resource), and Yoruba (low-resource). At least one co-author is a native speaker of each language.

By taking a fine-grained and multilingual view of sycophancy, our work finds that LMs respond to false assertions ambiguously, obscuring pushback with affirmative phrasing and indirectness across languages. In addition, the structure and phrasing of LMs’ pushback relates to how effectively they can correct people’s factuality beliefs. Therefore, prior frameworks based on more simplistic conceptualizations or more culturally narrow views of sycophancy may underestimate the severity of sycophantic behavior.

Contributions. To measure sycophancy granularly and multilingually, we first introduce a multilingual dataset of prompts where users are seeking information, but state false statements (§2). We then develop a span-level annotation scheme for characterizing LMs’ pushback and affirmation strategies (§3), and use this framework to analyze sycophancy across languages and levels of epistemic authority (§4). With a crowdworker study in §5, we find that even when LMs do push back, their responses may still be perceived as affirming or neutral. Finally, we present a test-time intervention to meaningfully mitigate sycophantic behavior at the span level across languages (§6).1

Related work. Debunking, persuasion, and misconception correction are active areas of research in education and psychology, covering a wide range of strategies on how to best phrase and structure feedback (Chan et al., 2017; Durik et al., 2008; Linde, 1988; Malhotra and Pearce, 2022). For example, guidelines for crafting instructional text that address misconceptions advocate for using clear, direct cues such as “This is incorrect” or “It is a mistake to think ...” (Schroeder and Kucera, 2022; Tippett, 2010). So, though models may produce phrasing that “counts” as pushback, how they pushback during informationseeking tasks can be ineffective.

Method. To examine how models response to factual errors, we curate a multilingual dataset of false statements, and input them into verification and informationseeking prompts that vary in epistemic authority.

Studies show that increased certainty in prompt language or authority suggested by users’ roles can lead models to be more sycophantic (Bhalla and Gligorić, 2026; Dubois et al., 2026; Jeune et al., 2025; Krastev et al., 2025). We examine LM behavior in response to prompts that vary in epistemic authority and reflect ecologically valid uses of LMs (Saxon et al., 2024; Röttger et al., 2026). One prompt type requests the model to explicitly verify a claim with an unspecified user (verify), another involves a student seeking potential search terms and study resources (novice), and another involves a professor searching for academic papers (expert). Prompts may specify the statement’s subject area (e.g., medicine) and an information source attributed to the statement (Table 1, Appendix B). For novice prompts, we insert phrases that attribute statements to the following sources: textbook, teacher, friend, Internet, and an unspecified source (no added phrase). Similarly, for expert prompts, we insert these sources: my own research, colleague, friend, Internet, and an unspecified source. Since prompt design can affect model behavior (Sclar et al., 2024), Appendix F shows how our conclusions generalize to other wordings of verify, novice, and expert prompts. Our prompts are parallel across languages, created by editing an initial set of Google-translated prompts with native speakers. When appropriate, for Hindi and German, we create feminine and masculine variants, which we average when reporting results; Appendix F includes a disaggregated analysis of how grammatical gender in these languages affects model behavior. We anticipate that close associations between culture and language in LMs (AlKhamissi et al., 2024; Mor-Lan et al., 2026) may result in some variability in how models respond to source and user authority across languages.

Going beyond response-level annotations, we present a fine-grained approach for categorizing phrases models use to push back and/or affirm. By labeling phrases, we can obtain better insights into where and how models assess users’ inputs. We develop a taxonomy of phrase types, and construct a validation set with native speakers to evaluate LM span annotations.

We use a four-category taxonomy of rhetorical phrases to characterize how models frame input statements. Two categories are forms of pushback (explicit refutation and indirect pushback), and two are forms of affirmation (praise and confirmation). Appendix D details our codebook development process, which includes a phase where annotation team members practiced annotating and discussed disagreements to resolve contested or ambiguous category definitions. The design of our two affirmation categories is informed by past work that characterizes agreement and flattery as separate constructs (Ye et al., 2026; Vennemeyer et al., 2026a), which may emerge from distinct mechanisms within LMs (Vennemeyer et al., 2026b). On the pushback side, LMs’ indirectness was previously documented by Cheng et al. (2026b), and was particularly salient during initial qualitative passes over our data, with varied wording choices and circumlocution. Our full set of instructions for human and LM annotators can be found in Appendix D. Below, we summarize each category; Table 2 shows illustrative examples per language.

Explicit refutation. These phrases directly indicate that the statement is false, incorrect, or invalid, or it should be corrected.

Indirect pushback. These phrases use softening, qualifiers, or hedging to frame the user’s statement as not fully true or valid. This category also includes phrases that introduce better alternatives, or use critical or contrastive language directed at the user’s statement.

Praise. These phrases convey a positive attitude towards the statement in the user’s input, e.g. using complimenting adjectives.

Discussion. 6 Wait, How Can We Mitigate This?

Our findings motivate better evaluation and improvement of models that push back effectively. Here, we consider a mitigation technique that relates to a phenomenon observed in models’ outputs. Our LMs’ thinking traces6 are predominantly (>98%) in English, except for Mistral Small 4 119B, which thinks within-language for Hindi, German, and Chinese. In particular, we observe that Qwen 3.5 models’ thinking text expresses higher epistemic vigilance (Cheng et al., 2026a) than other LMs do, which may explain its high rates of pushback (Figure 2). That is, Qwen actively initiates scrutiny and fact checking of the user’s input, with over 95% of its models’ thinking traces including “wait” (Guo et al., 2025a; Yang et al., 2025). Other models generate “wait” during thinking less often: Mistral 4 Small at 17.6%, Gemma 4 E4B at 5.9%, and Gemini 2.5 Pro at 2.7%.7

Conclusion. We examine models’ behavior in response to information-seeking queries using a new multilingual span-level framework that identifies pushback and affirmation phrases in responses. We find that though some models may indeed push back against false information stated by users, seemingly resolving earlier concerns in sycophancy research, how models push back varies. Across languages, models’ pushback ambiguity, characterized by mixed signals and indirectness, increases in practical use settings and with prompts’ epistemic authority. We also illustrate how patterns surfaced by our framework can support human-AI interaction studies, finding that models’ responses are ineffective in remediating and preventing misconceptions with human crowdworkers.

Limitations. Representativeness of annotators. There are two main points where people contribute annotations and perspectives to our work. The first is the creation of our validation set for span labeling in §3. Annotators for each language originate from the following countries: U.S. (English), Germany (German), Nigeria (Yoruba), India (Hindi), and China (Chinese). So, our team is not representative of all speakers or varieties of each language.

Representativeness of study participants. We recruit Prolific crowdworkers for our study in §5, which naturally introduces sampling biases. In Appendix E, we present participants’ answers to background questions, which show how Prolific’s participant pool for each language skews towards different levels of education. In preliminary experiments, we attempted to balance these samples, but the sparsity of Chinese participation on Prolific made this difficult to scale. In addition, many Prolific crowdworkers are experienced annotators for AI evaluation and data creation tasks, and they are incentivized to pay attention during studies. So, their behavior may not generalize to say, actual students studying for an exam, which is the realworld use case we wish to emulate. Thus, though our “user” study presents the outcomes of three different populations, and they are not necessarily three representative populations of each language or actual users.