@Grok Is This True? LLM-Powered Fact-Checking on Social Media

Source: Thomas Renault, Mohsen Mosleh, David G. Rand; PsyArXiv preprint (v2), not peer-reviewed · 2026-01-21

Large language models (LLMs) are increasingly embedded directly into social media platforms, enabling users to request real-time fact-checks of online content. Using an exhaustive dataset of 1,671,841 English-language fact-checking requests made to Grok and Perplexity on X between February and September 2025, we provide the first large-scale empirical analysis of how LLM-based fact-checking operates in the wild. Fact-checking requests comprise 7.6% of all interactions with the LLM bots, and focus primarily on politics, economics, and current events. We document clear partisan asymmetries in usage. Users requesting fact-checks from Grok are much more likely to be Republican than Democratic, while the opposite is true for fact-check requests from Perplexity – indicating emerging polarization in attitudes toward specific AI models. At the same time, both Democrats and Republicans are more likely to request fact-checks on posts authored by Republicans, and - consistent with prior work using professional fact-checkers and crowd judgments - posts from Republican-leaning accounts are more likely to be rated as inaccurate by both LLMs. Across posts rated by both LLM bots, evaluations from Grok and Perplexity agree 52.6% of the time and strongly disagree (one party rates a claim as true and the other as false) 13.6% of the time. For a sample of 100 fact-checked posts, 54.5% of Grok bot ratings and 57.7% of Perplexity bot ratings agreed with ratings of human fact-checkers, which is significantly lower than the inter-fact-checker agreement rate of 64.0%; but API-access versions of Grok had higher agreement with fact-checkers and did not significantly differ from inter-fact-checker agreement. Finally, in a preregistered survey experiment with 1,592 U.S. participants, exposure to LLM fact-checks meaningfully shifts belief accuracy, with effect sizes comparable to those observed in studies of professional fact-checking. However, responses to Grok fact-checks are polarized by partisanship when model identity is disclosed, whereas responses to Perplexity are not.

Introduction. There is widespread public concern about the spread of inaccurate information on social media. Over the past decade, technology companies have taken a variety of approaches to identifying inaccurate posts, including training machine learning classifiers, partnering with professional fact-checkers (although large platforms such as X and Meta have stepped back from this practice in recent years), implementing systems where users can label content they believe is inaccurate such as "Community Notes", and attempting to increase users’ digital literacy. Users themselves, however, have had little recourse when attempting to evaluate the accuracy of posts, beyond disrupting their browsing experience to consult fact-checking websites or searching for relevant information themselves (which does not always increase their ability to distinguish truth from falsehood).

With the advent of large language models (LLMs), however, social media users have a new tool at their disposal. For example, both X’s Grok and Perplexity have bot accounts on X that will provide evaluations of a post if a user replies to that post and tags the bot in their reply. How is this new approach to fact-checking being used? In this paper, we shed light on this question by collecting all publicly available tweets mentioning @Grok or @AskPerplexity from February 2025 through the end of September 2025, from which we identify 1,671,841 English-language fact-checking requests. We then reconstruct three-tweet sequences by matching each fact-check request to the original tweet it targeted and to the LLM’s subsequent veracity assessment (see Figure 1). Using this unique exhaustive dataset, we ask how users on X employ LLMs for fact-checking, what types of questions are posed, who initiates these fact-checks and who is fact-checked, and how the LLMs respond. We also examine the level of agreement between different LLMs, between LLMs and evaluations from the crowd-based Community Notes program, and between LLMs and professional fact-checkers. Last, we conduct a survey experiment to examine how LLM fact-checks affect beliefs and shed additional light on the polarization of LLM attitudes. Together, these data provide one of the first glimpses into this rapidly emerging approach to evaluating claims made on social media.

In prior work on the Community Notes system, users tended to fact-check out-group members more than in-group members (i.e., Republicans were more likely to fact-check Democrats, and vice versa). We ask if the same is true of LLM fact-checks (Figure 4B). On the contrary, we find that Republican users are targeted for fact-checking substantially more frequently than Democrats, be it across all requests (97.3% more, α=0.68, p<0.001), requests from Democrats (83% α = 0.60, p<0.001), or requests from Republicans (107.5% more, α=0.73, p<0.001). While the high rate of Republican-Republican intra-group checking may partly reflect network structures – users predominantly follow others with similar political leanings, increasing the likelihood of intra-group interactions – it also suggests that many fact-checking requests stem from genuine informational uncertainty rather than strategic or performative motives, as has sometimes been suggested in the context of Community Notes. Community Notes, when rated as helpful, are attached directly to the original tweet and visible to all users, giving them broad visibility and influence beyond those who proposed or voted on the note. In contrast, fact-check requests and LLM responses appear only as standard reply tweets, meaning their reach is generally limited to the user who initiated the request and to others who directly view the reply thread (the average LLM response receives 546 views, compared with more than 2 million views for the initial claims targeted by fact-check requests).

Method. Participants: To assess the impact of LLM fact-checks on users’ beliefs, we conducted an online survey experiment in December 2025, recruiting a U.S. sample composed of half Democrats and half Republicans from CloudConnect. A total of 1,683 participants began the survey. Of these, 18 dropped out prior to the experimental portion of the survey and 73 dropped out during the survey. The rate of post-treatment dropout did not differ significantly across conditions (χ2 = 1.84, p = 0.61). The final sample therefore consisted of N = 1,592 participants (mean age = 42.9 years, 57.6% female, 49.2% Republican). The pre-registered target sample size was 800 Democrats and 800 Republicans. Because of difficulty recruiting sufficient Republicans, we included 200 Republicans recruited for pilot studies conducted in the two weeks prior to the pre-registration and main experiment. These participants are included in the totals reported above. Excluding these participants does not meaningfully alter our conclusions.

Materials: To compare the relative effectiveness of fact-checks written by Grok and Perplexity, we selected 100 tweets from a pool of 9,890 tweets that had received fact-checks from both models. We restricted attention to tweets for which the two models gave the same accuracy rating, where the rating was either greater than or equal to 60 (“True”) or less than or equal to 40 (“False”). Tweets for which the models were uncertain were excluded. We further excluded tweets containing videos, tweets displaying a Community Note, and tweets about topics unlikely to be familiar to U.S. participants, such as events in India or Nigeria.

Discussion. Our findings demonstrate the growing demand for large language model fact-checking and real-time verification of information on social media, with a particular emphasis on politics, economics, and current events. This trend likely reflects not only a widespread desire for immediate clarification during periods of uncertainty, but also a broader transformation in how users engage with automated tools in public discourse. In contrast to earlier research that focused on controlled experimental settings, our study offers a rare empirical view of LLMs as fact-checkers in dynamic real-world environments.

Compared to crowd-sourced systems like Community Notes, LLMs deliver faster and broader coverage, especially on contentious or rapidly evolving issues where consensus is difficult to achieve. However, this speed comes at the expense of transparency: unlike Community Notes, LLMs often do not disclose their sources, logic, or uncertainty levels. This opacity can undermine user trust and complicate systematic evaluations of model performance. Moreover, whereas Community Notes are directly attached to the original claims and have been shown to substantially curb engagement with false content, at least as of now, LLM fact-checks appear only as replies and attract far less visibility and engagement. On average, LLM fact-checks receive about 546 views (2.8 likes, 0.2 retweet, 0.3 reply) compared with 2.4 million views (23,069 likes, 3,155 retweets, 1,315 replies) for the initial claims, suggesting that their impact on the diffusion of misinformation is likely to be limited, although further research is needed to rigorously assess their causal effect on tweet diffusion and deletion.

In terms of the contents of the LLM fact-checks, we observe strong correlations between the outputs of different LLMs, suggesting a reasonable degree of internal consistency. When compared to the classification of professional fact-checkers, we find less agreement between models and fact-checkers than agreement between fact-checkers. Nonetheless, the LLM evaluations were much more likely to agree with the fact-checkers than to strongly disagree (2.8x more likely for Grok, 4.1x more likely for Perplexity). Furthermore, API models – even Grok-4 – showed higher levels of agreement with fact-checkers that were on par with inter-fact-checker-agreement. Together, these observations suggest that the LLM ratings contain a meaningful level of signal, and that LLM fact-checking – particularly as models get more powerful and access to real-time information improves – may hold promise for increasing belief accuracy at scale.

We also observe a variety of striking patterns related to partisanship. First, we document clear polarization in attitudes towards large language models. We find that users requesting fact-checks from Grok were much more likely to lean Republican than Democrat, whereas users requesting fact-checks from Perplexity were somewhat more likely to lean Democrat than Republican. Our survey experiment provides further evidence of polarization, particularly in the context of Grok. Democrats were markedly more aware of, but less trusting of, Grok compared to Republicans. Furthermore, although participants across all subgroups meaningfully updated their beliefs in the direction of the LLM fact-checks, revealing that a fact-check was generated by Grok reduced updating among Democrats while increasing updating among Republicans. These findings give a glimpse of a possible future in which LLMs becomes as polarized as legacy media and social media are today, with companies designing LLMs to appeal to right- or left-leaning audiences by reinforcing each group’s preconceptions.

Conclusion. In conclusion, our results highlight the urgency of developing clear standards and safeguards for the use of LLMs in content verification. As AI systems become increasingly embedded in the architecture of online communication, questions surrounding who is fact-checked, by what mechanisms, and with what consequences will become increasingly salient. Given the scale and speed at which LLMs are now deployed, the absence of formal oversight or audit mechanisms poses a growing risk to the integrity of public information ecosystems.