Do AI fact-checkers on social media match human accuracy?

Renault, Mosleh and Rand (PsyArXiv preprint, v2, not peer-reviewed, 2026-01-21) analyze an "exhaustive dataset of 1,671,841 English-language fact-checking requests made to Grok and Perplexity on X between February and September 2025." For a sample of 100 posts fact-checked by both bots, they report "54.5% of Grok bot ratings and 57.7% of Perplexity bot ratings agreed with ratings of human fact-checkers, which is significantly lower than the inter-fact-checker agreement rate of 64.0%." Yet "API-access versions of Grok had higher agreement with fact-checkers and did not significantly differ from inter-fact-checker agreement."

The authors locate the shortfall in the deployment layer rather than a shared model ceiling: the live X bots, running against the same underlying model family, score below fact-checker consensus, while the API-access version does not. They note the disagreement is not random noise — "the LLM evaluations were much more likely to agree with the fact-checkers than to strongly disagree (2.8x more likely for Grok, 4.1x more likely for Perplexity)" — and read the API result as evidence that "LLM fact-checking... may hold promise for increasing belief accuracy at scale" as models and real-time information access improve.

This splits what Does AI fact-checking actually help people spot misinformation? found in a lab RCT — that an imperfectly accurate AI fact-checker can be net-harmful because mislabeling skews belief more than correct labels help — into a real-world setting: this paper documents the actual accuracy gap that would determine whether that RCT's harm condition applies to the live X bots as deployed. It also complicates Do AI assistants reliably answer questions about news?, which treats AI assistants as a largely interchangeable category of error-prone systems: the API-versus-consumer gap here suggests the surface a model ships through, not the model itself, may be doing part of the damage.

The excerpt is a preprint, not yet peer-reviewed, and its account of why the API version does better is inferential — the authors do not isolate whether real-time data access, prompting, or the consumer product's reply-length constraints drive the gap, nor whether API-level agreement would survive deployment at the bots' current visibility and speed. The implication the evidence does support is narrower: evaluating an LLM's fact-checking accuracy by testing only its consumer chat surface, as much misinformation research does, may understate what the underlying model is capable of.