Human fact-checkers only agree with each other about two-thirds of the time — so what's a fair bar for an AI fact-checker?

This explores how often human fact-checkers disagree with each other, and why that number is the right yardstick for judging AI fact-checkers. The corpus has one hard figure. In an analysis of 1.67 million fact-checks on X, human fact-checkers agreed with each other only about 64% of the time Do AI fact-checkers on social media match human accuracy?. So roughly one verdict in three is contested even before any AI is involved. That changes how to read the AI results. The consumer Grok and Perplexity bots agreed with humans 54–58% of the time. That looks bad if you assume 100% is the target, but it's only a few points below what humans manage among themselves. The more surprising finding is that the API version of Grok matched human consensus. The gap seems to come from how the bots are deployed, not from what the model can do.

If a third of the cases are contested, should disagreement count as failure at all? Research on AI systems that label text points the other way. When several LLM agents argued at length and stayed unresolved over how to label something, their accuracy was *higher*, and how often they disagreed was a reliable sign of quality Does disagreement between AI coders signal better accuracy?. Disagreement can mean the case is genuinely hard, and those are the cases where careful, interpretive work matters most. A related line of work adds a dedicated 'agreement detector' to AI debates. Its job is to tell real consensus apart from premature giving-in Can AI systems detect when they've genuinely reached agreement?. Human fact-checking teams rarely measure that distinction.

The catch is that readers don't see disagreement as a percentage. They see a single label. A randomized experiment found that AI fact-checking didn't improve people's overall ability to tell true from false, and the errors weren't symmetric. When the AI wrongly labeled a true headline false, people believed it less. When the AI sounded unsure about a false headline, people believed it more Does AI fact-checking actually help people spot misinformation?. On top of that, simply naming Grok as the source moved Democrats and Republicans in opposite directions on the same verdict Does naming Grok as fact-checker change how people update beliefs?. So two AI tools that score the same on agreement can land very differently with the public.

The takeaway you may not have expected: 'How often does the AI agree with humans?' is a weaker question than 'Which cases do humans themselves split on, and what does the AI do with those?' The corpus has only the single 64% baseline. It doesn't break down *which kinds* of claims make human fact-checkers disagree, so that part of the question is still open here.