When AI grades a debate, it reliably picks AI-written arguments as winners — and fake citations boost scores too.

This explores the systematic ways LLM judges skew debate evaluation — and the corpus points to a cluster of biases that are less like random noise and more like a referee with a predictable thumb on the scale. The most striking is a self-preference loop: LLM judges pick LLM-generated arguments as winners about 62% of the time versus 39% for human arguments, even when quality is held constant Do LLM judges systematically favor LLM-generated arguments?. Because this bias lives downstream of component scoring, it quietly corrupts any pipeline where AI grades AI output.

Underneath that sit four exploitable, content-agnostic biases — authority, verbosity, position, and beauty. Judges score responses higher when they carry fake references or rich formatting, regardless of whether the content is any good Can LLM judges be tricked without accessing their internals?, Can LLM judges be fooled by fake credentials and formatting?. The unsettling part is that these are zero-shot attacks: you don't need access to the model's internals to game it — you just dress up the answer. That makes AI benchmark scores fragile in a way that isn't obvious from the leaderboard.

Debates make this worse because of what LLMs structurally can't see. Human debates get settled by argument quality, social authority, reputation, and interpersonal trust; LLM debates collapse that into chain-of-thought probability ranking How do LLM debates differ from human expert consensus?. Since the model processes only text and not the social world where expertise is earned, it can't reliably tell an expert's claim from a widely-held assumption Can language models distinguish expert arguments from common assumptions?. So in exactly the contested domains where human judgment matters most, AI judges are prone to amplifying confident-sounding errors.

There's a deeper origin story here worth knowing: these tilts aren't mostly a fine-tuning artifact you can patch away. A causal study using random-seed variation showed cognitive biases are planted during pretraining and only nudged by instruction tuning Where do cognitive biases in language models come from?. And LLMs reproduce human belief-bias signatures item-by-item — they find believable-but-invalid arguments more convincing, the same way people do Do language models show the same content effects humans do?. The judge isn't a neutral logic engine; it inherits the reasoning shortcuts of its training data.

The hopeful counter-thread: these biases are partly fixable. Training judges with reinforcement learning to actually reason through evaluations — converting judgment into verifiable problems — cuts susceptibility to authority, verbosity, position, and beauty bias, because a judge that thinks stops leaning on surface features Can reasoning during evaluation reduce judgment bias in LLM judges?. Even a mechanical tweak helps: reading the expectation over scoring-token logits yields continuous scores that break the ties discrete judges produce and sharpens discrimination between answers Can reading logit distributions break ties in LLM judging?. The takeaway is that AI-judged debates aren't just noisy — they're biased in specific, nameable, and partly correctable directions, which is exactly why knowing the names matters.