If several AI reviewers all fall for the same polished writing, does their agreement prove the review is reliable?

This explores whether agreement across several LLMs about a paper's rhetoric shows that AI scientific review is stable. The corpus has no direct test of that exact claim, but the nearby evidence points toward no. Agreement across models is weak evidence, because the models tend to share the same blind spots.

The biases look built into how LLMs read, not quirks of one model. LLM judges fall for fake references and polished formatting regardless of content, and those attacks need no access to the model at all Can LLM judges be fooled by fake credentials and formatting?. Text-only models also can't tell an expert's hard-won argument from a commonly held assumption, because the reputation and track record that give claims their force never reach them Can language models distinguish expert arguments from common assumptions?. Generation itself is a smooth flow toward the training distribution, not an exploration of counterpositions Does LLM generation explore competing claims while producing text?. An audit of five models found they all persuade with logical appeals and quantitative framing in virtually every conversation Do LLMs persuade users more often than humans do?. A paper written in that register is speaking their shared native language, so consistent scores across models can mean the same bias is showing up five times.

Models also tend to mirror whatever argument is in front of them. They hold the shape of the current argument, not a defended position Do LLMs actually hold stable positions or just mirror user arguments?, and a paper is an argument with a shape. Emotional tone can shift what a model says on identical questions Does emotional tone in prompts change what information LLMs provide?. Models also struggle to hold several interpretations at once (GPT-4 disambiguates only 32% of cases versus 90% for humans) Can language models recognize when text is deliberately ambiguous?, so an author's framing can settle which reading the reviewer adopts. Humans are swayed by presentation too: users prefer responses with more citations even when the citations are irrelevant Do users trust citations more when there are simply more of them?. The fair comparison is therefore human reviewers, not a perfect reader.

What cross-model robustness does track is confidence. Models resist prompt rephrasing when they are confident, and confidence is higher for larger models, few-shot prompts and objective tasks Does model confidence predict robustness to prompt changes?. Robustness may therefore show up where the task is easy, not where the review is right, and judging novelty is neither easy nor objective. Consistency also has no ground truth of its own. One line of work finds that a trustworthy confidence signal comes from a model's accumulated outcome history, not from any single inference Can past performance predict when a model will be right?. Stability under rewriting is worth measuring, but it only means something once it is checked against what later turned out to be true.

The better route is to change the pipeline, not to rely on robustness as a signal. A three-stage process (extract the claims, retrieve related work, compare) reached 86.5% reasoning alignment and 75.3% conclusion agreement with human reviewers on 182 ICLR submissions, beating holistic LLM baselines Can structured pipelines make LLM novelty assessment reliable?. It works partly because it strips out the rhetorical layer before any judgment happens. There is reason for optimism about forward-looking judgment generally: fine-tuned LLMs beat neuroscience experts at predicting which experimental results occurred Can LLMs predict novel scientific results better than experts?. The useful stress test is to hold the science fixed, rewrite only the framing, and see whether scores move. Then check whether the surviving scores match real outcomes.