Does an AI reviewer grade what a paper says, or how boldly it says it, when the science stays the same?

This explores whether an LLM reviewer judges what a paper says or how it says it. The most direct evidence suggests presentation matters a lot. When researchers rewrote manuscripts' rhetoric and left the scientific content untouched, LLM reviewer scores moved in measurable, consistent ways How much does rhetorical style shift AI review scores?. Two levers had the biggest effect: how the evidence was framed and how boldly the paper claimed novelty. The effect also depended on which model did the reviewing and on how strong the paper was to begin with. So framing doesn't add a fixed bonus. It interacts with quality, and that makes it harder to spot.

This fits a wider pattern in how LLMs act as judges. In other settings, LLM evaluators fall for authority signals like fake references and for rich formatting. Exploiting this takes no access to the model and no optimization, only surface changes that carry no meaning Can LLM judges be fooled by fake credentials and formatting?. The same thing appears outside evaluation: GPT-4 changes what information it gives depending on the emotional tone of a question, pulling negative prompts back toward neutral-positive answers Does emotional tone in prompts change what information LLMs provide?. One detail there bears on 'critical topics': the tone effect was suppressed on sensitive subjects, where alignment constraints took over. That raises an open question the collection doesn't yet answer. Do reviewers react to framing differently when a paper's subject is politically or ethically charged? No note here tests that directly.

A quieter version of the problem is style that reads as machine-written. LLM judges pick LLM-written arguments as winners far more often than humans do Do LLM judges systematically favor arguments from other LLMs?. Early causal evidence suggests this happens because models recognize their own style Do LLMs favor their own text because they recognize it?. Human readers also rated LLM-edited abstracts as the clearest, even though they couldn't reliably tell which ones were AI-touched Can readers tell LLM abstracts from human ones?. Here's the twist. Across more than 125,000 real conference reviews, the apparent favoritism of LLM-assisted reviewers toward LLM-written papers disappeared once paper quality was controlled for. What was left was a general leniency toward weaker work Do LLM reviewers actually favor LLM-written papers?. Some 'framing bias' may really be a softness toward weak papers that polished presentation exploits.

What helps? Breaking review into structured steps seems to make it less vulnerable to impressions. A pipeline that pulls out a paper's claims, finds related work, and then compares them matched human novelty judgments much more closely than asking an LLM for one overall verdict Can structured pipelines make LLM novelty assessment reliable?. A pipeline like that checks a bold novelty claim against the prior literature instead of just reacting to the tone. The takeaway you may not have expected: the safeguard against rhetorical framing isn't banning LLMs from review. It's restructuring the task so that confident prose has less to grab onto.