When an AI reviewer grades a paper, does a bolder, more polished presentation shift its score even though the science is unchanged?
How much does rhetorical framing shift LLM reviewer scores independent of content?
This explores whether the way a paper is written (how it frames its evidence, how boldly it claims novelty, how polished it sounds) moves an AI reviewer's score even when the science underneath stays exactly the same, and what else besides content these reviewers respond to.
This explores whether LLM reviewers can be swayed by how a paper is presented rather than by what it shows. The most direct answer comes from an experiment that rewrote manuscripts' rhetoric while keeping their scientific content fixed. Scores moved measurably. The biggest shifts came from two choices: how the evidence was framed and how strongly the paper claimed to be new How much does rhetorical style shift AI review scores?. The effect isn't a single fixed number. It depends on which model is doing the reviewing, and it interacts with how strong the paper was to begin with. The same rewrite can help one paper and do little for another, so 'how much' has a messy answer: enough to matter, and different from one setup to the next.
This fits a wider pattern in how LLMs act as judges. One line of research found that LLM judges fall for fake authority signals, such as invented references, and for rich formatting. Both tricks work without any access to the model, because they don't depend on what the text actually says Can LLM judges be fooled by fake credentials and formatting?. Rhetorical framing in a paper is a gentler version of the same weakness. If a judge can be moved by surface cues meant to deceive it, it's no surprise that it also responds to ordinary persuasive writing. Something similar shows up outside evaluation: GPT-4 changes what information it gives depending on the emotional tone of a prompt, so identical questions get different answers Does emotional tone in prompts change what information LLMs provide?. Sensitivity to framing seems to be a general trait of these models, not a quirk of peer review.
There's a twist worth knowing about: some biases that look like framing effects turn out to be something else. Across more than 125,000 reviews, LLM-assisted reviewers appeared to favor LLM-written papers. The apparent favoritism disappeared once paper quality was held constant. LLM-written papers tended to be among the weaker submissions, and LLM reviewers are lenient toward weaker work in general Do LLM reviewers actually favor LLM-written papers?. Other studies do find a genuine same-source preference. LLM judges picked LLM-written arguments as winners 62% of the time Do LLM judges systematically favor arguments from other LLMs?, and a model's preference for its own text tracks how well it can recognize that text Do LLMs favor their own text because they recognize it?. The lesson is that measuring a pure framing effect requires the design used in the rhetoric study: same content, different wording. Simply comparing scores across papers isn't enough.
Humans don't make a clean control group either. Evaluators have rated AI-generated documents as better than human ones while also believing they were written by humans Does polished writing actually signal better quality work?. Expert readers couldn't reliably spot LLM abstracts, yet they gave LLM-edited abstracts the highest clarity ratings Can readers tell LLM abstracts from human ones?. Polish persuades people and machines alike, so swapping an LLM reviewer for a human one doesn't remove the problem; it just shifts where the bias sits. One promising fix is structure. A pipeline that pulls out a paper's claims, retrieves related work, and compares the two reached 86.5% agreement with human reviewers' reasoning on novelty, better than asking a model for an overall verdict Can structured pipelines make LLM novelty assessment reliable?. Forcing the reviewer to break down what a paper claims, rather than judge how it reads, may be the best available defense against framing. The corpus doesn't yet include a study that tests that pipeline directly against rhetorical rewrites.
Sources 9 notes
Rewriting manuscripts' rhetoric while preserving scientific content moves LLM reviewer scores measurably. Evidence framing and novelty stance produce the largest contrasts; effects vary by reviewer model and interact with the paper's original quality tier.
Research identified four evaluation biases in LLM judges, with authority and beauty biases being semantics-agnostic and trivially exploitable through fake references and formatting—zero-shot attacks requiring no model access or optimization.
GPT-4 exhibits emotional rebound (negative prompts yield ~86% neutral-positive responses) and a tone floor (positive prompts rarely go negative), causing identical questions to receive different answers depending on emotional framing. This bias is suppressed only on sensitive topics where alignment constraints override tone effects.
Across 125,000+ reviews, the apparent favoritism of LLM-assisted reviewers toward LLM papers disappears once paper quality is held constant. LLM papers cluster among weaker submissions, creating a spurious interaction driven by LLM reviewers' general leniency toward lower-quality work.
LLM judges selected LLM arguments as winners 62% of the time versus humans' 37%, while humans split votes 39% LLM / 37% human. This same-author bias operates downstream of component scoring and compounds existing judge vulnerabilities, creating a calibration ceiling in RLAIF pipelines.
Show all 9 sources
Fine-tuning LLMs to recognize their own summaries increased their preference for those summaries in a linear relationship, suggesting recognition capability drives self-preference bias. The authors present this as initial causal evidence, not proof.
Studies show evaluators perceived AI-generated documents as both human-written and better quality than human submissions. This suggests rhetorical polish misleads judgment and should not serve as a quality signal in evaluation.
Readers with ML expertise struggle to identify LLM-generated content reliably, tending to assume human involvement across all abstract types. However, LLM-edited abstracts received highest clarity ratings and were preferred 55% of the time when authorship was disclosed.
A three-stage pipeline (extract claims, retrieve related work, compare) reached 86.5% reasoning alignment and 75.3% conclusion agreement with human reviewers on 182 ICLR submissions, outperforming holistic LLM baselines.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- LLM or Human? Perceptions of Trust and Information Quality in Research Summaries
- LLM-REVal: Can We Trust LLM Reviewers Yet?
- Do LLMs Favor LLMs? Quantifying Interaction Effects in Peer Review
- Stop Automating Peer Review Without Rigorous Evaluation
- AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights
- Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
- Humans or LLMs as the Judge? A Study on Judgement Biases
- How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review