INQUIRING LINE

Rewrite a paper's wording without touching its science, and AI reviewers still change their scores — most when the evidence is framed differently.

Why do evidence framing choices move AI review scores more than other rhetorical changes?

This explores why, when a paper's science is held fixed and only its wording changes, the way evidence is framed swings an AI reviewer's score more than other stylistic changes.


This explores why, when a paper's science is held fixed and only its wording changes, the way evidence is framed swings an AI reviewer's score more than other stylistic changes. The corpus documents the effect but doesn't test the cause. Rewriting manuscripts' rhetoric while keeping the content identical moved LLM reviewer scores measurably. Evidence framing and novelty stance produced the biggest contrasts, and the size of the effect varied by reviewer model and by the paper's original quality tier How much does rhetorical style shift AI review scores?. Nothing in the library says why. The rest of this answer is inference from neighboring notes.

The leading explanation is that evidence framing is the part of a paper a reviewer can only judge by how it's presented. Evidence framing changes phrases like "extensive experiments confirm" versus "preliminary results suggest". These are credibility signals, and a reviewer that reads instead of re-running experiments has to treat them as stand-ins for whether the evidence is actually strong. A separate study of LLMs judging persuasion supports this. The models leaned on topical overlap and credibility, while humans responded more to novelty and assertive language, and the two agreed only slightly Do language models judge persuasion the way humans do?. That study is about arguments, not peer review, so treat the link as a hypothesis. Novelty, the other big lever in the review study, is what humans favored there, so LLM reviewers may be responding to both. A persuasion taxonomy built for AI explanations makes a related point: every piece of text carries logical, credibility and emotional appeals at once, whether or not the author intends it How do logos, ethos, and pathos shape AI explanations?. Evidence framing loads the logic and credibility channels directly, so it is hard to rewrite it without changing what the reviewer hears.

A second clue is confidence. Models that are unsure of their answer swing widely when the prompt is rephrased, while confident models barely move Does model confidence predict robustness to prompt changes?. Judging a paper's merit is a low-confidence task, especially for borderline papers. When the model can't tell whether the evidence is good, how the paper describes its evidence becomes the easiest thing to grab. This would predict the pattern where effects vary by reviewer model and interact with the paper's quality tier. The review study reports that interaction but not its direction, so the prediction is untested.

If that's right, the fix is a reviewer that checks the evidence itself. An agentic judge that collects its own evidence showed 0.27% judge shift, against 31% for a plain LLM judge on complex tasks Can agents evaluate AI outputs more reliably than language models?. That was not a paper-review benchmark, but the logic carries over. A judge that only reads the presentation inherits its framing, and one that verifies doesn't. In the same spirit, an agentic reviewer using extra test-time compute to check proofs and experiments line by line caught mathematical flaws that passed human review at STOC and ICML Can inference scaling help reviewers catch errors humans miss?. The likeliest answer to your question is that evidence framing is the cheapest substitute for actually verifying the evidence, and a reviewer that skips verification is easy to steer through it. The agent-judge work also cautions that its memory module cascaded errors, so a checking reviewer needs safeguards against compounding its own mistakes.


Sources 6 notes

How much does rhetorical style shift AI review scores?

Rewriting manuscripts' rhetoric while preserving scientific content moves LLM reviewer scores measurably. Evidence framing and novelty stance produce the largest contrasts; effects vary by reviewer model and interact with the paper's original quality tier.

Do language models judge persuasion the way humans do?

LLMs show only slight agreement with human-verified persuasion outcomes (Cohen's κ = 0.079–0.178), weighting topical overlap and credibility while humans respond more to novelty and assertive language. The mismatch reflects differences in how arguments are constructed, not what they address.

How do logos, ethos, and pathos shape AI explanations?

Aristotle's three appeals map onto explanation design across two goals (how AI works, why AI merits use), creating a 3×2 space where every explanation loads all three channels simultaneously. Naming these rhetorical channels lets designers account for unintended persuasive effects.

Does model confidence predict robustness to prompt changes?

ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.

Can agents evaluate AI outputs more reliably than language models?

Eight-module agentic evaluation achieved 0.27% judge shift versus 31% for LLM-as-a-Judge on complex tasks. However, the memory module cascaded errors, revealing that agentic systems need error isolation mechanisms to maintain gains.

Show all 6 sources
Can inference scaling help reviewers catch errors humans miss?

PAT, an agentic reviewer using test-time compute to check proofs and experiments line by line, achieves 34% better recall on math errors than zero-shot approaches and surfaced critical flaws at STOC and ICML that passed human review.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.