AI models that grade other AI answers can be fooled by fake citations, fancy formatting, longer answers, and answer order.
What are the four catalogued biases that make LLM judges vulnerable to prompt attacks?
This explores which specific biases the corpus catalogues in LLM judges (models that score other models' answers), and why they let attackers game a judge with plain prompt tricks.
This explores which biases the corpus catalogues in LLM judges and why they make the judges easy to attack. The four are authority, beauty, verbosity and position. One note reports that judges have four exploitable evaluation biases Can LLM judges be fooled by fake credentials and formatting?. The note on training judges to reason is the one that lists all four by name: authority, verbosity, position and beauty Can reasoning during evaluation reduce judgment bias in LLM judges?. The attack notes only work through two of them in detail, so those two are the best documented.
Authority and beauty are the two demonstrated as attacks. Authority bias means a judge scores a response higher when it includes fake references. Beauty bias means it scores higher when the response has rich formatting. Both are semantics-agnostic: they work regardless of whether the answer is any good. That makes them zero-shot attacks that need no access to the model and no optimization, just pasted citations or tidy markdown Can LLM judges be fooled by fake credentials and formatting? Can LLM judges be tricked without accessing their internals?. The notes retrieved here only name verbosity and position, without describing attacks for them. The names suggest favoring longer answers and favoring an answer because of where it sits in the comparison. The practical consequence is that any leaderboard or reward signal built on an LLM judge can be inflated by decoration rather than substance.
These four are not the whole problem. LLM judges also pick LLM-written arguments as winners 62% of the time, against 37% for human judges. That same-author bias operates on top of the other weaknesses and sets a ceiling on how well AI-graded training pipelines can calibrate Do LLM judges systematically favor arguments from other LLMs?.
The corpus says telling the judge to be unbiased does not reliably fix this, so systems should contain judge errors structurally instead Can prompting reduce bias in LLM judges reliably?. It offers three structural routes:
- **Reasoning judges:** train the judge with reinforcement learning to think through its verdict, so it stops leaning on surface features like credentials and formatting Can reasoning during evaluation reduce judgment bias in LLM judges?. - **Deterministic guardrails:** run mechanical checks that don't depend on the judge's own honesty, such as ordering unarguable checks first and planting known cases as alarms Can deterministic checks protect LLM judges from failure?. - **Debate:** have a generator and a critic argue in front of a weaker judge, which kept judge performance stable and avoided the early reward-hacking collapse seen with single-player training Can debate training prevent reward hacking by weaker judges?.
Sources 7 notes
Research identified four evaluation biases in LLM judges, with authority and beauty biases being semantics-agnostic and trivially exploitable through fake references and formatting—zero-shot attacks requiring no model access or optimization.
Training judges with reinforcement learning to reason about evaluations—by converting judgment tasks into verifiable problems with synthetic data pairs—produces judges that think through their decisions rather than relying on exploitable surface features, directly mitigating authority, verbosity, position, and beauty bias.
Research shows LLM evaluators systematically score higher when responses include fake references or rich formatting, independent of content quality. These biases are exploitable without model access, undermining AI benchmark credibility.
LLM judges selected LLM arguments as winners 62% of the time versus humans' 37%, while humans split votes 39% LLM / 37% human. This same-author bias operates downstream of component scoring and compounds existing judge vulnerabilities, creating a calibration ceiling in RLAIF pipelines.
Research evidence suggests that instructing LLM judges to reduce bias does not reliably work. The practical implication is that system design should focus on containing judge errors through structural checks rather than attempting to eliminate bias through better instructions.
Show all 7 sources
Research identifies four mechanical safeguards: ordering unarguable checks before contestable ones, measuring correctness against human labels, hiding test data from proposers, and using planted cases as alarms. None requires the LLM itself to verify compliance.
On math tasks, debate between a generator and critic adjudicated by a frozen weaker judge maintained judge performance throughout training and achieved 45% higher peak validation accuracy than single-player RLAIF, which quickly exploited the judge's errors and collapsed in accuracy.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Humans or LLMs as the Judge? A Study on Judgement Biases
- Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
- J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
- LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails
- When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injection
- The Thin Line Between Comprehension and Persuasion in LLMs