Line of inquiry
Inquiring lines›How can we understand and verify l…›What linguistic patterns explain L…›this line of inquiry
What biases affect LLM judges and how can they be reduced?
A broader line of inquiry — a family of 62 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 62
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- What other evaluation biases exist in LLM judge systems?
- Can an LLM judge's bias be reduced through prompting or other interventions?
- How do LLM judges' built-in biases influence the policies they help align?
- Do LLM judges with diverse personas resist individual biases better than single evaluators?
- Does LLM judge preference for LLM arguments amplify errors in contested factual domains?
- Can parallel evaluation reduce position and length bias in LLM judging?
- What calibration corrections can reduce LLM judge bias in automated evaluation pipelines?
- Does ensembling smaller judges reduce bias more effectively than single large judges?
- Why do LLM judges systematically favor outputs from their own model family?
- Do LLM judges systematically favor arguments from other LLMs?
- How does LLM judge bias amplify errors in multi-agent debate on contested factual questions?
- What biases do single large LLM judges introduce into comparisons?
- Do LLM judges have uneven skill across different evaluation criteria?
- Why do LLM judges show more extreme sycophancy bias than humans?
- What are the four catalogued biases that make LLM judges vulnerable to prompt attacks?
- Does LLM judge bias matter more when the judge allocates scarce opportunities?
- What systematic biases do LLM judges introduce into AI-evaluated debates?
- Do smaller LLM judge panels outperform single large judges in practice?
- Can structured evaluation pipelines reduce LLM reviewer bias?
- Can counterfactual invariance techniques address exploitable biases in LLM judges?
- Why do current language model judges collapse into coarse discrete scores?
- What four exploitable biases make current LLM judges vulnerable to zero-shot attacks?
- Does longer context and multi-step reasoning compound small preferences in graders?
- Why do LLMs show gender bias but humans evaluators do not?
- Why do LLMs fail inter-annotator agreement tests on argument evaluation?
- Do language models inherit gender bias from training data in grading tasks?
- Can judges trained on both verifiable and non-verifiable tasks transfer across domains?
- Which biases in LLM judges are exploitable through presentation alone?
- Can crowdsourced voting and automated panels both credibly evaluate LLM outputs?
- Can LLMs evaluate logical argument quality in debates they themselves can win?
- Do LLM judges remain vulnerable to gaming when anchored to external references?
- How robust is misalignment classification across different judge models?
- Why is consistency between argument scoring and winner selection lower for LLM judges?
- What happens when LLMs grade other LLMs in closed evaluation loops?
- Why do humans and zero-shot LLM judges perform worse than trained detectors?
- Can instruction prompts reliably steer an LLM judge toward specific alignment targets?
- How does reference-anchoring compare to training judges with reinforcement learning?
- What shared epistemic faults persist even when judges come from different families?
- Does debate training prevent reward hacking when judges show preference bias?
- What makes a judge's calibration at decision boundaries harder to improve?
- Can LLM judges be trained to think more rigorously during evaluation?
- Why do LLM judges assign high argument strength scores yet pick LLM winners anyway?
- Can an LLM judge reliably report its own biases rather than remove them?
- What biases might an LLM judge introduce into an on-policy alignment process?
- How sensitive are LLM bias measurements to analysis choices?
- How do calibration and reliability differ in LLM judge evaluations?
- Do mechanical guardrails around judges bound the cost of judge errors?
- How does same-author bias interact with the four adversarial judge biases already documented?
- Why do LLM-judged tournaments fail to estimate candidate value or uncertainty reliably?
- Can LLM judges reliably estimate when they lack sufficient persona information?
- Can masking company identity in grading materials eliminate the bias?
- Where should measurement systems sit to avoid recording bias?
- Can a diverse panel approach work for validators beyond text evaluation?
- How can deterministic checks make wrong judge decisions survivable?
- What did deleting the rubric do to the judge's error in practice?
- How much better is a panel of smaller judges than one large judge?
- Do own-company biases differ across model families in grading tasks?
- How do rater disagreements and consensus processes affect tacit knowledge production in RLHF?
- What role should stakeholders play in evaluating LLM fairness?
- Do humans and LLMs exhibit opposite biases in public versus private reviews?
- What does McDonald's omega reveal about LLM judgment consistency?
- When does an LLM judge's error become terrain that an optimizer can map and exploit?