Line of inquiry
Inquiring lines›What determines when language mode…›What systematic behavioral limitat…›this line of inquiry
What systematic biases do LLM judges exhibit in evaluation?
A broader line of inquiry — a family of 68 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 68
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- What other evaluation biases exist in LLM judge systems?
- Can an LLM judge's bias be reduced through prompting or other interventions?
- How do LLM judges' built-in biases influence the policies they help align?
- Does LLM judge preference for LLM arguments amplify errors in contested factual domains?
- Why do LLM judges systematically favor outputs from their own model family?
- Do LLM judges with diverse personas resist individual biases better than single evaluators?
- Do LLM judges systematically favor arguments from other LLMs?
- Can parallel evaluation reduce position and length bias in LLM judging?
- How does LLM judge bias amplify errors in multi-agent debate on contested factual questions?
- What calibration corrections can reduce LLM judge bias in automated evaluation pipelines?
- What biases do single large LLM judges introduce into comparisons?
- What are the four catalogued biases that make LLM judges vulnerable to prompt attacks?
- Does ensembling smaller judges reduce bias more effectively than single large judges?
- Does LLM judge bias matter more when the judge allocates scarce opportunities?
- Why do LLM judges show more extreme sycophancy bias than humans?
- What systematic biases do LLM judges introduce into AI-evaluated debates?
- Do smaller LLM judge panels outperform single large judges in practice?
- Can structured evaluation pipelines reduce LLM reviewer bias?
- Why do current language model judges collapse into coarse discrete scores?
- Can counterfactual invariance techniques address exploitable biases in LLM judges?
- How does stylistic matching contribute to LLM self-preference in evaluations?
- What four exploitable biases make current LLM judges vulnerable to zero-shot attacks?
- Does longer context and multi-step reasoning compound small preferences in graders?
- Why do LLMs fail inter-annotator agreement tests on argument evaluation?
- Why do LLMs systematically prefer text from their own family?
- Which biases in LLM judges are exploitable through presentation alone?
- Do LLM judges remain vulnerable to gaming when anchored to external references?
- Why do LLMs show gender bias but humans evaluators do not?
- Does a single LLM judge capture diverse human preferences in alignment training?
- Can instruction prompts reliably steer an LLM judge toward specific alignment targets?
- Can LLMs evaluate logical argument quality in debates they themselves can win?
- Can crowdsourced voting and automated panels both credibly evaluate LLM outputs?
- Can LLM judges be trained to think more rigorously during evaluation?
- Why is consistency between argument scoring and winner selection lower for LLM judges?
- What surface features do LLMs rely on when judging response quality?
- Why do LLM judges assign high argument strength scores yet pick LLM winners anyway?
- What shared epistemic faults persist even when judges come from different families?
- How robust is misalignment classification across different judge models?
- Why do humans and zero-shot LLM judges perform worse than trained detectors?
- What makes a judge's calibration at decision boundaries harder to improve?
- Can an LLM judge reliably report its own biases rather than remove them?
- Which prompting strategy for judges best resists semantic content manipulation attacks?
- Can critique-only calls in LLMs exploit a measurable gap between generation and evaluation?
- How sensitive are LLM bias measurements to analysis choices?
- What biases might an LLM judge introduce into an on-policy alignment process?
- Do mechanical guardrails around judges bound the cost of judge errors?
- Why do LLM-judged tournaments fail to estimate candidate value or uncertainty reliably?
- How can we validate LLM-based drift measurements against human judgment?
- How do calibration and reliability differ in LLM judge evaluations?
- Why do readers rate LLM-edited text more favorably?
- Why do LLMs swing on minor rewording yet ignore explicit bias correction instructions?
- Can LLM judges reliably estimate when they lack sufficient persona information?
- How does same-author bias interact with the four adversarial judge biases already documented?
- Do LLMs lack evaluative capacity or only default taste and stance?
- How does era sensitivity in legal cases compound with context length failures?
- What design choices make it survivable when an LLM judge holds final authority over an optimizer?
- Which LLM providers showed deeper mentalizing in the inspection game versus rock-paper-scissors?
- Can masking company identity in grading materials eliminate the bias?
- What did deleting the rubric do to the judge's error in practice?
- How can deterministic checks make wrong judge decisions survivable?
- What role should stakeholders play in evaluating LLM fairness?
- How much better is a panel of smaller judges than one large judge?
- What role do model-based critics play in validating LLM plans?
- When does an LLM judge's error become terrain that an optimizer can map and exploit?
- What does McDonald's omega reveal about LLM judgment consistency?
- What makes externally enforced governance reshape LLM behavior better than written rules?
- Does personalized rubric training in one writer's case actually generalize?
- Can LLM performance on zebra cases predict results in routine clinical practice?