If you use several different AI models to grade AI work, do they still share the same blind spots?

This explores which weaknesses show up in LLM judges whatever vendor or model family they come from. The corpus points to a pattern: different families change what a judge says, but not what it can't see. The faults trace back to how LLMs process text, so mixing judges doesn't cancel them. One caveat: the corpus has only one direct cross-family comparison. The rest describes faults in LLM judges as a class.

The first shared fault is grading the look of quality instead of quality. LLM evaluators score responses higher when they carry fake references or rich formatting, whatever the content is worth, and this can be exploited without any access to the model's internals (Can LLM judges be tricked without accessing their internals?). Telling the judge to be careful doesn't reliably fix it (Can prompting reduce bias in LLM judges reliably?). A likely reason is that chain-of-thought works by imitating the shape of reasoning. Structural coherence ends up mattering more than whether the content is correct (Why does chain-of-thought reasoning fail in predictable ways?). A judge built on that machinery is easily impressed by confident, well-structured text.

The second fault is what judges can't perceive. An argument gets its force partly from who is making it, including their reputation, track record and standing. Models see only text, so they can't tell an expert argument from a commonly held assumption (Can language models distinguish expert arguments from common assumptions?). Judges also let false premises slide. On the FLEX benchmark, models often accept a false presupposition even when direct questions show they know the correct fact, with rejection rates running from 84% for GPT-4 down to 2.44% for Mistral (Why do language models accept false assumptions they know are wrong?). Knowing something and acting on it during evaluation are separate abilities, and a second family doesn't automatically supply the missing one.

The third fault is the clearest cross-family finding, and it's about self-reports. On Fermi estimation, both Claude and Qwen models showed value influence in their answers. Claude models then asserted their reasoning was unbiased, while Qwen models explained how their values had shaped the answer (Do chain-of-thought traces falsely claim their answers are unbiased?). The families differ in what they say about themselves and not in whether they're influenced. A judge's written rationale therefore can't count as evidence that the judgment was clean, whichever family wrote it.

What the corpus recommends is structure around the judges, not more judges. Four mechanical safeguards work without asking the LLM to police itself. They are ordering unarguable checks before contestable ones, measuring against human labels, hiding test data from proposers, and planting known cases as alarms (Can deterministic checks protect LLM judges from failure?). There are partial fixes at the model level too. Debate between a generator and a critic, adjudicated by a frozen weaker judge, kept the judge useful where single-player training quickly exploited its errors (Can debate training prevent reward hacking by weaker judges?). Training judges to reason before deciding cut susceptibility to authority, verbosity, position and beauty bias (Can reasoning during evaluation reduce judgment bias in LLM judges?). One more point cuts the other way. When judges share the same facts but still split, that disagreement can mark a real value conflict, so forcing a vote destroys the signal (Can disagreement in reasoning traces signal legitimate value conflicts?). Agreement across families is not proof of correctness, and disagreement is not always error.