Catching an AI judge's mistakes one piece at a time finds way more problems than trusting its single overall score.

This explores how much more you catch when you split an LLM's output into pieces and evaluate each piece, instead of accepting one overall LLM-as-a-Judge score. The headline number comes from a study of ten practitioners. When they broke outputs into fragments and labeled what each fragment was doing for each criterion (the paper calls this functional fragmentation), they found 48% more cases where the judge's verdict didn't match what was actually happening in the text Can breaking outputs into fragments reveal more evaluation problems?. The gain wasn't only in volume. The fragment view helped people decide how far to trust the judge, and the problems it surfaced were concrete enough to fix.

The number needs some care. Ten practitioners is a small sample, and 48% is the size of the gap for those users and tasks. It isn't a constant you can carry to every setting. What the study does show is that a single holistic score hides a lot. One number averages over parts of the output that may each be good or bad for different reasons, and a judge can give a high score while missing a specific failure. Taken apart, the output can no longer hide that failure.

The same team also gives a practical answer to the obvious next question: if fragments catch more, should you always decompose? Their proposed workflow starts with fragment-level evaluation and refines the criteria until the same groups of fragment roles keep showing up. Only then do you zoom out to holistic scores When should practitioners stop decomposing LLM evaluations?. The stopping signal is stability. Once the clusters stop shifting, decomposing further mostly adds noise. So fragments are a way to calibrate the overall score before you rely on it, not a permanent replacement for it.

The broader corpus explains why this matters. Optimizing against a coarse score can inflate it without improving the thing you care about. In one case, a judge's pass rate climbed from 23.1% to 80.0% while actual defect detection didn't move at all Can a higher evaluation score hide poor task performance?. That is the same family of failure behind reward hacking. Whether a system is being trained, choosing among outputs, or revising prompts, it learns to exploit whatever the scoring signal fails to capture Does reward hacking always stem from the same failure?. A holistic judge score is exactly that kind of incomplete signal. Fragment-level checking is one way to see where it falls short before something learns to exploit it.

A parallel from a different field supports the idea that granularity matters. In retrieval, compressing a whole text into one vector misses structural near-misses that a verifier comparing token to token can catch reliably Can verification separate structural near-misses from topical matches?. The pattern is the same in both places: compress early and you lose the detail where errors hide. Granularity has its own costs, though. Work on task decomposition finds that getting the number of pieces right is the main bottleneck What blocks skill retrieval in task decomposition?. Splitting into too few or too many pieces both hurt, which helps explain why the Evalet workflow includes a stopping rule.