Can breaking outputs into fragments reveal more evaluation problems?

The paper's authors propose functional fragmentation, a method for evaluating LLM outputs that dissects each output into fragments and interprets the rhetorical function each fragment serves relative to an evaluation criterion, rather than collapsing judgment into a single holistic score. They instantiate the method in Evalet, an interactive system that surfaces these fragment-level functions for inspection, rating, and cross-output comparison. In a user study with ten practitioners (N=10), the approach helped them identify "48% more evaluation misalignments" than holistic LLM-as-a-Judge scores did, and the paper reports this let them "calibrate trust in LLM evaluations and rely on them to find more actionable issues in model outputs."

The mechanism: holistic LLM-as-a-Judge scores (e.g., "3 out of 5" for a criterion like Age Appropriateness) force a practitioner to manually re-read the whole output to discover which passage drove the score — the paper's example is an explanation of "T cells" that scores moderately because it mixes simple vocabulary with "potentially harmful war imagery." Functional fragmentation instead extracts fragments and labels their function (e.g., "personification," "war-related imagery") per criterion, so a single fragment can carry different functions across different criteria at once. Evalet builds three affordances on this: inspecting a per-criterion list of surfaced functions instead of scanning the full output; rating each function's alignment independently so practitioners can re-rate or flag functions rather than editing criteria descriptions; and comparing fragments across many outputs by clustering them on functional similarity rather than lexical similarity, which the paper says can reveal a model "over-relying" on a theme — its example clusters "shooting their special microscopic guns" with "move like giant armored tanks" as the same war-imagery function despite the different wording.

This extends, at the level of a single output, what Do all AI skills improve equally as models scale? establishes at the level of the evaluator: FLASK shows LLM judges have uneven skill profiles across criteria, while Evalet decomposes not the judge's skill set but the judged artifact itself, fragment by fragment. It also parallels Can breaking down instructions into checklists improve AI reward signals?, which decomposes is this good? into checklist or rubric items as training signal; Evalet's fragment-functions serve the same decomposition logic but for human inspection rather than RL reward, and are discovered inductively from the output rather than predefined. Both moves answer, from opposite ends, the construct-validity problem that Do LLM benchmarks actually measure what they claim to measure? documents across benchmarks: holistic scores can aggregate away the very phenomenon they claim to measure, and fragment-level functions are one way to keep that phenomenon visible.

What the study does not establish: ten practitioners in one session design is a small sample, and the paper says it compared fragmented against holistic evaluation "to clearly isolate their distinct affordances" rather than testing them combined in real workflows, adding that "further studies are needed to understand actual usage patterns." The 48% figure measures how many more misalignments practitioners identified during inspection, not whether the underlying LLM evaluator became more accurate — functional fragmentation changes what practitioners can see, not what the judge model gets right or wrong. The paper also notes the version studied omitted function-priority weighting and several controls (relabeling, cluster editing, fragment-size editing) that pilot users wanted but were cut for time.