When should practitioners stop decomposing LLM evaluations?

The paper's discussion section proposes a three-step "layered workflow" for combining fragmented and holistic LLM evaluation rather than treating them as competing methods: "(1) Start with Fragmented Evaluation on Broad Criteria... (2) Iterate and Concretize Criteria with Function Examples... (3) Zoom Out and In." On step 2 it says practitioners should "iteratively refine their criteria and example sets while re-evaluating the outputs until most misalignments disappear and the function clusters stabilize (i.e., similar clusters are presented each evaluation)," and only then, in step 3, move to trusting holistic justifications and scores, while retaining the ability to "dive deep back to the fragmented evaluations when they need more details."

Each step corresponds to a different level of abstraction in the authors' own terms. Step 1 surfaces fragment-level functions the practitioner "may have not previously considered," which the paper compares to inductive coding. Step 2 treats those functions as examples that sharpen the criteria themselves, which it compares to how "educators develop rubrics by assessing examples of student work." Step 3 becomes trustworthy only once the first two steps have removed disagreement at the fragment level, so the aggregated score is no longer hiding an unresolved dispute about what counts as a misalignment — the paper frames the whole sequence as practitioners needing "to flexibly alternate between levels of abstraction."

This is the companion claim to Can breaking outputs into fragments reveal more evaluation problems?: where that note is about what fragment-level decomposition finds, this one is about when a practitioner should stop decomposing and trust the aggregate again. It supplies a concrete stopping condition — "function clusters stabilize" — for a question that Can breaking down instructions into checklists improve AI reward signals? leaves implicit, since checklist decomposition there is useful only until the checklist itself stops changing.

The guideline is a design recommendation drawn from the same ten-person study and from what the paper calls guidance for "integrating fragmented and holistic evaluations... in practice," not from a separate test of the three-step workflow itself. The paper states plainly that "further studies are needed to understand actual usage patterns" of combining the two methods, so the sequencing is a proposal the authors offer rather than a workflow they measured end-to-end.