Researchers recommend a three-step process for grading AI output quality — but does anyone actually follow it once they're on the job?

This explores whether the 'fragment first, then refine, then zoom out' workflow for judging LLM outputs is how practitioners really behave, or only a recommendation. The short answer is that the collection doesn't show anyone using it in practice. Evalet's workflow is a guideline built from a small lab study. Its rule is to break outputs into fragments, keep refining criteria until similar clusters of fragment functions keep showing up, and only then move to holistic scores (When should practitioners stop decomposing LLM evaluations?). The evidence behind it comes from ten practitioners. Working at the fragment level helped them find 48% more cases where the automated judge's score didn't match what was actually in the output (Can breaking outputs into fragments reveal more evaluation problems?). That shows the first step pays off. It doesn't show that teams outside a study adopt the full sequence.

The more revealing evidence comes from a nearby study, which suggests real evaluation can't follow a clean three-step order. In EvalGen's study of nine practitioners, people needed criteria before they could grade outputs, but grading outputs kept changing what the criteria should be (Do evaluation criteria shift when we grade outputs?). This 'criteria drift' explains why Evalet's middle step exists. Its stopping rule, 'refine until clusters stabilize,' is essentially an admission that practitioners go back and forth between steps. A learner should picture the workflow less as a recipe and more as a loop with a test for when you can stop.

Looking more broadly, decomposed evaluation clearly beats one-shot judgment. Whether people stick with it is a separate question. A three-stage pipeline for judging how novel a research paper is works the same way: extract the claims, retrieve related work, then compare. It reached 86% reasoning agreement with human reviewers, beating holistic LLM judging (Can structured pipelines make LLM novelty assessment reliable?). Agent-based judges that gather evidence step by step cut how much a judge's verdict shifts from about 31% to 0.27%. They also showed that multi-step evaluation creates new failure points: errors in one module spread to the next (Can agents evaluate AI outputs more reliably than language models?). Structure helps, but every added step is one more place a busy practitioner might take a shortcut or something might break.

The field-level picture points the same way. Writing on interactive and agent evaluation argues that current practice is fragmented: teams adopt evaluation formats one piece at a time instead of following shared procedures (Should interactive evaluation be designed as a unified paradigm?). Old problems like comparability and reproducibility come back at the trajectory level instead of going away (Do interactive evaluations actually solve the benchmark comparison problem?). If shared standards are still missing at the field level, a specific three-step workflow is unlikely to be routine practice yet.

The takeaway is that the corpus contains a prescription and small studies, not data on adoption. What it does show is less obvious. The workflow's value may lie less in its order than in its stopping rule, a concrete answer to 'when have I looked closely enough?' Practitioners already do that loop informally, because criteria drift forces them to.