Evalet: Evaluating Large Language Models through Functional Fragmentation

Abstract Practitioners increasingly rely on Large Language Models (LLMs) to evaluate generative AI outputs through "LLM-as-a-Judge" approaches. However, these methods produce holistic scores that obscure which specific elements influenced the assessments. We propose functional fragmentation, a method that dissects each output into key fragments and interprets the rhetoric functions that each fragment serves relative to evaluation criteria—surfacing the elements of interest and revealing how they fulfill or hinder user goals. We instantiate this approach in Evalet, an interactive system that visualizes fragment-level functions across many outputs to support inspection, rating, and comparison of evaluations. A user study (N=10) found that, while practitioners struggled to validate holistic scores, our approach helped them identify 48% more evaluation misalignments. This helped them calibrate trust in LLM evaluations and rely on them to find more actionable issues in model outputs. Our work shifts LLM evaluation from quantitative scores toward qualitative, fine-grained analysis of model behavior.

Introduction. Large Language Models (LLMs) have enabled practitioners (e.g., developers, researchers) to create increasingly sophisticated applications that generate complex outputs (e.g., stories [16, 100], research papers [52, 80], and reasoning traces [29, 79]). Deploying these models safely requires rigorous verification that the outputs align [77] with practitioners’ intended goals. Evaluation is frequently manual as the applications are novel—lacking established benchmarks—and involve subjective aspects that require qualitative judgments [39], like how insightful or harmful the application’s outputs are. Identifying systemic and recurring issues requires reviewing hundreds of outputs, but the burden of manual inspection often leads practitioners to overgeneralize from small samples [5, 39, 76, 84]. To address this, practitioners have begun employing LLM-based evaluators (i.e., LLM-as-a-Judge [104]), where one LLM evaluates another’s outputs. By describing multiple criteria (e.g., Insightfulness , Harmlessness ) in natural language, practitioners can assess the alignment of the model outputs with their various goals [20, 35, 50, 105]. Current LLM-as-a-Judge approaches use holistic scores, where an entire output is summarized into numeric ratings (e.g., 3 out of 5) for each criterion. Holistic scores help practitioners quickly assess overall performance [39] but obscure the specific elements in the outputs that led to these assessments. For example, in Figure 1, an LLM explaining “T cells” to a young child received a moderate score for the criterion Age Appropriateness . To understand this rating, users must manually review the output to notice that while it uses simple vocabulary, it also employs potentially harmful war imagery. This manual process provides necessary insights but undermines the automation benefits. While some LLM evaluators provide brief justifications, practitioners must still map the justification to the specific fragments in the output [39]. This lack of detail or granularity becomes more critical at scale. When multiple outputs receive identical scores, practitioners have to read the justifications for each output’s evaluation to determine whether they share the same issues or different ones. Ultimately, the opaqueness of holistic scores inhibits practitioners from identifying systemic failure patterns in the outputs that require urgent attention [12, 73], and validating the accuracy and consistency of the LLM evaluator’s judgments [21].

To address these challenges, we propose functional fragmentation (Fig. 1): a novel LLM-based evaluation method that dissects each output into key fragments and interprets the functions of each fragment, where each fragment may serve multiple functions. With functions, we refer to the rhetorical roles or purposes that text fragments serve that are relevant to a given evaluation criterion. In Figure 1, the fragment describing T cells as “shooting their special microscopic guns” serves the function of “personification” for the criterion Engagement , but also “war-related imagery” for Age Appropriateness . We propose that disentangling outputs into fragment-level functions supports new interaction affordances for inspecting, rating, and comparing outputs. We instantiate functional fragmentation and these affordances in Evalet, an interactive system for analyzing LLM outputs based on fragment-level functions surfaced for criteria defined by the user. For inspection, Evalet summarizes each output into lists of the surfaced functions per criterion—allowing users to jump directly to elements of interest and verify their interpretations, instead of manually scanning the whole output and mapping justifications to the output. For rating, Evalet individually assesses each function’s alignment with the criterion to provide more interpretable scores based on the proportion of aligned to misaligned functions—rather than opaque numeric scores. Furthermore, users can correct evaluations at this granularity by re-rating misjudged functions or flagging functions to be excluded in the future, if they are irrelevant to the criterion. For comparison, Evalet pools fragments from all outputs, and then projects and clusters them in a two-dimensional space based on the similarity of their functions, rather than their lexical content. Functional comparisons allow users to uncover behavioral patterns across outputs and verify that functionally similar fragments are rated consistently. For example, in Figure 1, fragments with different wording (e.g., “shooting [...] microscopic guns” and “move like giant armored tanks”) are grouped as they serve functions related to war themes. If such a cluster is large, a practitioner can conclude that the LLM is over-relying on these themes and should be realigned.

Related work. This work aims to support interactive evaluation of LLMs through sensemaking of model outputs and evaluations at scale. To this end, we review literature in (1) interactive examination of machine learning (ML) models, (2) interactive testing and evaluation of LLMs, and (3) sensemaking of text at scale.

2.1 Interactive Examination of Machine Learning Traditionally, Machine Learning (ML) models are examined and understood by evaluating on benchmarks with automated metrics that aggregate performance into a single statistic or score. However, this provides limited understanding into how the model behaves, what its flaws are, and what are the areas for improvement [60, 73, 102]. Instead of relying on aggregated metrics, prior work has introduced systems that support more interactive and fine-grained evaluation of models. For example, Polyjuice [95] and AdaTest [71] allow practitioners to iteratively evaluate models by creating challenging input data and testing how the models behave on these cases. Furthermore, researchers have proposed various tools [11, 74, 78, 92, 94] that help practitioners to unpack evaluations by identifying slices or subsets of data, and testing models on these to identify specific flaws or limitations. Complementing these evaluation approaches, a rich body of work in explainable AI (XAI) [46] has also explored how to support understanding of models through more fine-grained analysis of behaviors [40]. For instance, the foundational methods LIME [72] and SHAP [53] explain model predictions by inferring the effect of individual features. Frameworks such as the XAI Question Bank [45] organizes users’ understanding and explainability needs into fine-grained questions that explore diverse aspects of models (e.g., inputs, outputs, performance). Collectively, this work highlights the importance and need for more granular analysis of model performance. Our work extends this to the evaluation of LLMs: instead of simply slicing datasets, we slice the data points themselves into fragment-level functions to provide a more finegrained understanding of LLM outputs.

2.2 Interactive Testing and Evaluation of LLMs The general-purpose capabilities of LLMs have enabled novel AI applications but also made it harder to verify that they behave as intended. As these models are applied to new tasks and contexts, there are no benchmarks or metrics to automate evaluation [39] and, due to their near infinite input-output space, models have to be tested with numerous and diverse samples [48, 101].

Method. 3 Functional Fragmentation: An Evaluation Approach To evaluate the alignment of an LLM, practitioners must not only quantify quality through numeric scores but also qualitatively understand how this models’ outputs are composed and their characteristics [19, 23, 73]. While existing LLM-based evaluation methods can assess outputs across various criteria [39, 99, 104], they only provide holistic judgments (i.e., overall scores, justification) for each dimension. Thus, practitioners must manually inspect outputs to identify the specific elements that satisfy or violate their goals.

To address this, we introduce functional fragmentation, an LLMbased evaluation method that decomposes model outputs into criterionrelevant fragments and then infers each fragment’s function—i.e., the role or effect it serves that influences the output’s fulfillment of that criterion. Our approach draws inspiration from inductive coding [87] (i.e., interpreting raw data into codes based on higher-level themes) and rubric design [66] (i.e., inspecting artifacts to define quality aspects to review). In this section, we outline the novel affordances that are supported by this approach for inspecting, rating, and comparison of LLM outputs.

3.1 Inspect Fragment-Level. To make sense of existing LLM-based evaluations, practitioners must manually review outputs, connect them with the evaluator’s justifications, and interpret each fragment’s significance in terms of the criteria [39]. Our approach automatically extracts criteria-relevant fragments and interprets their functions with respect to each criterion—directly presenting practitioners with qualitative interpretations to inspect and verify. As one fragment can serve different functions under different criteria, our approach allows practitioners to examine the same content from multiple perspectives and identify trade-offs. Given that criteria are often subjective, our approach can also uncover meaningful functions that the practitioner may have not previously considered— similar to inductive coding [87]. Conversely, if the LLM evaluator extracts functions that the practitioner considers irrelevant to the criterion, practitioners can directly flag these to be ignored in future evaluations.

Output-Level. Beyond identifying each fragment-level function in an output, practitioners may also need to inspect how these functions appear together in the output. For example, when evaluating Tension in LLM-generated horror stories, a practitioner may need to understand how the LLM uses various functions to gradually build tension in the story. Traditionally, this would require the practitioner to read the whole story, but not all of the content may be directly related to that criterion. With functional fragmentation, each output can be summarized into a list of functions related to a given criterion, allowing practitioners to easily inspect each output by focusing only on the aspects of interest.

3.2 Rate Fragment-Level. By disentangling outputs into fragment-level functions, each individual function’s alignment with a criterion can be rated independently. More fine-grained evaluations can support interpretability by clearly highlighting the specific aspects of an output that are aligned or misaligned with a criterion. Instead of correcting the LLM evaluator by editing criteria descriptions, practitioners can directly re-rate specific functions to control future evaluations—similar to how educators develop rubrics by assessing examples of student work [66]. Beyond supporting practitioners, LLM evaluators are also more consistent when performing more fine-grained evaluations through checklists [47, 75] or rubrics [35, 36, 99]. Unlike these approaches, which rely on predefining these checklists and rubrics, our fragment-level functions are emergent, identified dynamically based on the outputs and criteria.

Output-Level. Instead of providing uninterpretable and opaque scores (e.g., 2 out of 5) for each model output, our approach enables us to rate each output based on its proportion of aligned and misaligned fragment-level functions (e.g., 75% of surfaced functions are aligned)1.

Discussion. In this paper, we present functional fragmentation, a novel approach for evaluating and interpreting LLM outputs based on their constituent fragment-level functions, and Evalet, an interactive system that instantiates this approach. In this section, we suggest guidelines for integrating both fragmented and holistic evaluations in practice, discuss how functional fragmentation supports more nuanced analysis with LLM-as-a-Judge, and propose the need for further work in supporting qualitative and interactive evaluation of AI.

8.1 Guidelines for Integrating Fragmented and Holistic Evaluations As revealed by our user study, both fragmented (i.e., at the fragment and function-level) and holistic (i.e., at the output-level) evaluations have distinct merits. In practice, we suggest that these types of evaluation are complementary and practitioners should employ them together through a layered workflow:

(1) Start with Fragmented Evaluation on Broad Criteria: As described by study participants, functional fragmentation can surface diverse fragment-level functions that should be considered within each criterion. Through this, practitioners can more comprehensively identify concrete aspects to evaluate for each criterion, including those that were not initially considered. (2) Iterate and Concretize Criteria with Function Examples: By exploring fragment-level functions, practitioners can refine their criteria by deciding on what functions the evaluator should continue to surface and how these should be assessed. Practitioners should iteratively refine their criteria and example sets while re-evaluating the outputs until most misalignments disappear and the function clusters stabilize (i.e., similar clusters are presented each evaluation). (3) Zoom Out and In: Then, practitioners should zoom out to inspect the whole outputs, and their holistic justifications and scores to understand the overall qualities and similarities of outputs. As the underlying fragment-level evaluations have been corrected and aligned, practitioners can more reliably depend on the signals from the holistic scores. At this stage, practitioners can dive deep back to the fragmented evaluations when they need more details (e.g., identify concrete issues in low scoring outputs or the patterns of those with mixed scores). This allows users to flexibly alternate between levels of abstraction [83].

To fully support this integrated workflow, Evalet must also capture the holistic attributes of outputs (e.g., tone, structure). While the system does not currently support this, this can be easily addressed by introducing an additional LLM module after the functional fragmentation step that extracts holistic attributes from the output that are related to each criterion. These attributes can be represented as sentence-level descriptions (e.g., "comedic tone")—the same format as the function labels of fragments. Then, these holistic attribute labels can also be compared and visualized with the other fragment-level functions, and also factored into the holistic scores.

8.2 Calibrating Trust in LLM-as-a-Judge through Verification LLM-as-a-Judge can facilitate inspection, assessment, and analysis of LLM outputs at scales that are infeasible through human effort alone. However, practitioners must carefully calibrate their trust by recognizing where they disagree with the LLM evaluator, where it hallucinates, and when it is inconsistent. Our study showed that failing to calibrate trust often led practitioners to dismiss LLM-based evaluations and revert to manual review—losing both efficiency benefits and potentially valuable insights.

Conclusion. Evalet: Evaluating Large Language Models through Functional Fragmentation CHI ’26, April 13–17, 2026, Barcelona, Spain levels, which can help summarize and decompose the vast space of functions for users. Evalet can also incorporate proactive agentic guidance [15, 68, 69] to identify and highlight significant clusters or functions that can help kickstart users’ analysis. • User Study - Fragmented vs. Holistic: Our user study compared exploration of fragmented evaluations against holistic evaluations to clearly isolate their distinct affordances. In practice, these two methods should be used together. While participants offered insights into how to combine them, further studies are needed to understand actual usage patterns.

Limitations. 8.5 Limitations Our work has several limitations:

• Function Priority: Our technical evaluation showed that our approach does not account for the relative importance of each fragment-level function. Future work could add priority ratings to functions (e.g., manually by the user or suggested by the system). • Real-World Practice and Deployment: Further research into real-world use of functional fragmentation and Evalet is required to understand how this evaluation method integrates into practitioners’ workflows. To facilitate this, we plan to release Evalet as open-source. • Controllability: Evalet supports control of the functional fragmentation process by allowing users to exclude or re-rate fragment-level functions. We initially included or considered additional controls (e.g., function relabeling, cluster editing, fragment-size editing), but pilot users found that these added cognitive overhead without clear benefit. Due to the time constraints in the user study, we opted for not including these features in the system used during the study. However, as seen from the findings, some participants reported needing more control (e.g., fragment size), suggesting that this feature is valuable in certain workflows or to certain users. In the open-source release of Evalet, we plan to introduce these controls as advanced settings for users.