Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating LLM outputs. Yet LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation. We present a mixed-initiative approach to “validate the validators”— aligning LLM-generated evaluation functions (be it prompts or code) with human requirements. Our interface, EvalGen, provides automated assistance to users in generating evaluation criteria and implementing assertions. While generating candidate implementations (Python functions, LLM grader prompts), EvalGen asks humans to grade a subset of LLM outputs; this feedback is used to select implementations that better align with user grades. A qualitative study finds overall support for EvalGen but underscores the subjectivity and iterative process of alignment. In particular, we identify a phenomenon we dub criteria drift: users need criteria to grade outputs, but grading outputs helps users define criteria. What is more, some criteria appears dependent on the specific LLM outputs observed (rather than independent criteria that can be defined a priori), raising serious questions for approaches that assume the independence of evaluation from observation of model outputs. We present our interface and implementation details, a comparison of our algorithm with a baseline approach, and implications for the design of future LLM evaluation assistants.

Introduction. Large Language Models (LLMs) make mistakes—they hallucinate, ignore instructions, and generate outputs that require validation [25]. But validating the behavior of LLMs is challenging. In response, researchers and industry developers have created tools for prompt engineering and auditing that help people with testing outputs more systematically [1, 16, 23, 24, 27, 36, 41, 52]. Such approaches require metrics—a set of functions to automatically score LLM outputs, each typically an assertion with true or false values. These metrics increasingly include calls to “evaluator” LLMs (e.g., [1, 27, 52, 58]) that act as “judges,” grading outputs on qualities hard to articulate in code; for instance, the “conciseness” of an output. The problem is that, just like the LLMs they evaluate, LLMs that perform evaluations cannot be trusted. These “grader” prompts suffer from the same problems as any other prompt—they are unintuitively sensitive to seemingly minor changes in wording or structure [43]. Yet many existing systems do not include support for verifying the quality of LLM-generated evaluations, asking users to simply trust these outputs. How can users reap the efficiency benefits of LLM-assisted evaluation of LLM outputs, while minimizing or avoiding misalignment? How can we help users validate the validators? In this paper, we propose a mixed-initiative approach, EvalGen, to address this automated-evaluation alignment problem in the context of prompt engineering. Our approach streamlines the selection of metrics under practical constraints of user effort and latency. Specifically, an LLM suggests criteria in natural language, based on user context (e.g., the prompt under test), that the user can modify. An LLM then generates a pool of candidate assertions for each criterion—either code or LLM grader prompts that output “true” or “false.” While the user waits for the LLM to generate candidates, they are asked to grade outputs with a simple “good” (thumbs-up) or “bad” (thumbs-down) voting scheme. These grades then guide the automatic selection of assertions that optimize for alignment with user preferences (Section 4.2). After assertion selection, a final report card reveals the alignment between the chosen assertions and the user’s grades. Our approach generalizes beyond the particulars of our specific design, and could be extended to, for instance, update metric implementations with feedback from human preferences, or query the user for finer-grained individual grades. EvalGen is embedded inside an existing open-source interface for prompt engineering and auditing, ChainForge [1]. Our alignment algorithm adapts SPADE [45], a fully-automated algorithm for generating Python assertions from the revision history of a prompt. We performed an off-line verification of our human-guided alignment algorithm with SPADE as a baseline [45], then ran a qualitative user study with nine (9) industry practitioners that use

Related work. In response to the popularity of black-boxed LLMs like ChatGPT, prompt engineering (PE) has emerged as a new practice and research area. Alongside PE is the auditing of model behavior in practices such as “red-teaming,” used to identify harmful outputs in internal teams to tweak LLM behavior, usually prior to release [32, p.17]. These tasks have spurred the advent of new tools for “LLM operations” (hereafter called LLMOps) and new terminology such as “prompt template,” “chain of thought”, “chains,” etc.

Automating Evaluations of Prompts. When evaluating LLM behavior, users typically send off hundreds or thousands of queries to models. As users reach the limits of manual evaluation, users set up automated evaluation pipelines (Figure 1a) in code or with other LLMs (here we use the term LLM-based evaluators; other work uses terms such as “LLM-as-a-judge” [58] or “co-audit” [18]).1 Public PE tools like promptfoo [52] and ChainForge [1] allow users to write their own evaluation metrics to score LLM response quality, and support both code-based and LLM-based evaluators. For instance, in promptfoo users can write a rubric in a config file to specify how an LLM should evaluate responses, and may use pre-created grader prompt templates or customize them; an example is the assertion “The response is not apologetic.” Prototypes such as EvalLM [27] and PromptsRoyale [40] also support LLM evaluators, oftentimes exclusively, to help users compare between two prompts. Of PE tools, only EvalLM offers a way to help users calculate the alignment of LLM evaluators with their expectations, but this feature is mentioned only in authors’ Design section and is absent from their user study.2 At best, users of PE tools inspect LLM-generated evaluator outputs manually to double-check; at worst, the tool hides individual scores entirely. Regardless of aligning implementations of metrics with user preferences, even identifying what metrics to evaluate for custom tasks remains challenging for LLM practitioners, as evidenced by a recent study [37]. While many evaluation tools require users to declare metrics they care about, some prior work [45] and EvalGen employ LLMs to propose custom metrics based on prompts in the user’s LLM pipelines. Over-trust and Over-generalization of LLM Behavior. That tools provide little assistance to validate evaluator quality is alarming, considering that other research shows people tend to over-rely and over-trust AI systems [3, 28, 31, 50]. For instance, in one highprofile incident, researchers from MIT posted a pre-print on arXiv claiming that GPT-4 could ace the MIT EECS exam. Within hours, work by Chowdhuri et al. debunked the study [5], citing problems arising from over-reliance on GPT-4 to grade itself.

Method. In designing EvalGen, our goal was (1) to investigate how to assist developers in creating evaluators to grade LLM outputs, and (2) to help them “validate the validators” through both automated assistance and transparency around how aligned each evaluator is with their expectations. As we covered in Section 2, emerging practices in prompt engineering, LLM auditing, and prompt optimization involve the writing of evaluation functions (metrics) to automate grading. These functions may be code- or LLM-based. Based on this context, we set out to design an LLM-powered evaluation assistant that provided developers control over metric criteria, evaluator type (code or LLM), and implementation (i.e., function) generation and selection processes, without asking them to come up with criteria or write code or grader prompts themselves.

Finally, after the user is done grading and all candidate implementations are generated, executed, and filtered for alignment with grades, a Report Card screen appears with feedback on per-criteria and aggregate measures of alignment with user grades (Fig. 2e). Hovering over per-criteria metrics shows a confusion matrix of how aligned that particular criterion is to the human grades, while the aggregate metrics show the coverage and false failure rate (see Section 5) of the selected subset of EvalGen-generated assertions. The user then returns to the main ChainForge interface (not shown), where the selected implementations are available in a “Multi-Eval” node, titled by criteria. The user can edit or add more criteria, inspect and visualize evaluation results (Fig. 3), etc.; however, this is outside the scope of our design discussion. Our design reflects trade-offs between developer effort and robust human verification of LLM-generated metrics. The human cannot completely validate an LLM-based evaluator: the point of LLM evaluators is to reduce the effort required by the developer, who would otherwise have to grade outputs manually. The only way to fully align an LLM evaluator would be to ask the user to label all outputs; obviously, this defeats the purpose. Asking the developer to grade some outputs, using some time they would have spent waiting anyway, is the key idea behind our design.

Like prior work on evaluator assistants [27, 45], our solution decomposes evaluations into criteria and assertions (boolean functions that implement the criteria, evaluating outputs). We employ LLMs in generating criteria, based on the prompt [45], and in generating various candidate implementations of each criterion [27, 45]. As users grade, we rank candidate assertions that implement each criteria based on their alignment with user grades (see Appendix A.3 for how we define alignment). At a high level, alignment is a combination of the assertion’s coverage, or ability to fail outputs that the user thinks are bad, and the false failure rate, or ability to not fail outputs that the user thinks are good. We give a formal definition of assertion alignment in Appendix A.3. EvalGen’s architecture differs from prior work in two main components: first, EvalGen solicits grades from the user on a sample of LLM outputs—requiring some policy to sample LLM outputs to grade. Second, in contrast to SPADE [45], which operates offline and solves an integer linear program to generate the optimal assertion set, EvalGen employs an online (i.e., streaming) system architecture to progressively optimize for the most aligned assertion set. When EvalGen generates a new candidate implementation, it immediately executes this implementation on the set of LLM outputs, culling implementations that are obviously bad (e.g., Python functions with runtime errors). EvalGen maintains a dynamic estimate of selectivity (i.e., pass rate) for each candidate assertion, which in turn informs how grades are sampled in the interface. Our system, as depicted in Figure 1b, is structured into three components: Criteria Suggestion. We use an GPT-4 to propose various binary evaluation criteria in natural language, such as response length or tone. Developers can select from these suggestions or add their own criteria.

Discussion. Here, we unpack criteria drift and how it affects alignment, discuss the implications of our findings for future LLMOps evaluation assistants, and outline some open questions.

8.1 Implications of Criteria Drift for LLM Evaluation Assistants The practice of benchmarks in ML and NLP presume a world of well-defined criteria (and well-labeled data) on which to judge imperfect in a judge. We should encourage judges to believe and say: This is the best I can do now; it doesn’t solve all the problems, but it’s a start, and I’ll keep thinking” [17, p.1027]. This raises a deeper epistemic question for evaluation assistants—is “alignment” an actualizable goal? To what extent does our common terminology and assumptions—e.g., that there is a “ground truth” set of labels we merely need to elicit—fail us? Is validating the validators only ever a work-in-progress?

8.2 Operationalizing Assertions Participants expressed the desire to deploy their assertions in production. Some wanted to deploy assertions in the critical path of the LLM pipeline, to catch bad outputs before they are shown to end-users. Others wanted a more passive deployment—one participant requested for some form of report sent to their email inbox daily, containing results of the assertion set executed on a sample of LLM outputs from that day. In practice, assertions have different operational requirements. For example, if an assertion runs in the critical path, it should not result in many false failures, otherwise the LLM may get unnecessarily reprompted (i.e., rerun) and the overall latency might increase for our LLM pipeline. Moreover, as criteria changes over time, evaluation assistants should ask users to grade new LLM outputs observed in production and automatically adapt assertion sets. Our study confirmed the dual necessity for both code-based and LLM-based assertions among participants, but further showed that participants felt each type required distinct treatment, both in implementation selection and in how they perceive alignment. Often useful for sanity checks like output structure, code-based assertions can be used in the critical path of the LLM pipeline. In fact, a number of LLMOps tools exist for users to implement such code-based guardrails [19, 22, 42]. However, our participants highlighted the challenge of finding the right implementation for a given assertion—a decision that can depend intricately on the characteristics of LLM outputs. For example, determining the acceptable length of a response might vary significantly based on the observed output distribution. In fact, specifically for an output length criterion, P3 mentioned that they would look at a histogram of word lengths for a batch of outputs and see if there were some buckets of anomalous word lengths that exhibited other failure modes (e.g., a “rambling” response from the LLM). Some participants mentioned that they wanted their collaborators, such as product managers, to grade outputs in EvalGen. Importantly, when allowing multiple users to collaborate on grading, evaluation assistants have to consider inter-rater reliability and handle disagreements, if any. Again, we can draw inspiration from techniques from the crowdsourcing literature: for example, evaluation assistants could model each individual grader’s accuracy and apply corrections if necessary [59]. Moreover, collaborators may have different experience levels with coding: those with less experience may choose LLM-based assertions for criteria that might be better evaluated with code. While EvalGen provides initial suggestions for whether criteria should be evaluated with code or another LLM, evaluation assistants may want to make sure that all users from the same team “agree” on the right implementations.

Conclusion. This work presented EvalGen, a mixed-initative approach to aligning LLM-generated evaluation functions with human preferences. EvalGen assists users in developing both criteria acceptable LLM outputs and developing functions to check these standards, ensuring evaluations reflect the users’ own grading standards. In a qualitative study with 9 expert users, we observed a pattern we call criteria drift, where users refine their evaluation standards as they grade more LLM outputs. Recognizing the dependency of criteria on LLM outputs highlights new directions for designing future evaluation assistants.

Limitations. 8.3 Future Work and Limitations Assertions serve as a straightforward mechanism for evaluating LLM outputs, yet the potential of evaluation assistants extends beyond EvalGen’s supported binary judgments. Consider the example of word count: rather than setting rigid thresholds, one might find it more useful to monitor variations in word count across different LLM outputs. Like in traditional ML monitoring [44], there is, of course, still imprecision in what word count counts as “bad,” given its distribution. Does the AI assistant’s definition of bad align with the user’s? Tracking finer-grained information in evaluations, beyond simple true/false conditions, can aid in debugging issues within LLM pipelines, once a user knows the output is bad. Second, many users write their LLM pipelines as chains of multiple LLM calls [1, 11, 14, 53]. Evaluation assistants can facilitate end-toend alignment, which can be complicated not only with chains of LLM calls, but also in compound AI systems that leverage other components like retrieval-augmented generation [55]. Increasingly, user want their prompts to automatically improve based on assertion results. For instance, if an assertion fails for a group of LLM outputs, one might want to add an instruction to the prompt to solve this failure mode.