Evaluating Generative AI in the Lab: Methodological Challenges and Guidelines

Generative AI (GenAI) systems are inherently non-deterministic, producing varied outputs even for identical inputs. While this variability is central to their appeal, it challenges established HCI evaluation practices that typically assume consistent and predictable system behavior. Designing controlled lab studies under such conditions therefore remains a key methodological challenge. We present a reflective multi-case analysis of four lab-based user studies with GenAI-integrated prototypes, spanning conversational in-car assistant systems and image generation tools for design workflows. Through cross-case reflection and thematic analysis across all study phases, we identify five methodological challenges and propose eighteen practice-oriented recommendations, organized into five guidelines. These challenges represent methodological constructs that are either amplified, redefined, or newly introduced by GenAI’s stochastic nature: (C1) reliance on familiar interaction patterns, (C2) fidelity–control trade-offs, (C3) feedback and trust, (C4) gaps in usability evaluation, and (C5) interpretive ambiguity between interface and system issues. Our guidelines address these challenges through strategies such as reframing onboarding to help participants manage unpredictability, extending evaluation with constructs such as trust and intent alignment, and logging system events, including hallucinations and latency, to support transparent analysis. This work contributes (1) a methodological reflection on how GenAI’s stochastic nature unsettles lab-based HCI evaluation and (2) eighteen recommendations that help researchers design more transparent, robust, and comparable studies of GenAI systems in controlled settings.

Introduction. Generative AI (GenAI) technologies are increasingly integrated into interactive systems across domains—from productivity tools to creative applications—by generating diverse forms of content such as text, images, and voice [1, 32, 55]. As these systems become more common in everyday contexts, evaluating their usability and user experience has become an important topic in HCI. However, GenAI also introduces challenges for established evaluation practices, particularly in controlled lab studies, which are among the core methods in HCI research [28, 49, 68]. Unlike rule-based systems, such as traditional chatbots or incar voice assistants, which produce fixed responses based on predefined commands, decision trees, or state machines [56, 59, 64], GenAI models generate open-ended and context-dependent outputs that can differ with each interaction [20, 36, 45, 46, 57, 75]. This non-determinism, while enabling new forms of generative interaction, also disrupts key methodological assumptions that underlie lab-based evaluation, such as control, consistency, and comparability [31, 44]. Without adapted approaches, researchers risk drawing misleading conclusions about usability, trust, or user behavior when stochastic system behavior is mistaken for interface design flaws. We do not claim that these challenges are unique to GenAI. Similar challenges can arise when evaluating other adaptive or intelligent systems [65, 67]. However, in generative systems, where output variability tends to be higher, these challenges are amplified and reframed, and additional challenges can emerge. Designing studies that remain transparent, robust, and comparable under such stochastic conditions is thus an important step toward more reliable GenAI evaluation. This unpredictability affects all stages of user studies, from task definition and prototype development to data collection and interpretation, yet how to plan and conduct such studies remains underexplored. Numerous GenAI studies focus on system performance or user-facing outcomes [30, 40, 55, 70, 74], while the methodological decisions and trade-offs behind evaluation design are rarely discussed. Some acknowledge issues such as hallucinations, shifting output styles, or novelty effects that influence trust and user experience [23, 73], yet these aspects are often mentioned only as study limitations rather than systematically examined through internal materials and reflections. Making such methodological reasoning explicit can help researchers anticipate challenges and design more rigorous evaluations of GenAI systems in controlled settings. To address these issues, we conducted a reflective multi-case study of four controlled lab-based user studies involving GenAIintegrated prototypes. The cases span two domains—LLM-based conversational in-car assistants and GenAI image tools for professional design workflows—and vary in fidelity, modality, and participant groups. We adopted a multi-case approach because methodological challenges manifest differently across systems and study designs, and a single case would not capture this diversity or reveal recurring patterns [18]. Our goal was not to report userfacing outcomes such as task performance or satisfaction measures, but to analyze the methodological decisions, adaptations, and tensions that emerged throughout the research process. Drawing on study materials, researcher notes, and team discussions, we reflected on how GenAI’s stochastic nature shaped study planning, prototyping, data collection, and analysis. Our work addresses two research questions:

RQ1 What recurring methodological challenges arise when evaluating GenAI systems in controlled lab settings? RQ2 How can these challenges be addressed in the design and execution of such studies?

Through cross-case reflection, affinity diagramming, and inductive thematic analysis [6], we identified five methodological challenges (C1–C5) that complicate established HCI evaluation practices. These challenges represent methodological constructs that are either amplified, redefined, or newly introduced by the generative and non-deterministic nature of GenAI systems: (C1) amplified reliance on familiar interaction patterns, (C2) amplified trade-offs between fidelity and experimental control, (C3) redefined feedback loops and user trust, (C4) new methodological gaps in usability evaluation, and (C5) amplified interpretive ambiguity between interface and system behavior. Building on these findings, we propose five methodological guidelines (G1–G5), each linked to one of the challenges, and eighteen practice-oriented recommendations that offer actionable strategies for designing, conducting, and analyzing GenAI user studies.

Related work. User studies are a foundational method in HCI for evaluating interactive systems [34, 36]. By typically combining quantitative measures (e.g., surveys, task logs) with qualitative feedback data, user studies help assess usability, user experience, and task performance across diverse interface types [21, 45, 53]. Traditional evaluations often rely on controlled lab experiments, measuring completion time, error rates, or subjective satisfaction, and are well-suited to systems with well-defined tasks and stable behavior [21, 34, 45].

As interactive systems become more adaptive, open-ended, and embedded in dynamic contexts, conventional evaluation methods such as usability testing and short-term lab studies often prove insufficient. Poppe et al. [43] emphasize that systems involving novel sensing technologies and shifting user/system initiative require longitudinal observations and context-sensitive evaluation. Similarly, Greenberg et al. [21] argue that standardized usability testing may hinder innovation or overlook the critical experiential dimensions, especially in systems supporting exploration, creativity, or collaboration. Brdnik et al. [9], in a review of IUI papers from 2012 to 2022, found that many evaluations still rely on conventional experiments and questionnaires, with limited attention to metrics suited for system adaptivity or the dynamics of human-AI co-adaption. Similar limitations have been observed in newer interaction paradigms such as voice interfaces, AR/VR, and IoT [12]. In response, the HCI community has adopted a range of complementary methods, including in-the-wild studies [51], longitudinal deployments [27], Wizard-of-Oz studies [29], simulation and modeling approaches [39], and more interpretive, mixed-method designs that combine usage data with reflective user feedback [14]. These approaches are often applied to systems that are complex, adaptive, or open-ended, or that are embedded in real-world settings where traditional lab evaluations may be insufficient. The information visualization community provides a documented and explicit example of methodological evolution in response to similar challenges. Visualization tools are often used for exploratory tasks like analyzing large datasets or generating insights where no single correct answer or predefined success criterion exists [33, 42]. In such contexts, traditional metrics like task completion time and error rates may misrepresent how users interact or derive value from the system over time.

Method. We adopted a multi-case study approach supported by reflective and inductive thematic analysis to examine methodological challenges in evaluating GenAI systems through lab-based user studies. This approach enabled us to capture methodological challenges across varied GenAI systems and study designs while reflecting on our research decisions. A multi-case perspective supported the identification of both recurring patterns and context-specific nuances across studies [22]. By combining Eisenhardt’s systematic multi-case framework [18] with affinity diagramming [4] for initial structuring and inductive thematic analysis [7, 8], we moved from concrete study observations to broader methodological insights. The resulting five challenges (C1–C5) represent recurring methodological tensions in lab-based GenAI evaluation and form the foundation for the guidelines presented in Section 5. In summary, this analytic process combined affinity diagramming for initial structuring with thematic analysis to synthesize cross-case methodological challenges.

We analyzed four user studies conducted between January 2024 and September 2025, covering both conversational and visual GenAI systems. We selected these cases to support cross-case comparison, following established guidance for multi-case research [18], rather than to maximize the number of challenges identified. Accordingly, our inclusion criteria were guided by methodological considerations and practical access constraints. We included only lab-based user studies that we conducted ourselves, involved direct user interaction with a GenAI system, and empirically evaluated that interaction with participants. Because all authors were involved in the studies, we had detailed insight into design rationale, trade-offs, and study-planning decisions, supporting retrospective methodological analysis. The cases varied in system type, participant group, and prototype fidelity—from in-car voice assistants to design-oriented image generation tools—providing sufficient diversity to examine recurring methodological challenges across contexts. This scale aligns with established guidance for multi-case research, which suggests that a small number of heterogeneous cases can provide depth and analytical comparability [18, 22]. As such, the resulting challenges should be understood as illustrative rather than exhaustive, highlighting recurring methodological tensions rather than providing a complete landscape of GenAI evaluation issues. We approached reflection as a method for methodological inquiry rather than subjective introspection, triangulating researcher memos, study logs, and artifacts to ensure transparency. Each case featured different combinations of GenAI models, interaction modalities, user goals, and study designs. This diversity offered a comparative basis for identifying recurring methodological tensions across distinct study contexts. An overview of the cases is provided in Table 1, with detailed descriptions in Section A.2.

We collected a range of internal research materials, including study plans, interview guides, prototype specifications, and observational notes from user sessions. Researcher reflections and memos written Our analysis followed an inductive thematic analysis applied across the four lab-based studies. Rather than starting from predefined categories, we iteratively developed themes grounded in the collected data. The analysis builds on materials generated through an initial affinity diagramming step, which supported the organization and externalization of methodological observations before formal thematic analysis. This process unfolded in three stages: (1) organizing methodological observations within each case, (2) collaboratively clustering and comparing patterns across cases, and (3) synthesizing broader methodological challenges emerging from the analysis [18].

Step 1: Organizing Case Observations. In the first stage, we organized all methodological observations and reflections according to study phases (e.g., research planning, prototyping, participant interaction, data collection, and analysis), supported by affinity diagramming.

Discussion. Our findings highlight challenges and recommendations that illustrate how GenAI systems are reshaping controlled lab studies and HCI evaluation. Below, we discuss how these shifts affect study design, user trust, and evaluation metrics, and we reflect on how a reflective multi-case approach helps surface emerging methodological challenges.

Our results extend long-standing HCI debates about fidelity, control, and trust. As in prior work on adaptive or novel-sensing systems that require context-sensitive and longitudinal observation [27, 43, 50], GenAI variability disrupts the tight input–output mappings assumed by many lab protocols. It also intensifies the fidelity–control trade-offs familiar from Wizard-of-Oz and prototyping research [29]. GenAI does not merely introduce nondeterminism that complicates study design; it can also generate new challenges while amplifying and reframing existing ones. Several tensions identified in our analysis—such as users reverting to familiar interaction strategies, difficulties interpreting system behavior, and trade-offs between control and realism—have also been discussed in evaluations of other adaptive or intelligent systems, including speech recognition and recommendation systems [11, 13, 62]. Our contribution is to articulate how the high degree of output variability in GenAI systems amplifies these tensions by coupling probabilistic generation with feedback ambiguity. For example, variability may support exploration in some contexts while undermining trust in others, including through hallucinations. We therefore frame these challenges as amplified or reframed methodological concerns, or issues that emerge specifically in the context of generative systems, rather than as entirely unprecedented problems. Challenges C1–C3 primarily stem from stochastic outputs and the lack of clear input–output mappings. Unpredictable feedback can discourage deviation from familiar strategies (C1) and disrupt fidelity–control trade-offs (C2) and feedback loops (C3), thereby undermining both user confidence and experimental control. Challenges C4–C5 arise from the opacity of system behavior and the entanglement of model and interface effects, complicating evaluation and prompting a reconsideration of how success and usability are defined. One could argue that non-determinism ultimately reflects training data and tunable randomness (e.g., temperature or sampling strategies). However, we contend that this unpredictability is precisely what makes these systems worth studying: variability enables open-ended dialogue, exploration of alternatives, and forms of collaboration that deterministic systems cannot support. Rather than suppressing variability, researchers should examine whether systems support exploration and recovery, and how users adapt interaction patterns over multiple steps to reach their goals. In considering new approaches to studying GenAI, we argue that methodological transparency is critical. We examined our own research process—including the decisions made, trade-offs encountered, and evolving strategies—and how these choices shaped study procedures and outcomes. Decisions such as constraining inputs, simulating outputs, or tolerating variability were not neutral in their effects; they influenced what could be observed and how findings were interpreted. We therefore recommend reflective case comparisons as a useful way to surface methodological insights for emerging technologies. Looking ahead, as models expand their context windows and adopt more agentic behaviors, additional challenges will likely require longitudinal and in-the-wild studies to understand how trust, adaptation, and success criteria evolve over time.

Conclusion. In this work, we identified five recurring methodological challenges (C1–C5) in evaluating GenAI systems in controlled lab settings through a reflective analysis of four case studies. These challenges reveal how GenAI’s stochastic, non-deterministic behavior complicates established assumptions in HCI evaluation. In doing so, they expose tensions between control and realism, between deterministic expectations and probabilistic system behavior, and between user interpretation and model variability. Building on these insights, we proposed five methodological guidelines (G1– G5) and eighteen practice-oriented recommendations to support researchers in designing, conducting, and analyzing GenAI user studies more effectively. These guidelines include preparing participants for unpredictable behavior, aligning prototype fidelity with study goals, improving feedback interpretability, adapting evaluation metrics to account for stochasticity, and incorporating flexibility and transparency into study design and analysis. Overall, our work foregrounds the research process itself—its design choices, trade-offs, and interpretive challenges—when studying generative systems. Rather than offering a final framework, we aim to provide a foundation for continued methodological reflection as GenAI technologies become more integrated into everyday life. Continually revisiting and expanding these methodological discussions will be essential for building reliable and transparent HCI research on GenAI-integrated systems.