How does GenAI's unpredictability break traditional lab study methods?

A reflective multi-case study of four lab-based user studies with GenAI-integrated prototypes — spanning "LLM-based conversational in-car assistants and GenAI image tools for professional design workflows" — finds that GenAI's "non-deterministic" outputs destabilize the core assumptions of controlled HCI lab studies: "control, consistency, and comparability." The researchers derive five methodological challenges: "(C1) amplified reliance on familiar interaction patterns, (C2) amplified trade-offs between fidelity and experimental control, (C3) redefined feedback loops and user trust, (C4) new methodological gaps in usability evaluation, and (C5) amplified interpretive ambiguity between interface and system behavior." The authors are explicit that these are not entirely novel problems — similar issues "have also been discussed in evaluations of other adaptive or intelligent systems, including speech recognition and recommendation systems" — but ones GenAI's higher output variability amplifies and reframes, and in the case of C4, introduces outright.

The method itself departs from typical GenAI evaluation papers: rather than measuring task performance or satisfaction, the researchers performed a reflective, inductive thematic analysis of their own "study plans, interview guides, prototype specifications, and observational notes" across the four studies (conducted January 2024–September 2025), using affinity diagramming to surface methodological tensions before formal theme synthesis. From this they propose five guidelines (G1–G5) tied one-to-one to the five challenges, yielding "eighteen practice-oriented recommendations" — including "reframing onboarding to help participants manage unpredictability," "extending evaluation with constructs such as trust and intent alignment," and "logging system events, including hallucinations and latency, to support transparent analysis." The reasoning behind the guidelines is that non-determinism should not be engineered away to restore lab control, but studied on its own terms: "researchers should examine whether systems support exploration and recovery, and how users adapt interaction patterns over multiple steps to reach their goals."

The pairing of a GenAI-caused problem taxonomy with a matched set of recommendations echoes Does generative AI productivity actually improve or decline in practice? and Can Human Factors design fixes restore GenAI productivity losses?, though that pair diagnoses productivity loss during use, while this source diagnoses distortions in how GenAI use gets studied in the first place — a methodological problem one step removed from the usage-time problems those notes address. C2's fidelity–control trade-off also parallels the three-axis trade-off framework in Do generated analysis UIs really work better than chat?, but this source locates the trade-off in the experimenter's study design rather than in the end-user interface. Where How should generative UI be organized as a design space? maps design choices for building generative UIs, this source maps methodological choices for evaluating them — a complementary axis of the same design space rather than a competing one.

The four cases are described by the researchers themselves as "illustrative rather than exhaustive," drawn from only two domains, and conducted by the same team evaluating its own prototypes — so the five challenges are not established as a complete or generalizable taxonomy of GenAI evaluation problems, and the excerpt does not report whether applying the eighteen recommendations in practice measurably improved study outcomes; the guidelines are proposed, not tested against a baseline. The implication, held at that strength, is that lab-based usability findings for GenAI systems reported without comparable safeguards risk conflating stochastic model behavior with interface design flaws.