Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz

Recent advancements in multimodal generative AI (GenAI) enable the creation of personal context-aware real-time agents that, for example, can augment user workflows by following their on-screen activities and providing contextual assistance. However, prototyping such experiences is challenging, especially when supporting people with domain-specific tasks using real-time inputs such as speech and screen recordings. While prototyping an LLM-based proactive support agent system, we found that existing prototyping and evaluation methods were insufficient to anticipate the nuanced situational complexity and contextual immediacy required. To overcome these challenges, we explored a novel user-centered prototyping approach that combines counterfactual video replay prompting and hybrid Wizard of Oz methods to iteratively design and refine agent behaviors. This paper discusses our prototyping experiences, highlighting successes and limitations, and offers a practical guide and an open-source toolkit for UX designers, HCI researchers, and AI toolmakers to build more user-centered and context-aware multimodal agents.

Introduction. Recent advancements in Generative AI (GenAI) based on Large- Language Models (LLMs) and Visual Language Models (VLMs) offer unprecedented capabilities to build multimodal interactive systems with natural user interfaces that combine speech and screen-based content, such as video streams of work tasks. These systems can augment user workflows in novel and intuitive ways by providing contextual support that is sensitive to what users are doing in real time. Such visions build on a long history of research in mixedinitiative interfaces [24, 25] that sought to help users by observing their activities and offering timely assistance.

However, prototyping such applications involving pre-trained prompt-based multimodal GenAI foundation models, such as VLMs, is challenging due to their distinct qualities:

(1) Task generality: While models are trained to be broadly capable across many tasks, their domain-specific capabilities often remain opaque without clear task boundaries [34]. (2) Flexible input/output formats: The same model may accept text, images, or video and return multiple possible output forms, making it possible to structure the same task in different ways [46]. (3) Stochastic black-box nature: Their internal stochastic reasoning processes remain opaque, making behaviors difficult to anticipate [8, 46, 60].

In recent years, the HCI and AI communities have explored various methods to help researchers and practitioners prototype AI systems, including tools for prompt engineering [60], prompt chaining [52], and interactive evaluation [4, 27]. Benchmarking has also become a common evaluation strategy, where models are tested against large, standardized datasets to compare performance across tasks [11, 22, 51]. While benchmarks provide structured ways to assess model performance, they are limited to established domains with clear success criteria. For novel, domain-specific, and timesensitive interactive tasks, such benchmarks simply do not exist. As a result, prototyping prompt-based GenAI systems in early stages of development remains a trial-and-error process involving high uncertainty [59]. Moreover, currently existing interactive ad-hoc prototyping methods focus on text-based applications and do not support multimodal models involving continuous speech and video streams [4, 27]. While emerging multimodal models open new possibilities for HCI researchers, designers, and developers, prototyping such real-time systems in a user-centered way is especially challenging due to:

(1) Continuous, time-sensitive interactions: Systems must proactively respond to unfolding user actions within seconds rather than at discrete turns, requiring tight temporal alignment with user workflows. (2) Multiple input formats: Designers must manage heterogeneous modalities such as continuous speech, on-screen actions, and video frames, and integrate them coherently as model inputs. (3) Highly variable user contexts: Real-world tasks involve shifting goals, strategies, and representations, making it difficult to anticipate all situations during prototyping. (4) Prompt and context engineering: For complex and novel tasks, prompts (model instructions) require extensive refinement and testing to achieve desired behaviors. This process—often called prompt or context engineering—remains largely trial-and-error. (5) Prompt evaluation: Benchmarks typically don’t exist for such novel and domain-specific tasks, and also demand large datasets and narrowly defined tasks. Existing ad-hoc evaluation tools [4, 27] are text-based only and thus illsuited for proactive, multimodal systems in early stages of development.

Motivated by these challenges, we present a case study of prototyping SocraBot, a proactive real-time agent support system built atop a multimodal LLM (GPT-4o). SocraBot assists designers in mechanical design tasks by providing reflective questions and suggestions to help them think through their process more thoroughly. In building SocraBot, we found that existing prototyping and evaluation methods were insufficient to anticipate the nuanced situational complexity (e.g., when users quickly shift between planning, creating, and evaluating their work) and the contextual immediacy (e.g., the need for support to adapt within seconds of a new action) required in real-time multimodal interaction. To address these issues, we developed a novel prototyping pipeline combining two complementary methods: Counterfactual Video Replay Prompting, which allows designers to replay recorded user sessions and generate new AI messages at any moment, and a Hybrid Wizard-of-Oz method, in which a designer acts as a human wizard mediating between users and LLMs during live sessions with users. These methods, paired with data from live user sessions, enabled us to iteratively refine model instructions, decompose prompts into chains of deliberate tasks, and evaluate SocraBot’s behavior in both replayed and live contexts.

Related work. The increasing complexity of AI systems challenges traditional prototyping approaches. Interaction design relies on iterative processes where designers create externalized concepts and ideas to explore alternatives and refine experiences prior to implementation [31]. In doing so, prototyping interactive user experiences typically follows a sequence of activities starting with rough lowfidelity sketches toward functional implementations that closely mirror final products [7, 44]. However, AI-powered systems often resist these conventional approaches for several key reasons. First, designers often struggle to develop accurate mental models of AI capabilities and limitations [13, 30]. This creates a fundamental disconnect between designer expectations and actual system behavior. Second, implementing even basic AI functionality traditionally required specialized technical knowledge and substantial development resources—expertise and time that most design teams cannot readily access [55]. To articulate these challenges, HCI researchers have introduced the notion of “AI as a design material” [13, 54, 56]. Unlike conventional materials, AI exhibits probabilistic behavior (outputs vary even for identical inputs), emergent properties (e.g., unexpected capabilities outside of trained task scenarios), and rapidly evolving capabilities (frequent model updates that may alter system behavior). These characteristics complicate traditional prototyping logics and demand methods that embrace uncertainty and change rather than seek to eliminate them. Against this backdrop, the emergence of pre-trained foundation models, such as Large Language Models (LLMs), represents a significant shift. By enabling designers to generate sophisticated AI behaviors through natural language instructions (prompts), LLMs have democratized AI prototyping, removing many technical barriers that previously limited meaningful exploration to those with specialized expertise [6, 34]. This democratization has expanded further with multimodal Vision Language Models (VLMs) like GPT- 4o [37] and Claude-3 [3], which integrate text, images, and video inputs within unified model architectures. Designers can now prototype experiences that seamlessly blend multiple modalities without requiring expertise across multiple specialized AI systems. However, while prompt-based prototyping has simplified the creation of functional GenAI prototypes, this ease shifts design challenges toward prompt and context engineering—the systematic process of crafting and refining input instructions and input data to achieve consistent behaviors across diverse contexts [12, 21, 34, 45].

Method. To investigate the methodological complexities that emerge when prototyping and evaluating multimodal VLM-powered real-time experiences, we adopted a Research through Design (RtD) approach [18, 63], which recognizes that design knowledge emerges through direct engagement with situated design challenges. Thus, we conducted our methodological explorations in the context of a specific project and goal: implementing SocraBot, a multimodal real-time assistant a top of the prompt-based pre-trained VLM GPT-4o. Rather than starting from fixed hypotheses, we immersed ourselves in the concrete design challenges presented by this project, allowing insights to emerge organically through cycles of creation and critical reflection. This approach enabled us to generate procedural, conceptual, and pragmatic insights that can inform HCI researchers and practitioners who seek to prototype or study similar multimodal agentic systems, offering transferable design knowledge about methods for creating real-time, proactive, and situated AI support systems [16]. To support methodological transparency in documenting our design process and reasoning, we adopted Bayazit’s three-stage framework [40]:

(1) Knowledge elicitation in an unstructured and unanalyzed form. We used Git’s version control commit history to document all our code and LLM instruction (prompt) changes through granular commits (49 commits in total) for later qualitative data analysis [49]; During all prototyping sessions with users, we recorded the screen, webcam, voice, and system interaction data of participants alongside data from our "wizard" operator interface (screen recording, webcam, system log files). (2) Data analysis and interpretation. After the project ended, we performed a thematic analysis of the data collected to identify key design decisions in response to emerging challenges and reflection-in-action that happened during the design process. (3) Finding validation. Throughout the project, the research team held regular meetings to review ongoing design challenges and emerging insights. After the thematic analysis, we revisited these themes collaboratively to cross-check interpretations and ensure agreement among team members.

Our goal was to design and implement a proactive cognitive realtime support agent that can assist users in working on a mechanical design task in an AI-assisted 3D CAD workflow (Designing an engine mounting bracket using Autodesk Fusion360 Generative Design). Building on findings from Gmeiner et al.’s formative study on metacognitive support agents—which showed that agent-facilitated metacognitive scaffolding can help designers overcome cognitive challenges when working with GenAI tools [17]—we designed SocraBot as a situated instantiation of these ideas. The aim of the SocraBot support agent is to follow along with the user’s screen actions and verbalizations and to help the user better think through the design task by (1) asking reflective questions, (2) offering sketching and planning support, and (3) providing suggestions for design strategies and software operation.

3.1.1 UX Design and Implementation Goals. SocraBot’s design goals are directly informed by Gmeiner et al.’s formative Wizard-of-Oz study [17], which found that asking (the right) reflective questions and supporting planning and sketching activities has a positive effect on helping designers create more feasible designs compared to non-supported users. Based on insights from this previous study, we derived the following design goals for SocraBot:

(1) SocraBot should have real-time access to the users’ screen and speech and follow designers’ verbalizations and screen actions closely; (2) SocraBot should possess (non-exhaustive) knowledge of additive manufacturing and generative design tasks; (3) SocraBot should proactively send messages to the user at opportune moments.

Discussion. While prompt-based approaches have significantly reduced the effort needed to build GenAI prototypes, creating sophisticated user experiences with these models remains a substantial challenge. This work documents our first-hand experience implementing SocraBot, a proactive cognitive real-time support agent built atop a multimodal VLM, revealing the complex challenges that emerge during this process. Drawing on prior formative work by Gmeiner et al. [17], which provided a comprehensive understanding of designers’ support needs and articulated concrete design principles, we nevertheless struggled (like many researchers and practitioners in other task domains, e.g., [21, 59]) to translate this understanding into effective prompting strategies. The core difficulty stemmed from a fundamental uncertainty: For such a domain-specific task without existing benchmarks, we had no reliable way to evaluate whether our prompting approach would deliver the intended experience when confronted with the nuanced complexity of real-world and real-time user interactions. Our initial reliance on semi-automated tabular evaluations proved inadequate for capturing the contextual richness of such time-based multimodal interactions. This uncertainty drove us to develop a custom prototyping and evaluation pipeline combining Counterfactual Replay Prompting with a Hybrid Wizard-of-Oz approach. These methods enabled us to refine SocraBot’s prompting structure iteratively while maintaining a direct connection to authentic user contexts. By immersing ourselves in realistic usage scenarios, we could directly observe how different prompting strategies affected system behavior across varying situations—insights that traditional evaluation methods would have failed to surface at this early prototyping stage. In the following sections, we discuss the implications of these novel approaches for prototyping multimodal real-time GenAI agent experiences. We examine how they address complexity challenges in prompt engineering, complement existing prototyping methods, and support a more designerly approach to creating GenAI-based systems that meet user needs in complex, real-world contexts.

6.1 Taming Complexity with Iterative Prompt Decomposition Through Interactive Human-in-the-loop Steering and Evaluation This case study highlights how evaluating GenAI model performance for novel multimodal tasks without established test data and benchmarks feels like navigating blindfolded. In these cases, practitioners typically resort to iterative trial-and-error testing when working with prompt-based models [60]. Currently existing ad-hoc prompt evaluation tools largely focus on text generation [4, 27], falling short in handling the multimodal and time-based data needed for evaluating proactive real-time systems like SocraBot. This limitation becomes more evident when working with emerging multimodal capabilities that combine visual, verbal, and contextual interactions. Overall, our research demonstrates how interactive and experiential evaluation methods such as Counterfactual Replay Prompting and Hybrid Wizard-of-Oz bridge the gap between blind trial-and-error and rigid automated benchmark testing. These approaches fostered deeper problem understanding early in the development process by letting us observe how model outputs played out in realistic task contexts. Counterfactual replays, for example, revealed how suggestions that looked linguistically sound in a table often failed when judged against the unfolding user activity (Section 4.2).

Conclusion. Recent advances in multimodal GenAI open up possibilities for building proactive, context-aware agents that can follow users’ activities and provide real-time assistance. Yet, prototyping such systems remains difficult, as existing prompt engineering and evaluation methods fall short in addressing the situational complexity and immediacy of real-world workflows. Through our case study of building SocraBot, we developed a user-centered prototyping pipeline comprising Counterfactual Video Replay Prompting and a Hybrid Wizard-of-Oz approach. These complementary methods allowed us to iteratively decompose and refine prompts, evaluate model outputs across both recorded and live contexts, and better align system behaviors with users’ unfolding activities. Our experiences highlight how counterfactual replay and hybrid Wizard-of-Oz strategies can help HCI researchers and practitioners tame the uncertainty of early-stage GenAI prototyping by grounding evaluation in experiential, context-rich user data. Beyond our specific case study, we contribute the open-source Counterfactual Replay Toolkit, enabling others to rapidly test and adapt multimodal prompting strategies for their own domains. Together, these contributions open new pathways for building and evaluating proactive, contextaware, real-time AI agents, and point to future opportunities for developing more robust methods and infrastructures that support the iterative, user-centered prototyping of emerging GenAI systems. Looking ahead, we see this work as a step toward an expanded methodological repertoire for HCI+AI that enables researchers and practitioners to more effectively study and shape the evolving role of GenAI real-time agents in real-world human workflows.

Limitations. 6.4 Limitations While the methods described in this work offer promising advantages for prototyping multimodal GenAI experiences, several limitations stand out. First, this research centers on a single case study in a specific domain, and further research is required to validate its generalizability across other GenAI applications. Second, these approaches face scalability challenges due to their reliance on human oversight, making large-scale simultaneous evaluations impractical.