How can designers prototype real-time multimodal AI agents?

A Research through Design case study of building SocraBot — a proactive, multimodal real-time support agent built on GPT-4o for mechanical design tasks — reports that "existing prototyping and evaluation methods were insufficient to anticipate the nuanced situational complexity... and the contextual immediacy... required" for multimodal, real-time GenAI agents. The paper names five specific obstacles: continuous time-sensitive interactions that must be answered "within seconds rather than at discrete turns"; multiple input formats (speech, screen actions, video frames) that must be integrated coherently; highly variable user contexts; prompt and context engineering that "remains largely trial-and-error"; and prompt evaluation, since "benchmarks typically don't exist for such novel and domain-specific tasks" and existing ad-hoc evaluation tools are "text-based only."

To close that gap the authors combined two methods. Counterfactual Video Replay Prompting lets a designer "replay recorded user sessions and generate new AI messages at any moment," testing alternative agent responses against the same recorded context. Hybrid Wizard-of-Oz has a designer act as a human wizard mediating between users and LLMs during live sessions, so prompts can be adjusted in situ rather than only after the fact. Paired with live user-session data, the two methods let the team "iteratively refine model instructions, decompose prompts into chains of deliberate tasks, and evaluate SocraBot's behavior in both replayed and live contexts." To document the process, the team adopted "Bayazit's three-stage framework": unstructured knowledge elicitation (49 git commits of code and prompt changes, plus screen, webcam, voice and wizard-interface recordings), thematic analysis after the project ended, and finding validation through recurring team review of the emerging themes.

The paper frames this as filling a gap that benchmark-style evaluation cannot reach for novel, domain-specific tasks, which complements rather than duplicates Do generative UI tools actually implement their stated design rationales?: that paper measures, after the fact, how far a generated interface's stated design rationale diverges from what five commercial tools actually built across 24 fixed tasks, while this case study argues no such fixed task or benchmark exists yet for proactive, real-time multimodal agents, so evaluation has to happen experientially, through replay and live mediation, rather than by scoring a finished artifact against a rubric. It also puts an empirical face on the difficulty of operationalizing guidance like How should AI systems be designed across different interaction phases?: SocraBot's design goals — reflective questions, planning support, and proactive messages "at opportune moments" — read as an attempt to implement exactly the during interaction and timing concerns those guidelines name, yet the authors still "struggled... to translate this understanding into effective prompting strategies," even starting from Gmeiner et al.'s formative findings on what support designers need.

The paper is explicit about its limits: it is "a single case study in a specific domain" (CAD-based mechanical design with GPT-4o), so generalizability to other GenAI applications is not established, and both counterfactual replay and hybrid Wizard-of-Oz "face scalability challenges due to their reliance on human oversight, making large-scale simultaneous evaluations impractical." It is also an arXiv preprint rather than a peer-reviewed publication, and it reports the authors' own documented process rather than a controlled comparison against the text-based tools it describes as insufficient. What it supports, at the strength of a single documented build, is that proactive, multimodal, real-time GenAI agents may need experiential, human-mediated evaluation methods where no benchmark or fixed success criterion yet exists — not that such methods outperform benchmark-style evaluation wherever one is already available.