"When to Hand Off, When to Work Together": Understanding Concurrent Human-Agent Interaction in Shared Co-Creative Workspaces

Shared co-creative workspaces allow users to contribute while agents execute. Yet how users adjust their participation as their work and the agent’s execution shape one another remains less understood. We conducted two design probe studies with professional designers. In Study 1 (N=10), participants identified opportunities to contribute by observing the agent’s progress, but avoided intervention when the agent misinterpreted their edits. Therefore, we developed Cleo, a second probe that tracks user actions and selectively updates its plan. In Study 2 (N=12), we analyzed 255 interaction turns and stimulated recall interviews, identifying five action categories, ten codes, six triggers, and four enabling factors. Participants engaged in concurrent interaction during 33.73% of turns, expressing newly recognized design ideas or preferences through edits and changing the division of work by completing pending subtasks or independently developing copies of unfinished outputs. We present a decision model and five design implications for concurrent human–agent collaboration.

Introduction. AI agents are increasingly capable of executing complex tasks autonomously [83, 84] and support workflows in design [55, 58, 76], programming [71], and web navigation [41]. As agents gain access to shared workspaces through emerging tool-calling infrastructures (e.g., the Model Context Protocol [5]) and their execution becomes visible to users, human–agent collaboration can extend beyond sequential delegation, where users specify goals and review outputs. This opens up opportunities for concurrent human–agent collaboration, where users and agents can work simultaneously in a shared workspace. This raises a coordination question: what interaction patterns emerge during concurrent human–agent co-creation, and what support does this collaboration require?

Prior work has examined user control during web navigation [36], editing plans and intermediate research outputs [22], collaboration without prescribed turns in writing and data analysis [75], and simultaneous human–AI drawing on a shared canvas [52]. However, less is understood about how users adjust their participation when their own work and the agent’s execution shape one another. A user’s edit can change what the agent should do next, just as the agent’s progress can change what the user intends to do. For example, users may edit an element to demonstrate a change for the agent to propagate, complete a pending subtask themselves, or reuse a partial output independently. To inform the design of agents that support concurrent collaboration, we examine when and why users make these contributions, and how they expect the agent to incorporate their edits, adjust the division of work, or preserve their separate work. To explore this, we conducted two design probe studies in a shared co-creative visual workspace, where designers can develop their ideas by directly editing and reusing parts of the agent’s output. We first built a process-transparent agent that executes design tasks in Figma through direct tool operations (e.g., creating frames, applying shadows), helping designers track the agent and coordinate accordingly [27, 28, 68]. In a single-session study with 10 professional designers (Study 1), participants used visible progress to evaluate the agent’s direction and identify opportunities to contribute. They wanted to reuse intermediate results and demonstrate intent through direct manipulation. However, when users edited the shared canvas while the agent was executing, the agent sometimes misinterpreted or reverted their modifications, creating coordination conflicts and discouraging further intervention (Figure 1). This reluctance to intervene limited us to observe the collaboration strategies participants wanted to pursue. These findings highlighted the need for collaborative context awareness on the agent’s side, so it could respond to users’ contributions in the ways they expected.

Related work. People follow their collaborators’ progress while considering when to help and when to continue their own work. Coordinating contributions involves managing turns in conversation [69], as well as considering how to share and integrate distributed work and the costs involved in collaborative learning [48]. In shared workspaces, gestures and changes to artifacts reveal others’ activities [68, 78], helping collaborators decide whether to build on that work or address another part first. Workspace awareness research describes the information supporting these decisions [20, 27, 28]; interfaces that convey gaze and gestures help remote collaborators coordinate their actions [39, 59]. These considerations also apply to human–agent collaboration. Users may take on parts of a delegated task or change its direction. Mixed-initiative principles emphasize combining direct manipulation and automation as circumstances demand [33], supported by easy correction [3] and agents’ responsiveness to changing needs [15]. Supporting users’ workflows therefore requires understanding when they work directly and when they delegate again. We analyze transitions among observation, direct work, and delegation, and the reasons behind them, to inform agent support in shared workspaces.

Research on human–agent collaboration has explored how users delegate tasks and steer agents’ work. LLM tool use and interleaved reasoning and action [72, 85] enable multistep delegation in programming [9, 42, 71], web tasks [18, 46, 88], scientific research and data analysis [26, 32], and general digital work [1]. To help users inspect and revise work, researchers have connected AI explanations to the task interface. Talk to the Hand points to relevant on-screen elements with a separate AI cursor and speech bubbles [64], while WaitGPT visualizes filtering, aggregation, and intermediate results to help users identify what to change [81]. Tools also incorporate users’ judgments into subsequent execution through editing and rewinding [21] or pausing to clarify ambiguous preferences [63]. Users can also participate directly in agents’ workflows. CowPilot and Orca let users take control of web tasks [36, 41]; CowPilot incorporates their action histories when execution resumes. Cocoa supports planning and assigning responsibilities before execution, as well as revising intermediate artifacts and subsequent plans during execution [22]. Collaborative Gym examines performance, initiative, collaborative behaviors, and failures in a shared-control environment without prescribed turns [75]. However, more opportunities to intervene do not always make collaboration easier.

Method. To explore how process visibility creates opportunities for cocreative interactions and reveals the coordination challenges it entails in creative workflow, we implemented an initial design probe and conducted an exploratory study with 10 designers.

3.1.1 Three Basic Components for Collaborative Agent Design. Existing design agents are largely opaque—they accept instructions and return text reasoning (or high-level sub-goals) and final outputs, giving users little visibility into execution. To make agent’s execution observable and collaborative, we grounded the probe’s design consideration in three core components drawn from human-human collaboration literature, where these principles have been shown to support coordination, mutual understanding, and fluid collaboration, reasoning that they could equally scaffold more natural human-agent collaboration: (1) Interaction Modality for peer-like communication, (2) Process Transparency for observable agent work, and (3) Presence Awareness for mutual attention visualization. (1) Interaction Modality: Voice Interaction with Flexible Object Referencing. The first probe uses voice as the primary interaction channel, enabling natural turn-taking, real-time intention sharing, and a sense of horizontal partnership [8, 37, 66]. Users can also reference canvas objects directly during speech, enabling fluid deictic communication (e.g., “make this larger” while clicking) [47]. (2) Process Transparency: Sharing Implementation Workflow. The first probe system exposes its design process in real-time through step-by-step execution, so users observe not just what the agent produces but how it progresses, establishing common ground around both goals and process [15, 31, 49, 68]. (3) Presence Awareness: Visualizing Attention in the Shared Workspace. Following Gutwin’s workspace awareness framework [27], the agent’s position reveals its current focus, enabling users to anticipate upcoming actions and coordinate accordingly [20, 23].

3.1.2 System Design. Interface and Interaction Scenario. The first probe is implemented as a Figma plugin, in which users combine voice input with canvas element selection to provide context-based instructions to the agent (Figure 2-a-b). The agent responds through iterative tool execution—reasoning, acting, and self-evaluating in cycles—with each step reflected immediately on the canvas via a floating agent character alongside interim messages. (Figure 2-c-f). Agent Design and Pipeline. The first probe agent is built on the ReAct structure [85], operating through a three-step iteration cycle: (1) reason, (2) act, and (3) feedback (Figure 2-c; Appendix A.4). At the reasoning stage, the agent selects from 38 Figma operations (e.g., create rectangle) and generates an interim progress message, which is executed on the canvas alongside the agent character. After each cycle, the summary module generates a text description of what occurred, while the feedback module compares the canvas state before and after execution to evaluate plan completion. If the plan is fulfilled, the agent terminates and delivers a final response; otherwise, it begins a new iteration, up to a maximum of 10 cycles . We use Claude Sonnet 4.5 [7] as the main reasoning model and Claude Haiku 4.5 [6] for all other modules. Full prompt instructions, including Figma design guidelines (adapted from [40]) and the complete set of 38 Figma operation tools (See Appendix A.1.2).

Study 1 investigates two questions: (1) How does process transparency— observing the agent’s step-by-step workflow on the canvas—affect users’ collaboration experience and enable a new type of collaboration? (2) What design considerations are needed for better humanagent collaboration in co-creative workflow? We observed 10 professional UI/UX designers collaborating with our design probe across different task types to examine how process transparency shapes their understanding and collaboration strategies.

4.1.1 Understanding Collaborative Context for Flexible Human- Agent Co-Creation. Study 1 revealed that process transparency prompts users to interleave their actions with the agent’s ongoing execution.

Discussion. DI1. Adapt process visibility granularity to user needs and task stage for personalized synchronization. Agents should structure execution as observable milestones that users can use as synchronization points for timing interventions, delegation, and parallel work. Observational monitoring appeared in 70.2% of turns, with P7 and P8 actively timing their own work around the agent’s subtask completions, directly enabling triggers like “Idea Spark from Agent’s Work-in-Progress” (Table 2) and reducing clarification costs by catching misunderstandings mid-execution (P1, P4, and P8). However, rather than exposing progress uniformly, agents should adapt granularity to user needs and task stage: finer-grained operations for high-stakes tasks requiring close oversight (P2, P5, and P9), and larger subplan completions for efficiency-focused stages (P6-P8, and P10). This adaptive granularity [77] lets users determine their own synchronization rhythm rather than being locked into a single representation pace. Figure 7 illustrates this proposed adaptation between operation-level detail and milestone-level summaries. DI2. Interpret the scope and context of concurrent user actions to adapt agent behavior. Concurrent actions occurred in 33.73% of turns and comprised five concurrent action patterns (Table 1). These patterns suggest different ways for agents to respond to users’ ongoing work. Direct edits may contribute a local change or demonstrate a change the user wants applied to similar elements. Agents should consider this intended scope before carrying changes into subsequent operations. Artifact takeover, in which users duplicate work-in-progress, could prompt the agent to preserve separate work areas so that the user can refine the copy while the agent continues the original. Opportunistic takeover calls for updating the division of work: when the user completes a pending subtask, the agent should account for that contribution in its remaining plan. Intermediate result appropriation identifies partial outputs that the user has chosen to reuse; an agent could preserve those outputs while continuing its assigned work. Agents should therefore consider which artifact or subtask the user acts on and what changes, using that context to distinguish correction, parallel work, task sharing, and reuse. Figure 8 illustrates these proposed responses. DI3. Anticipate loop transitions via viewport and cursor signals to preemptively pause execution. Agents could combine viewport focus, cursor movement, and task context to infer when users may be preparing to intervene. Our findings show that observation can serve multiple purposes, including building a mental model of the agent’s capabilities and coordinating parallel work. Sustained observation alone therefore does not establish an intention to intervene. For example, a viewport centered on the agent’s current work together with a cursor approaching an editable element could provide cues of a possible transition to C or D interaction. When these cues suggest an upcoming intervention, an agent could finish its current operation, pause before the next, and expose an editable intermediate state. This design could leave room for users to redirect execution and reduce work that would need to be revised. Figure 9 illustrates this proposed use of combined viewport and cursor cues to prepare for possible intervention.

DI4. Enable direct manipulation of agent presence as unambiguous collaboration intent signals. While the third implication focuses on agents inferring intent from indirect user information, direct manipulation of agent presence could provide explicit signals.

Conclusion. This work examined concurrent human–agent interaction in shared co-creative workspaces through two design probe studies with professional designers. Analysis of 255 interaction turns and stimulated recall interviews in Study 2 yielded a taxonomy of user action patterns and a decision model explaining how triggers and enabling factors shape participation. Participants expressed emerging design preferences through concurrent edits and changed the division of work by completing pending subtasks or independently developing copies of unfinished outputs. These findings inform five design implications for adapting process visibility and coordinating agent execution with users’ edits, task sharing, and independent reuse. Looking ahead, users’ edits, reuse of intermediate outputs, and shifts in participation could help agents learn how they prefer to collaborate. We envision agents that interpret these actions in context and adapt when to offer help, make room for user contributions, or continue independently.

Limitations. Our second study was limited to 12 participants over two days, requiring larger-scale longitudinal validation to understand how interaction patterns evolve as users develop mental models of the agent’s capabilities. Additionally, our ReAct-based agent architecture represents one design choice among many [34]; different architectures (e.g., reflection-based [61], hierarchical planning, continuous execution) may elicit different interaction patterns, and investigating this could reveal which design elements are critical for effective co-creative work. Finally, we did not systematically vary execution speed or agent capabilities, or study collaboration across applications. How these changes affect participation remains an open question (Sections 5.2 and 5.4).