Can a shared canvas serve both human and agent memory?
Does representing project state as typed nodes and links—visible to both humans and AI agents—enable better continuity, reuse, and recovery than isolated prompt-response systems or hidden agent memory?
The JarvisHub paper argues that long-horizon creative work needs the agent to work from the whole "evolving project state" rather than from isolated prompt-output pairs. The abstract and introduction list what that state contains: "references, drafts, alternatives, edits, failed attempts, version relations, tool actions, evaluation signals, and human feedback." The discussion then answers "Why should the canvas be treated as an agent workspace?" by representing "prompts, references, candidates, edits, versions, dependencies, and feedback" as "typed and addressable canvas nodes and links," which "turns the canvas into both an external memory and an action space for the agent."
The reasoning is that intermediate materials "are not incidental by-products of creation" but the context that later planning, revision, and evaluation depend on. Prompt-based, chat-based, and node-based systems "only partially support this state" because they "often discard intermediate context, rely on linear conversations, or require manually specified workflows." Putting the state on a shared canvas is meant to fix all three at once. Because "users can inspect and guide the same state that the agent reads and modifies," the agent "can reuse prior artifacts, perform local updates, maintain dependencies, and continue unfinished work without hiding the process in private tool calls or transient chat history." The design choice with the most weight is that the human-facing surface and the agent-facing memory are one object, not a visual layer over a hidden agent state.
This sits close to Do generated analysis UIs really work better than chat?, which also moves work out of a long chat into a structured interface the user can revisit and reuse. The difference is who the structure serves. TaskArtisan's widgets are built for the human analyst, while the canvas here is described as a workspace for both human and agent. It also contrasts with How can GUI agents adapt when software constantly changes?, where memory is a component of the agent's planner. Here memory is placed in a workspace the user can see. And where Do text-based GUI agents actually work in the real world? concerns agents that must read interfaces built for people, the canvas is a surface built for both readers from the start.
The excerpt does not show that any of this works. It gives no evaluation, no comparison against chat or node-based baselines, and no measure of consistency over time or failure recovery. The abstract says closed commercial systems make it "difficult to study how agents represent context, choose tools, revise artifacts, recover from failures, and maintain consistency over time," and the excerpt promises an open harness for that study but ends before any of it is reported. It also does not say how the agent perceives the canvas or how its node and link types are defined. The claims about reuse, local updates, and continuing unfinished work are stated as consequences of the design, not as findings. What follows at that strength is a design argument to weigh: if project state is worth keeping, the choice of where it lives, in private tool calls, chat history, or a shared inspectable structure, decides who can audit and steer the agent.
Inquiring lines that read this note 36
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How should GUI agents perceive and represent user interface structure? What makes decomposed prompts more reliable than monolithic instructions?- What evidence shows canvas workspaces recover from failures better than chat baselines?
- Can visual intent capture through nodes replace typed prompts in practical workflows?
- How should design roles adapt when prompt artifacts become shared?
- Why do analysts prefer visible structured interfaces over hidden agent memory systems?
- How can users inspect and correct agent memory if it remains hidden from view?
- Can unified state objects replace switching between decomposition and writing tools?
- What design patterns help preserve superseded plans for later reference?
- What signals indicate a user is reusing partial outputs the agent created?
- Should agents preserve user-duplicated work separately from their own ongoing tasks?
- Does recoverable content elision in context management match externalized memory benefits?
- Should memory tools and prompts be governed structurally rather than patched?
- Who owns organizational memory when agents and workflows span platforms?
- Does task persistence across device disconnection require a shared causal timeline?
- What vulnerabilities emerge when agents operate continuously across sessions with persistent memory?
- Does memory transparency require different design than memory accuracy?
- What happens when user preferences conflict with AI memory handling decisions?
- Why do memory panels listing content miss the transparency problem users face?
- How often do users' memory organization needs change during different task stages?
- How does seamlessness in agent systems hide failures from users?
- What happens when agent output gets adopted without explicit human sign-off?
- Can agents preserve developer intent better to reduce monitoring burden?
- What should an AI interface surface to disclose an agent's proposed actions?
- What role does visible agent reasoning play in user mental models?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do generated analysis UIs really work better than chat?
TaskArtisan investigates whether putting a GUI into LLM-assisted analysis workflows improves usability and clarity, and what trade-offs emerge when analysts need to modify or reuse generated interfaces.
also replaces long chat with structured interface, but built for the human analyst rather than shared with the agent
-
How can GUI agents adapt when software constantly changes?
Can desktop automation agents stay current by combining real-time web documentation with learned task patterns and concrete execution memories? This explores how to avoid training obsolescence in open-world software environments.
agent memory inside the planner, versus memory placed in a user-visible workspace
-
Do text-based GUI agents actually work in the real world?
Can language-only agents that rely on HTML or accessibility trees handle actual user interfaces without structured metadata? This matters because deployed systems face visual screenshots, not oracle data.
agents reading human interfaces, versus one surface designed for human and agent together
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents
- "When to Hand Off, When to Work Together": Understanding Concurrent Human-Agent Interaction in Shared Co-Creative Workspaces
- Demystifying Agent Skills: Why They Work-Until They Don't
- Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents
- Challenges in Human-Agent Communication
- Towards a Science of Scaling Agent Systems
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- Cocoa: Co-Planning and Co-Execution with AI Agents
Original note title
a canvas of typed and addressable nodes and links is both external memory and action space for a creative agent — humans inspect and guide the same state