When an AI works on a canvas of labeled boxes and links, how does it see them and find its way?

This explores how an agent takes in a canvas of typed nodes and links, and how it moves through that structure to find what it needs. The collection has one note that describes such a canvas directly and none that follow an agent walking across one. Its neighbors on graph navigation and structured screen parsing fill in most of the picture, but some of what follows is inference.

The canvas note makes perception a design choice. In JarvisHub, prompts, references, versions and feedback are typed nodes that users and agents can both see, instead of agent memory buried in chat or transient state (Can a shared canvas serve both human and agent memory?). The agent doesn't have to reconstruct the project from a conversation log. It looks at addressable objects, each labeled with what kind of thing it is. That is what allows local updates, reuse of earlier artifacts, and picking up unfinished work.

GUI-agent research suggests why labels matter so much. OmniParser found that GPT-4V struggles when it must work out what an icon means and pick an action from a raw screenshot at the same moment. Pre-parsing the screen into described elements lets the model concentrate on the action (Why do vision-only GUI agents struggle with screen interpretation?). Agent S pairs a visual view with an image-augmented accessibility tree, a structured list of what is on screen, and gained 9.37% over baseline (Can structured interfaces help language models control GUIs better?). A typed canvas looks like that accessibility tree without the parsing step. The type labels are the pre-parsing.

For moving through the graph, the strongest evidence is that agents shouldn't read all of it. Graph-O1 replaces whole-graph ingestion with step-by-step navigation trained with Monte Carlo Tree Search and reinforcement learning. That keeps it inside the context window and lets it learn domain-specific routes, at the cost of never being certain about the parts it skipped (Can learned traversal policies beat exhaustive graph reading?). MRAgent goes further and treats memory as something rebuilt during traversal. It prunes paths as evidence accumulates, gaining up to 23% on reasoning tasks while spending fewer tokens than fixed retrieve-then-reason pipelines (Can agents reconstruct memory on demand instead of retrieving it?). On a canvas, that would mean reading a node, following the links that look useful and dropping the rest. Node and link types would be cheap signals for deciding which branches to drop.

Three other notes treat the graph as more than a thing to walk. One represents the agent itself as a computational graph, with nodes as operations and edges as information flow. In that view chain-of-thought, tree-of-thought and Reflexion turn out to be the same kind of structure, and both prompts and connectivity can be optimized (Can we automatically optimize both prompts and agent coordination?). Another skips walking for discovery and finds agents by semantic similarity in an index, with no hand-wired links (Can semantic capability vectors replace manual agent routing?). The third shows that links don't say everything about meaning. In iterative graph reasoning, roughly 12% of edges stay semantically surprising even though they are structurally connected, and that keeps feeding new discoveries (Why do reasoning systems keep discovering new connections?).

The open question is whether an agent on a canvas like JarvisHub's reads everything or navigates selectively. The corpus has no test of that. The navigation results above suggest selective traversal wins once the canvas outgrows the context window.