Could a drawn box or painted mask tell an AI exactly what part of an image to change?

This explores how spatial signals, like a painted mask or a box drawn around a region, tell a generative system what a user wants. The collection doesn't cover inpainting or region-guided image editing, the most literal reading of the question. What it does have is a cluster of work on how location and layout carry meaning for AI systems, plus a twist: in some of this work, masking teaches the model what users want rather than being how the user tells it.

Start with coordinates as meaning. DocLLM shows that a bounding box (four numbers marking where text sits on a page) can do much of the work people assume needs a full image encoder Can bounding boxes replace image encoders for document understanding?. The model attends to 'what the text says' and 'where it sits' as separate channels and learns how they relate. That's the core idea behind a user-drawn region too: a coordinate range is a compact, explicit statement of 'this part, not that part.' It's cheaper and less ambiguous than asking a model to work out from pixels which part matters. OmniParser points the same way from the GUI side. Vision models do poorly when they must both interpret a raw screenshot and decide what to do, and they do much better when the screen is first parsed into labeled regions Why do vision-only GUI agents struggle with screen interpretation?. Spatial structure takes over the 'where is it' half of the job, so the model can focus on intent.

Now the twist. In UI-JEPA, masking isn't something the user does. It's a training trick. Chunks of screen-recording video are hidden, and the model learns to predict them. Doing that well forces it to model what the person on screen is trying to accomplish, without needing text labels Can unlabeled UI video teach models what users intend?. Poppins goes further and drops explicit signals altogether: it watches the screen and infers short-lived goals, and people preferred its outputs over a standard LLM's 66–86% of the time Can systems infer what users want from watching their screen?. So intent can be pointed to (a mask or a box), learned from what's hidden (predictive masking), or inferred from behavior (passive observation).

Why does pointing matter? Because typed descriptions lose information. Generative UI tools skip about a quarter of the design reasoning stated in prompts, and about a third of functional requirements Do generative UI tools actually implement their stated design rationales?. Moran argues that most people can't describe the interface they want anyway Can generative UI serve users who cannot specify interfaces?. A mask or a box is one way past that limit: you don't have to say what you want if you can show where. Rasa's shift from sorting user requests into intent categories to generating structured commands reflects the same instinct. Intent travels better as something precise and actionable than as a fuzzy label Can command generation replace intent classification in dialogue systems?.

The thing you might not have expected to want to know: a mask means opposite things depending on who holds it. Held by the user, it narrows the scope ('change only here'). Held by the trainer, it's a learning signal ('guess what's missing, and you'll learn what people are doing'). If you want the image-editing side specifically (inpainting, region prompts, layout-to-image), that's a gap in the collection.