Can you just tell an AI to 'swap that table for a chart' and have your workspace rearrange itself?

This explores whether an analyst can tell an AI assistant something like 'swap that table for a chart' or 'put the filter next to the map' and have the interface's widgets rearrange mid-task. The corpus suggests yes, people want this and it works to a point. But natural language doesn't remove the cost of changing a generated interface. It moves the cost into finding the right words.

The clearest evidence is TaskArtisan, which generates GUI widgets for LLM-assisted analysis. The widgets made results clearer and easier to present than chat did, but they brought rigidity and prompting overhead. The trade-off looks hard to escape. The easier a generated UI is to use, the harder it is to customize halfway through a workflow. The more flexible it is, the more it asks non-programmers to think like engineers when phrasing requests (Do generated analysis UIs really work better than chat?). So the answer to 'can natural language modify widgets?' is yes, but it is not free.

The appetite for this is real, though. In a separate study, users preferred LLM-generated task-specific interfaces (dashboards, tools, animations) over plain chat in over 70 percent of cases. The preference was strongest for structured, information-dense work like analysis. The mechanism that made it work was iterative refinement, where you keep adjusting the generated UI in conversation rather than getting it right the first time (Do generated interfaces outperform text-based chat for most tasks?).

One reason phrasing is hard is that people often don't know what layout they want until they see one. Work on the 'gulf of envisioning' finds that intent matures through interaction, and that models mostly respond instead of probing. Its fix is to have the model offer options, which turns an open-ended 'describe what you want' into a smaller 'which of these?' (Why can't users articulate what they want from AI?). For widgets, that would mean showing three candidate arrangements and letting the analyst point. That is my extension of the finding. The note itself doesn't test widget layouts.

Two neighboring ideas could make the talk-to-UI channel less brittle, again as extensions rather than tested results. Rasa's dialogue system turns what users say into commands in a small domain-specific language instead of sorting it into intent categories. That handles context well and needs no labeled training data (Can command generation replace intent classification in dialogue systems?). A small command vocabulary for 'move', 'resize', 'swap' and 'link' widgets could sit between speech and the interface. Conversation analysis adds insert-expansions, where the system asks a quick clarifying question before acting instead of silently guessing (When should AI agents ask users instead of just searching?). That would help with ambiguous requests like 'make that bigger.'

The corpus has no note that directly tests natural-language editing of widget layouts during analysis. What it does show is that the cost shifts from clicking to phrasing. The designs most likely to work combine free-form talk with offered options and a constrained set of commands.