Why do we still poke, type, and drag to control AI, instead of just talking or gesturing to it naturally?

This explores why voice and gesture are still unusual ways to work with generative AI, even though they feel like the most natural ways for people to communicate. The headline number is lopsided. A review of 291 papers on how people interact with GenAI found direct controls like sliders and text boxes in 234 of them, while only 49 used intuitive modalities like gesture and voice Why do GenAI interfaces favor sliders over voice and gesture?. The authors read this as researchers building what is easier to implement rather than what users would find most natural. So the first barrier is mostly an engineering habit, not something users are asking for.

The practical reason that habit exists is that voice is a noisy channel. Speech recognition in real-world noisy settings still gets 15 to 30 percent of words wrong, which is why older dialogue systems had to stop following fixed flowcharts and instead keep a running set of probabilities about what the user probably meant Why do dialogue systems need probabilistic reasoning?. A slider never mishears you. When you add a generative model on top, a misheard word doesn't only produce a wrong answer. It produces a confident, fluent wrong answer. One way to see the problem is that what a GenAI system actually receives is an "augmented instruction": your prompt plus whatever constraints the interface encodes in machine-readable form How do interaction signals reshape what generative AI actually receives?. Sliders and buttons convert cleanly into that format. A shrug, a pointing hand, or a hesitant spoken phrase has to be interpreted first, and each interpretation step is a new place for errors to creep in.

The second barrier is one people don't expect: even with perfect recognition, voice still runs into the problem that people often can't say what they want. Research on the "gulf of envisioning" finds that intent forms *through* interaction, so open-ended expression tends to fail unless the system offers options to react to Why can't users articulate what they want from AI?. Friedman argues that typed chat already makes users spell out their intent too slowly, and that the fixes are visual and direct: canvases, node graphs, style lenses Can task-oriented UI patterns reduce the friction of AI chat interfaces?. Voice is just a faster way of doing the same open-ended articulation, so it inherits that weakness. It also runs into the fact that LLM agents are passive by design. They answer but don't lead or probe Why can't conversational AI agents take the initiative?. A natural spoken conversation needs a partner who takes initiative, and current models mostly don't.

The third barrier is in testing and design. Real-time multimodal agents, which respond continuously to what they see and hear, are hard to evaluate with standard benchmarks. One case study (SocraBot) had to combine replaying recorded video with "what if" prompts and a hybrid Wizard-of-Oz setup, where a hidden human covers part of the system's behavior, just to refine its design How can designers prototype real-time multimodal AI agents?. Vision has a similar weakness on the output side. Models struggle when they must interpret a raw screen and decide what to do at the same moment, and do better when the input is first parsed into structured parts Why do vision-only GUI agents struggle with screen interpretation?. Gesture input probably faces the same two-jobs-at-once problem, though the corpus doesn't test that directly.

One caveat: the collection has little material that studies gesture input to GenAI head-on. Most of the evidence is indirect, drawn from speech recognition, intent articulation, and interface design. The surprising lesson is that the hardest part of voice and gesture isn't recognizing them. It's that they are open-ended channels, and open-ended input is exactly where both people and current AI systems are weakest.