Skip the transcript: AI voice agents that work straight from sound, not text, can reply in a fraction of a second.
Does direct speech-to-speech generation actually reduce latency over ASR-LLM-TTS?
This explores whether voice systems that go straight from speech to speech, skipping the transcribe-think-speak pipeline, are really faster, and what else changes when you remove the transcription step.
This explores whether voice systems that go straight from speech to speech are faster than the classic cascade, where speech is transcribed to text (ASR), passed to an LLM, and then turned back into speech (TTS). The corpus says yes, and gives a specific reason. LLaMA-Omni reaches about 226 milliseconds of response latency by never producing a transcript Can skipping transcription make voice assistants faster?. The speed doesn't come from doing each step faster. In a cascade, each stage waits for the one before it to finish. A model that works on speech embeddings can start generating before the user has finished talking, so the waiting disappears.
The more surprising point is that latency covers more than processing speed. It also includes knowing when to speak. A cascade has to bolt on separate parts to detect the end of a turn and decide whether to interrupt, and every one of those parts adds delay and can fail. Wan-Streamer puts language, audio and video into a single stream of tokens, so one Transformer learns turn-taking as part of its training instead of having it engineered in Can a single model learn when to speak and respond?. That gets it under a second while handling the timing of a real conversation. On this view, the main gain from going direct is that the model can learn conversational rhythm, which a pipeline struggles to express.
Removing the transcript also removes a well-documented weak point. Real-world speech recognition gets 15 to 30 percent of words wrong in noisy settings, which is why older dialogue systems had to track a range of possible meanings instead of trusting one transcript Why do dialogue systems need probabilistic reasoning?. Those errors are not spread evenly: commercial ASR systems make nearly twice as many errors for Black speakers, and the cause is in the acoustic models, not the vocabulary Why do speech-to-text systems fail more often for Black speakers?. In a cascade, every one of those errors passes straight to the LLM. A direct model at least keeps the acoustic signal, including tone and hesitation, that a transcript throws away. Whether that makes it fairer is an open question, because it could carry over the same gaps in its training data.
There's a tension the corpus suggests but doesn't settle. Starting to answer before the user has finished is what makes direct models fast, but committing early is also how LLMs go wrong in text conversations. Models that lock in early guesses about what a user wants lose about 39% of their performance and rarely recover Why do language models fail in gradually revealed conversations?. A voice model that starts replying at 226 milliseconds is making that kind of early bet.
One caveat: the corpus has no head-to-head benchmark that runs a well-optimized streaming cascade against a direct model on the same tasks. It reports strong latency numbers for direct systems and explains why they're faster, but it doesn't measure how much is gained, or what is given up, when both designs are equally tuned.
Sources 5 notes
LLaMA-Omni generates speech responses directly from speech input without transcribing to text first, achieving 226ms latency. This works because speech embeddings preserve acoustic information that text loses, enabling generation before full input is received.
Wan-Streamer represents language, audio, and video as one interleaved causal token stream, allowing response timing and turn management to be learned jointly within a single Transformer rather than engineered as separate modules, achieving sub-second latency.
Real-world speech recognition achieves 15-30 percent error rates in noisy environments, making deterministic flowchart dialogue systems unworkable. POMDP-based systems handle this by maintaining belief distributions over user intent rather than committing to single interpretations.
Five major commercial speech-to-text systems produced word error rates nearly twice as high for Black speakers as white speakers, traced to acoustic models trained on insufficient African American Vernacular English data rather than vocabulary differences.
Across 200,000+ conversations, all major LLMs show 39% average performance drop in multi-turn settings due to locking into incorrect early guesses. Agent mitigations recover only 15-20% of this loss.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- LLMs Get Lost In Multi-Turn Conversation
- Are LLMs All You Need for Task-Oriented Dialogue?
- Racial disparities in automated speech recognition
- Realtime-Venus: A full-duplex interaction system with asynchronous delegation
- Efficient Streaming Language Models with Attention Sinks
- Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
- POMDP-based Statistical Spoken Dialogue Systems: a Review