Speech AI is judged mostly on getting the words right — but almost never on whether it actually understood what was said.

This explores how speech models get tested on understanding rather than on accurate transcription. The short answer is that the corpus mostly records an absence. It doesn't catalog named comprehension benchmarks. What it shows is that the field has largely failed to build them. Speech evaluation is concentrated on transcription accuracy and translation quality. Question-answering, summarization and reasoning over audio have no standardized equivalents What speech tasks remain without standardized benchmarks?. That gap shapes what gets built: models are optimized for the thing being measured, so speech systems get steadily better at turning sound into text without anyone checking whether they understand it.

The gap matters more now because the architecture is changing. Direct speech-to-speech models skip the transcript entirely. They respond in about 226 milliseconds and keep information like tone, timing and how sounds are physically produced, which transcription throws away How can dialogue systems handle the noise in speech input?. A word-error-rate score can't measure a model that never writes the words down. The systems most likely to understand speech are the ones the dominant metric is least equipped to judge.

Word error rate also hides things even on its own terms. Five commercial speech-to-text systems made nearly twice as many errors for Black speakers as for white speakers. The cause was acoustic models trained on too little African American Vernacular English, not vocabulary Why do speech-to-text systems fail more often for Black speakers?. A single average score conceals who the errors land on. Older dialogue-systems research makes a related point from the other side. With real-world error rates of 15 to 30 percent in noisy settings, the useful question is whether the system still worked out what the user wanted, not how many words it got right. Those systems handled it by keeping several guesses about the user's intent open instead of committing to one transcript Why do dialogue systems need probabilistic reasoning?. That is an early form of comprehension-centered evaluation: score the understanding, not the transcript.

If you want to see what a speech comprehension benchmark should avoid, look at what went wrong in text. Standard NLP benchmarks quietly drop examples where human annotators disagreed. That hid a gap between 32% and 90% accuracy on recognizing ambiguity Do standard NLP benchmarks hide LLM ambiguity failures?. Spoken language is full of exactly that kind of ambiguity: sarcasm, hedging, a rising tone that turns a statement into a question. Theory-of-mind benchmarks turned out to be solvable by matching surface patterns without any real reasoning Can language models solve ToM benchmarks without real reasoning?. Models also do better on common phrasings than on rare ones that mean the same thing Do language models really understand meaning or just surface frequency?. A spoken-QA benchmark built from templated questions or clean read speech would likely reward the same shortcuts.

Long audio adds one more consideration. Text reasoning accuracy falls from 92% to 68% with only 3,000 tokens of padding, well below the context limit Does reasoning ability actually degrade with longer inputs?. An hour-long meeting recording is far longer than that. Any serious speech comprehension test probably needs to vary recording length on purpose rather than rely on short clips. Taken together, the corpus suggests the missing benchmarks would need to cover four things: whether the model gets the meaning, holds up under ambiguity, works equally well across speakers, and keeps working over long recordings. Today's main metric measures none of these.