Five speech-to-text systems made nearly double the errors on Black speakers' voices — what should a fair bias test actually check?

This explores what a fair test of speech-to-text systems for demographic bias should look like. The library has only one study that audits speech recognition directly, but it sets the template. Why do speech-to-text systems fail more often for Black speakers? ran the same kind of speech through five major commercial systems and compared word error rates for Black and white speakers. Black speakers' error rates were nearly double. The useful part for auditors is where the authors traced the cause: not to vocabulary, but to acoustic models that hadn't heard enough African American Vernacular English. So an audit should do more than report one accuracy number. It should split errors by speaker group and then ask which layer of the system is failing: the part that hears sounds or the part that predicts words.

The second lesson comes from text-model research. Bias usually starts in the data a model is first trained on, not in later tuning. Where do cognitive biases in language models come from? found that models built on the same pretrained base show similar biases whatever they were fine-tuned on. Where does LLM recommendation bias actually come from? found that standard debiasing methods can't remove a skew inherited from pretraining. Applied to speech, this means checking only the final product isn't enough. An audit should ask whose voices were in the training audio. Fine-tuning afterward may soften a gap without closing it.

The third lesson is to be careful with the statistics. Do large language models show racial sentiment bias? found a small racial effect that disappeared after a reasonable change to the analysis. The result was neither evidence of bias nor evidence of its absence. A speech audit needs enough speakers in each group, analysis choices decided in advance, and results that hold up under other reasonable analyses. A twofold gap like the one in the ASR study is robust. A 3% gap may not be.

The less obvious lesson is that a transcription error matters because of what happens next. Why do dialogue systems need probabilistic reasoning? shows that real-world speech recognition already gets 15–30% of words wrong in noisy settings. Voice systems survive this only by keeping track of several possible meanings at once. If one group's error rate is twice as high, that group is pushed past the point where those safeguards still work. So a good audit follows the errors downstream: does the assistant misread the request, take the wrong action, or refuse? Other work shows identity signals can change AI behavior on their own. Do AI guardrails refuse differently based on who is asking? found refusal rates that varied by the user's apparent demographics. Do users worldwide trust confident AI outputs even when wrong? found that people trust confident outputs even when they are wrong. Together these suggest one more check: whether the system is just as confident about its wrong transcripts for some groups as for others. An error that looks certain is the one users won't catch.

The corpus doesn't cover accents, age, disability, or non-English dialects, or give a standard audit protocol. Those are clear gaps. What it does give is a framework. Measure error rates for each group, trace gaps back to the training data, check that the statistics hold up, and follow the errors into what the system does next.