What we did
This is the description of the test as it stands in the status of the SNR measure on /lenses. We show it word for word, so that this page cannot say more than the status does.
Predicting speech-recognition errors: in our own test of 10 October 2026: 16 invented Dutch citizen reports (street and number, town, surname, phone, email), read by 3 synthetic voices in 13 conditions (clean, telephone band, street noise and babble at 10 and 5 dB, television, four measured reverberation times, combinations): 624 recordings, 1,248 transcriptions by Whisper small and large-v3-turbo, 48 per condition and model, margin about 13 points. It did not predict errors in street, number and town (AUC 0.52 to 0.55; 0.50 is a coin flip). Synthetic voices, Whisper only, simulated telephone band, Dutch text; real speakers, other recognisers and languages are untested.
The result
The SNR measure did not predict errors in street, number and town: AUC 0.52 to 0.55, where 0.50 is a coin flip. We therefore say: do not use the SNR number to predict speech-recognition accuracy.
A measure of the audio tells you something about the audio. What a recogniser does with it depends on the recogniser. To measure the recogniser itself, run the same recordings through it before and after a change: that is what the stress test is for.
How far this reaches
Dutch citizen reports read by synthetic voices, with Whisper small and large-v3-turbo, in 13 conditions including a simulated telephone band: 624 recordings and 1,248 transcriptions, 48 per condition and model, with a margin of about 13 points.
What we have not tested
- Real speakers.
- Other recognisers than the two Whisper models.
- Other languages than Dutch.
Sources and next steps
Other evidence
- Level and clipping
- Codecs
- Packet loss
- Echo: in preparation
- Noise and SNR
- A competing voice
- Scene measures and content type
- Speech detection
- Cut-off at the start and end
- Room and reverberation
- A cleaning step, before and after: in preparation
- Where we were wrong