What we did
Three studies asked how close the estimate is to the true ratio. All of them mix speech with noise at a known ratio, so the right answer is known. The first is descriptive; the other two fixed their pass criteria in advance.
- SNR-1. 680 mixtures: 40 read recordings from LibriSpeech dev-clean, each clean and with four DEMAND noises at 20, 10, 5 and 0 dB.
- SNR-2. 1,344 recordings: 48 synthetic utterances in nine DEMAND environments at 20, 10 and 5 dB, plus clean controls. It tested a hybrid estimator against a bar fixed in advance.
- SNR-3. 520 mixtures of LibriSpeech test-clean with three environments that were in no development set: a town square, street traffic and a metro. It tested a correction that applies below 16 dB, on top of the second speech detector.
Separately, the stress test measures what noise costs a recogniser: 24 English read sentences in the same conditions for two open Whisper models.
The result
The default estimate reads too high at low ratios (SNR-1). With kitchen noise the average error was +1.2 dB at 20 dB, +4.1 dB at 10 dB, +6.8 dB at 5 dB and +10.7 dB at 0 dB. With hallway noise it was +3.4, +6.9, +9.0 and +12.5 dB. The noise level itself read 2.6 to 5.4 dB too low in the kitchen, and up to 8.5 dB too low in the hallway. Where the true ratio is 10 to 30 dB, the estimate is off by at most 6 dB in 90% of cases; below that it reads too high.
A hybrid estimator was not adopted (SNR-2). It missed its first bar: the 90th percentile of the error at 10 dB was 6.21 dB against a bar of 6.0, and at 10 dB it overcorrected, by 3.4 dB on average. The test did confirm SNR-1: the default estimate read +1.6 dB too high at 20 dB, +3.2 dB at 10 dB and +5.6 dB at 5 dB.
A correction was adopted as an opt-in (SNR-3). All four criteria were met. At 5 dB the average error went from +5.95 dB to +1.87 dB, and the 90th percentile from 7.44 dB to 2.98 dB. At 10 dB the 90th percentile went from 5.28 dB to 2.01 dB without overcorrection; at 20 dB and on clean recordings nothing changed. The measure stays Validated, with its margin.
What noise costs a recogniser
Whisper small
| Condition | WER % | Change against clean, points (95% interval) | Reading |
|---|---|---|---|
| Clean (reference) | 5.0 | n/a | reference |
| Street noise at the level of the speech (0 dB) | 5.9 | +0.9 (−0.9 to +3.7) | no clear change |
| Restaurant noise, 5 dB below the speech | 11.4 | +6.4 (+1.9 to +12.2) | clearly worse |
| Restaurant noise at the level of the speech (0 dB) | 33.8 | +28.8 (+19.5 to +40.1) | clearly worse |
Whisper large-v3-turbo
| Condition | WER % | Change against clean, points (95% interval) | Reading |
|---|---|---|---|
| Clean (reference) | 3.4 | n/a | reference |
| Street noise at the level of the speech (0 dB) | 4.1 | +0.7 (−0.3 to +1.9) | no clear change |
| Restaurant noise, 5 dB below the speech | 7.1 | +3.6 (+1.7 to +5.8) | clearly worse |
| Restaurant noise at the level of the speech (0 dB) | 22.6 | +19.2 (+10.8 to +30.6) | clearly worse |
English read speech, 24 sentences per condition, the same sentences in every condition, so each recording is compared with its own clean version. Clearly worse means the 95% interval of the paired change lies above zero and the change is at least 2 points. No clear change does not mean no change: the intervals are wide. The conditions were chosen after a first trial with Whisper small. Set stresstest-core-en version 0.2.
How far this reaches
The margin of plus or minus 6 dB holds for read speech with steady background noise at a true ratio of 10 to 30 dB. With the second speech detector it holds from about 5 dB for a town square and for street traffic, and pooled over kitchen, living room, hallway and cafeteria (90th percentile 4.4 dB at 5 dB). At 10 dB the pooled value over those four is 6.25 dB and a metro reaches 7.3 dB. At 0 dB the estimate still reads about 6 dB too high. Without the opt-in the margin holds only from 10 dB.
The recogniser numbers are a small test with wide intervals: street noise at the level of the speech made no clear difference, restaurant noise did.
What we have not tested
- Real conversations and real phone lines.
- Noise from a real television or from music behind the speaker.
- Other languages, and other recognisers than the two Whisper models.
- Whether the estimate predicts which recordings a recogniser gets wrong: in our own test the SNR measure did not. See Do audio measures predict errors?
Sources and next steps
Studies in our register: SNR-1, SNR-2, SNR-3, stress test v0.2. The status of every measure is on /lenses, and how we test is on /methodology.
Other evidence
- Level and clipping
- Codecs
- Packet loss
- Echo: in preparation
- A competing voice
- Scene measures and content type
- Speech detection
- Cut-off at the start and end
- Room and reverberation
- Do audio measures predict errors?
- A cleaning step, before and after: in preparation
- Where we were wrong