What we did
- 24 sentences from LibriSpeech test-clean, English, read by 24 different speakers. The same 24 sentences are in every condition, so each recording is compared with its own clean version.
- Eight conditions: clean, telephone band, street noise, restaurant noise at two levels, a second voice at two levels, and about 10% packet loss without concealment. The noise comes from the DEMAND recordings. The second voice is another LibriSpeech speaker reading different text. A level of 0 dB means as loud as the speech; 5 dB means 5 dB quieter.
- Two open Whisper models, small and large-v3-turbo, run on a local computer. We report word error rate (WER) with one fixed way of normalising the text, written down so that another team gets the same number.
- For every condition we report the change in WER against clean, in percentage points, with a 95% interval from a bootstrap over the 24 sentences.
- The conditions were chosen after a first trial with Whisper small. Some did not separate from clean, so we replaced them with ones that did. The set therefore says something about what Whisper is affected by, not about every system.
The result
Restaurant noise and a second voice cost the most. At the level of the speech, a second voice raised the error rate by about 40 points for both models. Street noise at the level of the speech, the telephone band and about 10% packet loss without concealment made no clear difference.
Whisper small
| Condition | WER % | Change against clean, points (95% interval) | Reading |
|---|---|---|---|
| Clean (reference) | 5.0 | n/a | reference |
| Telephone band, 300 to 3400 Hz, mu-law | 5.3 | +0.2 (+0.0 to +0.8) | no clear change |
| Street noise at the level of the speech (0 dB) | 5.9 | +0.9 (−0.9 to +3.7) | no clear change |
| Restaurant noise, 5 dB below the speech | 11.4 | +6.4 (+1.9 to +12.2) | clearly worse |
| Restaurant noise at the level of the speech (0 dB) | 33.8 | +28.8 (+19.5 to +40.1) | clearly worse |
| A second voice, 5 dB below the speech | 10.5 | +5.5 (+0.6 to +11.9) | clearly worse |
| A second voice at the level of the speech (0 dB) | 46.6 | +41.5 (+22.3 to +62.7) | clearly worse |
| About 10% packet loss (20 ms packets, no concealment) | 4.8 | −0.2 (−1.3 to +0.7) | no clear change |
Whisper large-v3-turbo
| Condition | WER % | Change against clean, points (95% interval) | Reading |
|---|---|---|---|
| Clean (reference) | 3.4 | n/a | reference |
| Telephone band, 300 to 3400 Hz, mu-law | 2.5 | −0.9 (−2.0 to +0.0) | no clear change |
| Street noise at the level of the speech (0 dB) | 4.1 | +0.7 (−0.3 to +1.9) | no clear change |
| Restaurant noise, 5 dB below the speech | 7.1 | +3.6 (+1.7 to +5.8) | clearly worse |
| Restaurant noise at the level of the speech (0 dB) | 22.6 | +19.2 (+10.8 to +30.6) | clearly worse |
| A second voice, 5 dB below the speech | 8.2 | +4.8 (−0.2 to +14.1) | no clear change |
| A second voice at the level of the speech (0 dB) | 43.6 | +40.2 (+19.6 to +64.2) | clearly worse |
| About 10% packet loss (20 ms packets, no concealment) | 3.6 | +0.2 (−0.7 to +1.3) | no clear change |
English read speech, 24 sentences per condition, the same sentences in every condition, so each recording is compared with its own clean version. Clearly worse means the 95% interval of the paired change lies above zero and the change is at least 2 points. No clear change does not mean no change: the intervals are wide. The conditions were chosen after a first trial with Whisper small. Set stresstest-core-en version 0.2.
Sometimes the second voice is what gets written
At equal level, in 6 of 24 recordings both models wrote mostly the other speaker's sentence instead of the target speaker's. With the second voice 5 dB quieter this happened in 0 of 24 recordings for Whisper small and 1 of 24 for Whisper large-v3-turbo. In the other six conditions it happened in 0 of 288 recordings.
How we counted. A recording counts when the transcript is closer to the other speaker's sentence than to the target speaker's sentence, and at least half of the other sentence's words are right. This is our own rule for a small test; it is not a standard measure.
How far this reaches
This covers English read speech with a second voice in the same language, mixed in at a known level, for two open models. Because the intervals are wide, a second voice 5 dB below the speech is clearly worse for one model and not for the other.
What we have not tested
- A real television. Our second voice is a person reading in the same language, mixed in at a known level. A television adds music, effects, the reverberation of the room it plays in, and compression, and it changes with the programme.
- Other recognisers and other languages, including the ones inside commercial voice agents.
- Real calls: real phone networks, real conversation, interruptions, accents and dialect. The telephone band here is a filter and a codec, not a call.
- How large the effects are for you. 24 sentences per condition is small. Read the intervals, not only the numbers.
- Whether a cleaning or enhancement step in front of the recogniser helps. The set can show it, by running it with and without the step. We publish no result on it yet.
Where AudioLab fits
AudioLab does not detect a second voice or a television in a recording. The stress test is a way to measure your own recogniser or agent under known conditions: the same recordings, run before and after a change, with a paired report. It is in preparation; the set and the runner are not published yet and there is no date.
Sources and next steps
Studies in our register: stress test v0.2. The status of every measure is on /lenses, and how we test is on /methodology.
Other evidence
- Level and clipping
- Codecs
- Packet loss
- Echo: in preparation
- Noise and SNR
- Scene measures and content type
- Speech detection
- Cut-off at the start and end
- Room and reverberation
- Do audio measures predict errors?
- A cleaning step, before and after: in preparation
- Where we were wrong