What we did
The study (DR-1) fixed its five pass criteria in advance and measured on data it was not developed on.
- What counts as a gap. A stretch of digital silence of 8 to 500 ms that is not at the edge of the recording and cuts into sound: the level just before and just after it is within 20 dB of the active level. A natural pause fades out first, so it is not counted.
- Built gaps. 40 read recordings from LibriSpeech test-clean (40 speakers) in six conditions, with and without DEMAND noise, and with natural pauses of digital silence as a hard negative. Every gap we made has a known place and length.
- The packet-loss situation of our scenario set. Synthetic voices with 20 ms packets, a 6% chance of loss per packet and a loss of one to three packets, set to zero. Recordings without gaps, including synthetic voices that have natural pauses of digital silence, were the negatives.
Separately, the stress test cut about 10% of the packets (20 ms each) out of 24 English read sentences, without concealment, for two open Whisper models.
The result
DR-1: all five criteria were met. 99.9% precision on 8,406 gaps, 99.8% of the 8,411 gaps that fell in sound found, and no false alarms on 1,615 recordings without gaps. The three measures are Validated, with an analytic basis.
The count is a lower bound. About 60% of the gaps we made fell in sound. A loss inside a pause cannot be told from the pause, so it is not counted. A loss that a codec concealed leaves no gap to find, and AMR silence is not counted either.
What packet loss costs a recogniser
Whisper small
| Condition | WER % | Change against clean, points (95% interval) | Reading |
|---|---|---|---|
| Clean (reference) | 5.0 | n/a | reference |
| About 10% packet loss (20 ms packets, no concealment) | 4.8 | −0.2 (−1.3 to +0.7) | no clear change |
Whisper large-v3-turbo
| Condition | WER % | Change against clean, points (95% interval) | Reading |
|---|---|---|---|
| Clean (reference) | 3.4 | n/a | reference |
| About 10% packet loss (20 ms packets, no concealment) | 3.6 | +0.2 (−0.7 to +1.3) | no clear change |
English read speech, 24 sentences per condition, the same sentences in every condition, so each recording is compared with its own clean version. Clearly worse means the 95% interval of the paired change lies above zero and the change is at least 2 points. No clear change does not mean no change: the intervals are wide. The conditions were chosen after a first trial with Whisper small. Set stresstest-core-en version 0.2.
How far this reaches
Gaps of digital silence in the middle of sound, in read English speech and in synthetic voices, with and without noise. The recogniser numbers are a small test: about 10% loss in 20 ms packets, with no concealment, made no clear difference for either model.
What we have not tested
- Gaps that a codec concealed, and the silence an AMR codec inserts.
- Real network loss, and real phone calls.
- What packet loss with concealment, the usual case in a real call, costs a recogniser.
- Other recognisers and other languages.
Sources and next steps
Studies in our register: DR-1, stress test v0.2. The status of every measure is on /lenses, and how we test is on /methodology.
Other evidence
- Level and clipping
- Codecs
- Echo: in preparation
- Noise and SNR
- A competing voice
- Scene measures and content type
- Speech detection
- Cut-off at the start and end
- Room and reverberation
- Do audio measures predict errors?
- A cleaning step, before and after: in preparation
- Where we were wrong