# We shrank Whisper large-v3 by 68%, below the size of turbo

Tileward research report R-009. Whisper large-v3 is a speech recogniser, and as published its weights are 3.09 GB. Tileward's TWX4 build is 0.98 GB, 68% smaller (3.2×) and smaller than the published turbo version at 1.62 GB. On nine of ten test sets it adds at most 0.3 points to the original's error rate; on Danish it adds 1.06.

Page: https://tileward.com/research/a-smaller-speech-recogniser/

## How it was measured

Every build ran on the same card, the same server and the same clips, and every error rate is compared with [Whisper large-v3 as published](https://huggingface.co/openai/whisper-large-v3), clip for clip.

- **Builds:** Whisper large-v3 and [large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo) as published, and a Tileward TWX4 build of each.
- **Card:** one 20 GB card. Every build ran on vLLM from its own weights, with the server allowed 90% of the card's memory.
- **Size:** the weights file; loaded on the card, the weights take the same space.
- **Test sets:** [LibriSpeech](https://www.openslr.org/12) test-clean (2,611 clips) and test-other (2,932); [FLEURS](https://arxiv.org/abs/2205.12446) test in German (841), Spanish (907), Hindi (417), Arabic (428), Mandarin (944), Japanese (650) and Danish (929); and 384 synthetic call-centre clips, scripted call lines in eight synthetic voices, each clean, through a telephone band and with noise.
- **Scoring:** greedy decoding, with the language given and one 30-second window a clip. Word error rate after Whisper's own text normaliser, or character error rate for Mandarin and Japanese. Each change has a paired bootstrap 95% range from 2,000 resamples of the clips.
- **Live calls:** one second of audio transcribed per second, on one 20 GB card. 512 windows of continuous read English, 28 seconds each, sent at once, with up to 256 in progress; each run took 35 to 83 seconds, so start-up and tail are inside the figure.

## Results

| Build | Weights | Live calls on one 20 GB card |
|---|---|---|
| Whisper large-v3 as published | 3.09 GB | 140 |
| Tileward TWX4 of large-v3 | 0.98 GB, 68% smaller (3.2×) | 152, 8% more |
| Whisper large-v3-turbo as published | 1.62 GB | 301 |
| Tileward TWX4 of turbo | 0.56 GB, 65% smaller (2.9×) | 331, 10% more |

Error rate is the share of words transcribed wrong, or of characters for Mandarin and Japanese. Each build's column is its change from the original, in points, with its 95% range.

| Test set | Original | TWX4 of large-v3 | Turbo as published |
|---|---|---|---|
| LibriSpeech test-clean | 1.89% | +0.01 (−0.08 to +0.09) | +0.11 (−0.07 to +0.28) |
| LibriSpeech test-other | 3.57% | +0.16 (+0.08 to +0.25) | +0.36 (+0.20 to +0.51) |
| Synthetic call-centre clips | 0.79% | −0.02 (−0.19 to +0.12) | −0.10 (−0.42 to +0.22) |
| FLEURS German | 5.93% | +0.11 (−0.03 to +0.26) | +0.43 (+0.17 to +0.70) |
| FLEURS Spanish | 4.21% | +0.09 (+0.00 to +0.17) | +0.39 (+0.25 to +0.55) |
| FLEURS Hindi | 18.48% | +0.16 (−1.19 to +1.05) | +2.04 (+0.30 to +4.01) |
| FLEURS Arabic | 16.69% | +0.27 (−0.09 to +0.62) | +0.57 (+0.05 to +1.09) |
| FLEURS Mandarin, characters | 6.23% | +0.27 (+0.05 to +0.49) | +0.49 (+0.16 to +0.82) |
| FLEURS Japanese, characters | 6.06% | −1.22 (−4.75 to +0.62) | +0.02 (−4.25 to +3.95) |
| FLEURS Danish | 14.02% | +1.06 (+0.71 to +1.38) | +2.29 (+0.96 to +4.49) |

- Japanese turns on single clips: the original repeats itself to the length limit on one of them, so read that set as level.
- Against turbo, clip for clip, the large-v3 build is 40% smaller (1.7×) and its error rate is lower on four sets: test-other by 0.19 points, German by 0.32, Spanish by 0.31 and Hindi by 1.88. It is level on the other six. Turbo is the faster of the two, by about two to one on the same card.
- Large-v3 fills this card's memory, so a larger card holds more calls; turbo reaches the server's limit of 256 requests first.
- The turbo build is level with turbo on six sets and behind on four: test-other by 0.09 points, Arabic by 0.68, Mandarin by 0.53 and Japanese by 0.31.

## Which build

The large-v3 build is for accuracy: it adds at most 0.3 points on nine sets, at about a third of the original's size. The turbo build is for calls per card: 331 live calls on one 20 GB card, trailing turbo by up to 0.68 points on four sets.

## Related

- [Speech recognition eval card](https://tileward.com/evals/#compressed-speech-recognition): the result in one screen
- [Whisper large-v3 on Hugging Face](https://huggingface.co/openai/whisper-large-v3): the published model
- [Talk to an engineer](https://calendar.app.google/hg2DzFTxKJgwFWm19): bring the audio you transcribe and the card you run it on
- [Research](https://tileward.com/research/): every report