We shrank Whisper large-v3 by 68%, below the size of turbo.
Whisper large-v3 is a speech recogniser. As published its weights are 3.09 GB. Our TWX4 build is 0.98 GB, smaller than the published turbo version at 1.62 GB. On nine of ten test sets it adds at most 0.3 points to the original’s error rate. On Danish it adds 1.06.
See the matching eval card for the result in one screen.
Four builds, ten test sets, one 20 GB card.
Large-v3 is the full model. Large-v3-turbo is its published faster version. We made a TWX4 build of each and measured all four the same way.
| Build | Weights | What it is |
|---|---|---|
| Whisper large-v3 as published | 3.09 GB | The reference for every error rate below |
| Whisper large-v3-turbo as published | 1.62 GB | The published faster version |
| Tileward TWX4 of large-v3 | 0.98 GB | 68% smaller than large-v3 (3.2×) |
| Tileward TWX4 of turbo | 0.56 GB | 65% smaller than turbo (2.9×) |
At most 0.3 points added on nine of ten sets, and 1.06 on Danish.
Error rate is the share of words transcribed wrong, or of characters for Mandarin and Japanese. Lower is better. Each build’s column is its change from the original, in points, with its 95% range.
| Test set | Original | TWX4 of large-v3 | Turbo as published |
|---|---|---|---|
| LibriSpeech test-clean | 1.89% | +0.01 (−0.08 to +0.09) | +0.11 (−0.07 to +0.28) |
| LibriSpeech test-other | 3.57% | +0.16 (+0.08 to +0.25) | +0.36 (+0.20 to +0.51) |
| Synthetic call-centre clips | 0.79% | −0.02 (−0.19 to +0.12) | −0.10 (−0.42 to +0.22) |
| FLEURS German | 5.93% | +0.11 (−0.03 to +0.26) | +0.43 (+0.17 to +0.70) |
| FLEURS Spanish | 4.21% | +0.09 (+0.00 to +0.17) | +0.39 (+0.25 to +0.55) |
| FLEURS Hindi | 18.48% | +0.16 (−1.19 to +1.05) | +2.04 (+0.30 to +4.01) |
| FLEURS Arabic | 16.69% | +0.27 (−0.09 to +0.62) | +0.57 (+0.05 to +1.09) |
| FLEURS Mandarin, characters | 6.23% | +0.27 (+0.05 to +0.49) | +0.49 (+0.16 to +0.82) |
| FLEURS Japanese, characters | 6.06% | −1.22 (−4.75 to +0.62) | +0.02 (−4.25 to +3.95) |
| FLEURS Danish | 14.02% | +1.06 (+0.71 to +1.38) | +2.29 (+0.96 to +4.49) |
Japanese turns on single clips: the original repeats itself to the length limit on one of them, so read that set as level.
Smaller than turbo, and level with it or ahead on every set.
The large-v3 build is 40% smaller than turbo (1.7×). Compared clip for clip, its error rate is lower on four sets and level on the other six. Turbo is the faster of the two, by about two to one on the same card.
| Test set | TWX4 of large-v3, against turbo |
|---|---|
| LibriSpeech test-other | −0.19 (−0.35 to −0.03) |
| FLEURS German | −0.32 (−0.61 to −0.03) |
| FLEURS Spanish | −0.31 (−0.48 to −0.14) |
| FLEURS Hindi | −1.88 (−3.51 to −0.72) |
| The other six sets | Level: each range spans zero |
8% more live calls for large-v3, and 10% more for turbo.
A live call here is one second of audio transcribed per second, on one 20 GB card. We sent 512 windows of 28 seconds of speech at once.
| Build | Weights | Live calls on one 20 GB card |
|---|---|---|
| Whisper large-v3 as published | 3.09 GB | 140 |
| Tileward TWX4 of large-v3 | 0.98 GB | 152, 8% more |
| Whisper large-v3-turbo as published | 1.62 GB | 301 |
| Tileward TWX4 of turbo | 0.56 GB | 331, 10% more |
Large-v3 fills this card’s memory, so a larger card holds more calls; turbo reaches the server’s limit of 256 requests first.
65% smaller than turbo, and behind it on four sets.
Our turbo build is 0.56 GB against 1.62 GB. Compared clip for clip with turbo, it is level on six sets and behind on these four, by 0.09 to 0.68 points.
| Test set | TWX4 of turbo, against turbo |
|---|---|
| LibriSpeech test-other | +0.09 (+0.02 to +0.17) |
| FLEURS Arabic | +0.68 (+0.34 to +1.03) |
| FLEURS Mandarin, characters | +0.53 (+0.31 to +0.78) |
| FLEURS Japanese, characters | +0.31 (+0.16 to +0.46) |
Large-v3 in TWX4 for accuracy, turbo in TWX4 for calls per card.
The large-v3 build adds at most 0.3 points on nine sets, at about a third of the original’s size. The turbo build carries 331 live calls on one 20 GB card, and trails turbo by up to 0.68 points on four sets.
Every build on the same card, the same server and the same clips.
Each error rate is compared with Whisper large-v3 as published, clip for clip.
The card. One 20 GB card. Every build ran on vLLM from its own weights, with the server allowed 90% of the card’s memory.
Size. The weights file. Loaded on the card, the weights take the same space.
Test sets. LibriSpeech test-clean (2,611 clips) and test-other (2,932). FLEURS test in German (841), Spanish (907), Hindi (417), Arabic (428), Mandarin (944), Japanese (650) and Danish (929). And 384 synthetic call-centre clips: scripted call lines in eight synthetic voices, each clean, through a telephone band and with noise.
Scoring. Greedy decoding, with the language given and one 30-second window a clip. Word error rate after Whisper’s own text normaliser, or character error rate for Mandarin and Japanese. Each change has a paired bootstrap 95% range from 2,000 resamples of the clips.
Live calls. 512 windows of continuous read English, 28 seconds each, sent at once, with up to 256 in progress. The figure is seconds of audio transcribed per second of wall time. Each run took 35 to 83 seconds, so start-up and tail are inside it.
Bring the audio you transcribe and the card you run it on, and we will measure the same things. Talk to an engineer, or compare the speech recognition eval card.