24% smaller, and within half a point of the original on MMLU Tileward DeepSeek-V4-Pro against DeepSeek-V4-Pro as DeepSeek publishes it, already mostly 4-bit
Nearly a quarter smaller, within half a point on MMLU, and level on made-up facts, tool calls and million-token passages. It holds 1.2× the original’s cache on the same eight cards, and picks the original’s next word 87.6% of the time, 92.8% past the first 4,096 tokens of a long passage.
- The original is the release DeepSeek publishes, with most of its weights already stored in 4 bits, so the 24% is against a mostly 4-bit model, not a 16-bit one.
- MMLU: five-shot, 14,042 questions, no sampling, scored by the answer letter the model rates most likely. The bar, set before the run: no more than one point below the original on the paired test. 247 answers right only for the original and 192 only for TWX3 (p = 0.0099; at most 0.64 points below).
- Fabrication: each build reads 360 passages from Tileward’s own document set and writes out the facts they state; every fact is matched back to its passage, and one that no passage supports counts as made up. The original wrote 1,782 facts and 32 were not supported; TWX3 wrote 1,825 and 34 were not (p = 0.90, treating facts as independent).
- Tool calls: six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, run five times and scored mechanically on tool name, call count and argument shape: 600 calls at each thinking effort. Both builds made 600 right at low effort and 600 at high.
- Hidden facts: ten short facts in each of two passages at 32,768, 131,072, 524,288 and 1,040,000 tokens, asked for in one answer; both builds found all 80.
- Agreement: teacher-forced on 300 corpus chunks (82,492 positions), top-1 agreement with the original 87.6%, mean KL 0.079; past the first 4,096 tokens of 28 long passages (1,196,032 positions), 92.8%, KL 0.044.
- Bars are weights on disk; loaded, 83.5 GiB on each card for TWX3 and 108.9 for the original. With 90% of each card the cache holds 6,942,711 and 5,707,924 tokens. Every figure on this card is from one session on eight B300 cards and vLLM. DeepSeek-V4-Pro is DeepSeek’s, under the MIT License.