# We shrank DeepSeek-V4-Pro by 24%, within half a point of the original on MMLU

Tileward research report R-016. DeepSeek-V4-Pro is DeepSeek's model built to read up to a million tokens at once. Even in the form DeepSeek publishes, with most of its weights already stored in 4 bits, it takes 892.7 GB, and here it ran split across eight B300 cards. We made a smaller build, TWX3, in the same format as our Kimi K3.1 Pro build of that name, and tested whether it still answers as well. It takes 24% less space (675.2 GB, 1.3×). On a 14,042-question knowledge test it scored 89.6% against the original's 90.0%, inside the one-point bar we set before the run, and it found every fact we hid in passages up to 1,040,000 tokens long. On every card, the memory it frees holds 1.2 times as much conversation, and it needs four B300 cards where the original needs eight.

Page: https://tileward.com/research/a-smaller-deepseek-v4-pro/

Built with: vLLM · 8× B300 · MMLU

## At a glance

Against DeepSeek-V4-Pro as published: smaller, within half a point, more room. DeepSeek's own release is the version of DeepSeek-V4-Pro people run today, so each line here is our build against it. Every section below starts with its own line, then gives the full measurement.

| Against DeepSeek-V4-Pro as published | Tileward TWX3 |
|---|---|
| Size of the weights | Smaller by 24%: 675.2 GB against 892.7 GB |
| Knowledge, on MMLU | Within half a point: 99.6% of the original's score, 89.6% against 90.0% |
| Facts made up | Level: 1.9% against 1.8% |
| Tool calls right, at both thinking efforts | Level: 100% of the original's, 600 of 600 at each, against 600 of 600 |
| Hidden facts found, out to 1,040,000 tokens | Level: 100% of the original's, 80 of 80 against 80 of 80 |
| B300 cards to serve it, with room for a million-token conversation | Half as many: 4 cards against 8 |
| Room for the cache on each card | More: 141.4 GiB against 116.2 GiB, 1.2× the room |

## Benchmarks

Within half a point on MMLU; level on facts, tools and long passages. Every test on this page, with each build's score drawn from zero. Each card links to the full result below.

- **MMLU:** Knowledge: 14,042 multiple-choice questions across 57 subjects. Higher is better. DeepSeek-V4-Pro as published 90.0%; Tileward TWX3 89.6%.
- **Made-up facts:** Each build writes out the facts in 360 passages; a fact no passage supports is made up. Lower is better. DeepSeek-V4-Pro as published 1.8%; Tileward TWX3 1.9%.
- **Tool calls:** 600 calls at each of two thinking efforts, scored on calling the right tool the right way. Higher is better. DeepSeek-V4-Pro as published, low effort 600 of 600; DeepSeek-V4-Pro as published, high effort 600 of 600; Tileward TWX3, low effort 600 of 600; Tileward TWX3, high effort 600 of 600.
- **Hidden facts, long passages:** 80 short facts hidden in passages of 32,768 to 1,040,000 tokens, and each build asked for all of them back. Higher is better. DeepSeek-V4-Pro as published 80 of 80; Tileward TWX3 80 of 80.
- **Next-word agreement:** TWX3 reads 300 short passages and 28 long ones, and at each place we ask whether it picks the original's next word. Higher is closer. Tileward TWX3, short passages 87.6%; Tileward TWX3, long passages 92.8%.

## Findings · Quality

Within half a point on MMLU, and level on made-up facts and tool calls. We asked three things of TWX3. Does it still know what the original knows? [MMLU](https://arxiv.org/abs/2009.03300) is 14,042 multiple-choice questions across 57 subjects, and higher is better. Does it make things up? Each build wrote out the facts in 360 passages, and we counted the facts no passage supports, so lower is better. Can it still use tools? It made 600 calls at each of two thinking efforts, scored on calling the right tool the right way, and higher is better.

Before the run we set the pass mark for MMLU: no more than one point below the original, compared question by question.

| Build | MMLU, higher is better | Against the original, closer to zero is better | At worst, closer to zero is better |
|---|---|---|---|
| DeepSeek-V4-Pro as published | 90.0% | The reference | – |
| Tileward TWX3 | 89.6% | 0.4 points below | 0.64 points below |

On made-up facts and tool calls TWX3 is level with the original.

| Test | The original | Tileward TWX3 |
|---|---|---|
| Facts made up, lower is better | 1.8% (32 of 1,782) | 1.9% (34 of 1,825) |
| Tool calls right, low thinking effort, higher is better | 600 of 600 | 600 of 600 |
| Tool calls right, high thinking effort, higher is better | 600 of 600 | 600 of 600 |

The counts behind the MMLU comparison: 247 answers right only for the original and 192 only for TWX3 (p = 0.0099). Made-up facts against the original: p = 0.90, treating facts as independent.

## Findings · Long passages

TWX3 found all 80 hidden facts out to 1,040,000 tokens, as the original did. DeepSeek-V4-Pro reads up to a million tokens at once, so we tested whether TWX3 still finds what is buried deep in a long passage. At each of four lengths we hid 10 short facts in a passage, twice, and asked for all of them back: 20 facts at each length, 80 in all. Higher is better.

| Passage length | The original, higher is better | Tileward TWX3, higher is better |
|---|---|---|
| 32,768 tokens | 20 of 20 | 20 of 20 |
| 131,072 tokens | 20 of 20 | 20 of 20 |
| 524,288 tokens | 20 of 20 | 20 of 20 |
| 1,040,000 tokens | 20 of 20 | 20 of 20 |

Long passages also keep TWX3 closer to the original word by word: next-word agreement is higher past the first 4,096 tokens of a long passage than in short ones.

## Findings · Against the original

TWX3 picks the original's next word 87.6% of the time, 92.8% in long passages. A test score can hide small changes in how a model writes. Next-word agreement is a finer check: TWX3 reads the same passages as the original, and at each place we ask whether it would pick the same next word. Higher is closer to the original. The last column measures how far its word choices drift from the original's, where lower is closer.

| Passages | Places compared | Next-word agreement, higher is closer | Drift from the original, lower is closer |
|---|---|---|---|
| 300 short passages | 82,492 | 87.6% | 0.079 |
| 28 long passages, past the first 4,096 tokens | 1,196,032 | 92.8% | 0.044 |

The long passages were 32,768 and 131,072 tokens: agreement past the first 4,096 tokens was 93.0% and 92.4%.

## Memory · Size

TWX3 is 24% smaller than DeepSeek-V4-Pro as published. DeepSeek publishes DeepSeek-V4-Pro with most of its weights already stored in 4 bits, so the percentage is against a model that is already compressed, not a 16-bit one. Both builds ran on the same eight B300 cards, with the same serving software, and answered the same questions.

| Build | Weights, smaller is better | Against the original, more is better |
|---|---|---|
| DeepSeek-V4-Pro as published | 892.7 GB | The reference |
| Tileward TWX3 | 675.2 GB | 24% smaller (1.3×) |

## Memory · One card

TWX3 takes 83.5 GiB of each card; the original takes 108.9. Each of the eight cards holds one slice of the model. Whatever the model leaves free, the server gives to conversations: the text the model keeps in memory while it answers. On one card, the model as published takes 108.9 GiB and leaves 116.2 GiB for conversations, enough for 5,707,924 tokens; TWX3 takes 83.5 GiB and leaves 141.4 GiB, enough for 6,942,711 tokens (1.2×). The same card, the same server, and 90% of the card for both builds.

## Memory · Cache

25.4 GiB less on every card, and 1.2× the room for the cache. Every card has the same memory. What the weights leave free, the server gives to the cache: the conversation the model holds while it answers. A bigger cache means more people, or longer conversations, on the same cards. At a million tokens each, TWX3 has room for six whole conversations at once, where the original has room for five. The server was allowed 90% of each card for both builds.

| Build | Weights on each card, smaller is better | Room for the cache, more is better | Tokens the cache holds, more is better |
|---|---|---|---|
| DeepSeek-V4-Pro as published | 108.9 GiB | 116.2 GiB | 5,707,924 |
| Tileward TWX3 | 83.5 GiB | 141.4 GiB | 6,942,711 |

## Memory · Fewer cards

TWX3 runs on four B300 cards; the original needs eight. How few cards can serve each build, with room for at least one whole conversation of 1,048,576 tokens? We measured both on eight B300 cards; every other count here is worked out from that run's own figures, not run. As in the run, the server gets 90% of each card. The weights are split across the cards, about 5.7 GiB of them sit whole on every card, the server keeps about 16.7 GiB of each card for its own work, and one whole conversation needs 21.4 GiB on every card. Fewer cards is better.

One card of each build, at the fewest cards that serve it. One B300 card gives the server 241.7 GiB: as published, on 8 cards (measured), the model takes 108.9 GiB of it and leaves 116.2 GiB for conversations, room for 5 whole ones; TWX3 on 4 cards takes 161.4 GiB and leaves 63.6 GiB, room for 2. One B200 card gives it 160.5 GiB: as published on 8 cards, 108.9 GiB and 34.9 GiB, room for 1; TWX3 on 8 cards, 83.5 GiB and 60.3 GiB, room for 2. One RTX PRO 6000 Blackwell card gives it 86.0 GiB: as published on 32 cards, 31.5 GiB and 37.9 GiB, room for 1; TWX3 on 16 cards, 44.6 GiB and 24.8 GiB, room for 1. The rest of each card is the server's own share.

| Card | DeepSeek-V4-Pro as published, fewer is better | Tileward TWX3, fewer is better |
|---|---|---|
| B300, 288 GB | 8 cards, measured | 4 cards |
| B200, 180 GB | 8 cards | 8 cards, with room for 2 whole conversations against 1 |
| RTX PRO 6000 Blackwell, 96 GB | 32 cards | 16 cards |

Sixteen or thirty-two cards means two or four servers of eight. A conversation shorter than a million tokens needs less room, so a build can run on fewer cards than these when its conversations are shorter.

## Which build

TWX3 is the build to run; two earlier builds fell short. TWX3 is the one build of DeepSeek-V4-Pro we stand behind: 24% smaller, within half a point on MMLU, and level on everything else we measured. It is in the [model library](https://tileward.com/model-library/), ready to serve on request. Two earlier builds did not make it. The first, a little larger than TWX3, fell 0.02 points short of the one-point bar on MMLU: at worst it could be 1.02 points below the original. The second, smaller still, failed outright on MMLU, tool calls and long passages. Neither is offered.

## How it was measured

Both builds ran in one session, on the same machine and engine. The original and TWX3 were served the same way, one after the other on one machine, and asked the same questions.

- **Machine:** eight B300 cards, with vLLM serving each build from its own weights, split across all eight, with the engine's 8-bit cache, a window of 1,048,576 tokens and 90% of each card.
- **Size:** the weights files, in decimal gigabytes.
- **MMLU:** five-shot, no sampling, scored by the answer letter the model rates most likely. The paired test counts the answers that differ.
- **Made-up facts:** each build reads 360 passages from Tileward's own document set and writes out the facts they state. Every fact is matched back to its passage, and one that no passage supports counts as made up.
- **Tool calls:** six cases, 20 trials each, run five times: 600 calls, once at low thinking effort and once at high. Each call is scored on tool name, call count and argument shape, with vLLM's DeepSeek-V4 tool parser.
- **Hidden facts:** ten short facts placed at even depths through a passage of filler text, two passages at each of 32,768, 131,072, 524,288 and 1,040,000 tokens, asked for in one answer with thinking off and no sampling.
- **Agreement:** each build reads the same 300 short passages (82,492 token positions) and 28 long ones of 32,768 and 131,072 tokens (1,310,692 positions, 1,196,032 of them past the first 4,096), and we count where its top next token is the original's.
- **The model:** DeepSeek-V4-Pro is DeepSeek's, released under the [MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813/blob/main/LICENSE).

## Related

- [DeepSeek-V4-Pro eval card](https://tileward.com/evals/#compressed-model-dsv4-pro): the result in one screen
- [Kimi K3.1 Pro write-up](https://tileward.com/research/a-smaller-kimi-k3-1-pro/): TWX3 on another model published in 4 bits
- [DeepSeek-V4-Pro on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813): the published model
- [Talk to an engineer](https://calendar.app.google/hg2DzFTxKJgwFWm19): bring the work you would run on DeepSeek-V4-Pro and the cards you run on
- [Research](https://tileward.com/research/): every report