# We shrank Kimi K3.1 Pro by 35%, within a point of the original on MMLU

Tileward research report R-013. Kimi K3.1 Pro is Moonshot AI's 2.8-trillion-parameter model, post-trained from Kimi K3. Even in the form Moonshot publishes, already stored in 4 bits, its weights take 1,561.3 GB, and here it ran split across eight B300 cards. We made two smaller builds, in the same two formats as our Kimi K3 builds, and tested whether they still answer as well. TWX2.5 takes 35% less space (1,008.3 GB, 1.5×) and TWX3 25% less (1,178.4 GB, 1.3×). On a 14,042-question knowledge test both scored 90.5% against the original's 91.1%, inside the one-point bar we set before the run. On every card, the memory they free holds 2.3 to 2.9 times as much conversation.

Page: https://tileward.com/research/a-smaller-kimi-k3-1-pro/

Built with: vLLM · 8× B300 · MMLU

## One card

Each of the eight cards holds one slice of the model. Whatever the model leaves free, the server gives to conversations: the text the model keeps in memory while it answers. On one card, the model as published takes 192.5 GiB and leaves 36.3 GiB for conversations; TWX3 takes 147.9 GiB and leaves 83.5 GiB; TWX2.5 takes 128.1 GiB and leaves 103.3 GiB.

## The setup

Moonshot publishes Kimi K3.1 Pro already stored in 4 bits, so every percentage here is against a model that is already compressed, not a 16-bit one. All three builds ran on the same eight B300 cards, with the same serving software, and answered the same questions.

| Build | Weights | Against the original |
|---|---|---|
| Kimi K3.1 Pro as published | 1,561.3 GB | The reference |
| Tileward TWX3 | 1,178.4 GB | 25% smaller (1.3×) |
| Tileward TWX2.5 | 1,008.3 GB | 35% smaller (1.5×) |

## Quality

We asked three things of each build. Does it still know what the original knows? [MMLU](https://arxiv.org/abs/2009.03300) is 14,042 multiple-choice questions across 57 subjects, and higher is better. Does it make things up? Each build wrote out the facts in 360 passages, and we counted the facts no passage supports, so lower is better. Can it still use tools? It made 600 calls at each of two thinking efforts, scored on calling the right tool the right way, and higher is better. Before the run we set the pass mark for MMLU: no more than one point below the original, compared question by question.

| Build | MMLU | Against the original | At worst |
|---|---|---|---|
| Kimi K3.1 Pro as published | 91.1% | The reference | – |
| Tileward TWX3 | 90.5% | 0.6 points below | 0.82 points below |
| Tileward TWX2.5 | 90.5% | 0.6 points below | 0.86 points below |

On made-up facts and tool calls both builds are level with the original.

| Test | The original | Tileward TWX3 | Tileward TWX2.5 |
|---|---|---|---|
| Facts made up | 0.0% (0 of 1,666) | 0.1% (2 of 1,565) | 0.1% (2 of 1,630) |
| Tool calls right, low thinking effort | 600 of 600 | 599 of 600 | 600 of 600 |
| Tool calls right, high thinking effort | 600 of 600 | 598 of 600 | 598 of 600 |

The counts behind the MMLU comparison: 267 answers right only for the original and 187 only for TWX3 (p = 0.0002); 301 and 218 for TWX2.5 (p = 0.0003). Between the two builds, 290 and 287 (p = 0.93). Made-up facts against the original: p = 0.23 for TWX3 and 0.24 for TWX2.5, treating facts as independent.

## Against the original

A test score can hide small changes in how a model writes. Next-word agreement is a finer check: each build reads the same 300 passages, and at each of 82,712 places we ask whether it would pick the same next word as the original. Higher is closer. Drift is the mean KL divergence from the original's word choices, where lower is closer.

| Build | Next-word agreement | Drift from the original |
|---|---|---|
| Tileward TWX3 | 91.2% | 0.038 |
| Tileward TWX2.5 | 88.4% | 0.062 |

## Memory

Every card has the same memory. What the weights leave free, the server gives to the cache: the conversation the model holds while it answers. A bigger cache means more people, or longer conversations, on the same cards. The server was allowed 90% of each card for every build.

| Build | Weights on each card | Room for the cache | Tokens the cache holds |
|---|---|---|---|
| Kimi K3.1 Pro as published | 192.5 GiB | 36.3 GiB | 2,135,421 |
| Tileward TWX3 | 147.9 GiB | 83.5 GiB | 4,915,678 |
| Tileward TWX2.5 | 128.1 GiB | 103.3 GiB | 6,086,715 |

## Which build

Choose TWX3 when matching the original matters most: it picks the original's next word more often. Choose TWX2.5 when room matters most: it carries 170 GB less weight than TWX3 and has 24% more room in the cache. On MMLU the two score the same.

## How it was measured

The original and both builds were served the same way, one after another on one machine, and asked the same questions.

- **Machine:** eight B300 cards, with vLLM's own image for Kimi K3.1 serving each build from its own weights, split across all eight, with the engine's 8-bit cache and 90% of each card.
- **Size:** the weights files, in decimal gigabytes.
- **MMLU:** five-shot, no sampling, scored by the answer letter the model rates most likely. The paired test counts the answers that differ.
- **Made-up facts:** each build reads 360 passages from Tileward's own document set and writes out the facts they state. Every fact is matched back to its passage, and one that no passage supports counts as made up.
- **Tool calls:** six cases, 20 trials each, run five times: 600 calls, once at low thinking effort and once at high. Each call is scored on tool name, call count and argument shape, with vLLM's Kimi K3 tool parser.
- **Agreement:** each build reads the same 300 passages (82,712 token positions), and we count where its top next token is the original's.
- **The model:** Kimi K3.1 Pro is Moonshot AI's, released under the [Kimi K3.1 Pro License](https://huggingface.co/moonshotai/Kimi-K3.1-Pro/blob/main/LICENSE).

## Related

- [Kimi K3.1 Pro eval card](https://tileward.com/evals/#compressed-model-k31-pro): the result in one screen
- [Kimi K3 write-up](https://tileward.com/research/a-smaller-kimi-k3/): the same two formats on Kimi K3
- [Kimi K3.1 Pro on Hugging Face](https://huggingface.co/moonshotai/Kimi-K3.1-Pro): the published model
- [Talk to an engineer](https://calendar.app.google/hg2DzFTxKJgwFWm19): bring the work you would run on Kimi K3.1 Pro and the cards you run on
- [Research](https://tileward.com/research/): every report