# We made Qwen3.8-27B 8.5% smaller than NVIDIA's NVFP4 build, and beyond Tileward research report R-019. Qwen3.8-27B is Qwen's dense model of 27 billion parameters, and it reads images as well as text. As Qwen publishes it, its weights take 55.56 GB, and NVIDIA publishes a smaller NVFP4 build of 21.92 GB. We made TWFP4 at 20.05 GB: 8.5% smaller again (1.09×), and the build our hosted API serves. Against NVIDIA's build we found no measurable difference on MMLU's 14,042 knowledge questions, MMLU-Redux, made-up facts or tool calls. NVIDIA's build does follow the original model more closely, picking the same next word more often. Further down, our smaller research builds, TWX3 and TWX2, are measured against TWFP4. Page: https://tileward.com/research/a-smaller-qwen3-8-27b/ Built with: vLLM · RTX PRO 6000 Blackwell · MMLU · lm-evaluation-harness ## At a glance Against NVIDIA's NVFP4: smaller, similar results on our tests, more room for the cache. Each line is TWFP4 against NVIDIA's NVFP4 build, the smaller public version of Qwen3.8-27B, measured in the same session. Every section below starts with its own line, then gives the full measurement. | Against NVIDIA's NVFP4 build | Tileward TWFP4 | |---|---| | Size of the weights on disk | Smaller by 8.5%: 20.05 GB against 21.92 GB | | Knowledge on MMLU | No measurable difference: 100.4% of the NVFP4 build's score, 82.4% against 82.1% | | Facts made up | No measurable difference: 3.8% against 4.7% | | Tool calls right | The same: 100% of the NVFP4 build's, 240 of 240 for both | | Room for the cache on the card | More by 2.9%, with 1.74 GiB less weight on the card | | Next-word agreement with the original model | Lower, NVIDIA's build is closer to the original: 87.8% against 88.9% | ## Benchmarks No measurable difference from NVIDIA's NVFP4 on four tests; NVFP4 stays closer to the original. TWFP4 against NVIDIA's build on every test on this page, with each build's result drawn from zero. Each card links to the full result below. - **MMLU:** Knowledge: 14,042 multiple-choice questions across 57 subjects, with five worked examples before each. Higher is better. NVIDIA's NVFP4 build 82.1%; Tileward TWFP4 82.4%. - **MMLU-Redux:** Knowledge again: 5,700 questions, each answered in writing with thinking on. Higher is better. NVIDIA's NVFP4 build 89.4%; Tileward TWFP4 89.4%. - **Made-up facts:** Each build writes out the facts in 360 passages; a fact no passage supports is made up. Lower is better. NVIDIA's NVFP4 build 4.7%; Tileward TWFP4 3.8%. - **Tool calls:** 240 calls, with thinking off and then on, scored on calling the right tool the right way. Higher is better. NVIDIA's NVFP4 build 240 of 240; Tileward TWFP4 240 of 240. - **Next-word agreement:** How closely each build follows the original model: at 87,955 places in 300 passages, does it pick the original's next word? Higher is closer. NVIDIA's NVFP4 build 88.9%; Tileward TWFP4 87.8%. ## Findings · TWFP4 against NVIDIA's NVFP4 No measurable difference from NVFP4 on MMLU, MMLU-Redux, made-up facts or tool calls. TWFP4 and NVIDIA's build ran in the same session on the same card, set up the way our hosted API serves the model, and we asked three things of each. Does it still know what the original knows? [MMLU](https://arxiv.org/abs/2009.03300) is 14,042 multiple-choice questions across 57 subjects, and higher is better. Does it make things up? Each build wrote out the facts in 360 passages, and we counted the facts no passage supports, so lower is better. Can it still use tools? It made 240 calls, scored on calling the right tool the right way, and higher is better. MMLU-Redux, 5,700 questions answered in writing with thinking on, ran in a session of its own. "No measurable difference" means the gap is smaller than chance alone produces. | Test | NVIDIA NVFP4 | Tileward TWFP4 | Against NVIDIA's build | |---|---|---|---| | MMLU, 14,042 questions, higher is better | 82.1% (11,522 right) | 82.4% (11,571 right) | No measurable difference | | MMLU-Redux, 5,700 questions, higher is better | 89.4% | 89.4% | No measurable difference | | Facts made up, lower is better | 4.7% (81 of 1,722) | 3.8% (63 of 1,638) | No measurable difference | | Tool calls right, higher is better | 240 of 240 | 240 of 240 | The same | The counts behind the comparisons: on MMLU, 487 answers right only for TWFP4 and 438 only for NVIDIA's build (p = 0.11); on MMLU-Redux, 133 and 129 (p = 0.85), both builds in a session of their own. Made-up facts against NVIDIA's build: p = 0.23, treating facts as independent. TWFP4 also answered one request of 199,803 tokens and found the one fact planted in it, and it read the digits in an image correctly: the build keeps the model's long window and its image input. ## Findings · Against the original NVIDIA's NVFP4 build follows the original more closely: 88.9% against 87.8%. A test score can hide small changes in how a model writes. Next-word agreement measures how closely a build follows the original model: each build reads the same 300 passages, and at each of 87,955 places we ask whether it would pick the same next word as the original. Higher is closer to the original. NVIDIA's build agrees 88.9% of the time and TWFP4 87.8%, both builds in a session of their own. The chart adds our other builds, each read on the same passages in the session of its MMLU pair: TWX4 follows the original most closely of any build here, 93.4%, and TWX2 least, 68.1%. Drift is how far a build's word choices move from the original's, as mean KL distance; lower is closer, and 0 would mean the same word choices as the original. | Build | Next-word agreement, higher is closer | Drift from the original, lower is closer | Where it was read | |---|---|---|---| | Tileward TWX4 | 93.4% | 0.023 | Read beside NVIDIA's NVFP4 build, which agreed 89.1% there | | NVIDIA's NVFP4 build | 88.9% | 0.043 | Read beside Tileward TWFP4 | | Tileward TWFP4 | 87.8% | 0.050 | Read beside NVIDIA's NVFP4 build | | Tileward TWX3 | 85.7% | 0.084 | Read beside Tileward TWFP4, which agreed 87.8% there | | Tileward TWX2 | 68.1% | 0.491 | Read beside Tileward TWFP4, which agreed 87.8% there | ## Findings · Research builds against TWFP4 TWX3 holds at 17.11 GB; TWX2 is 8.1 points behind at 14.97 GB. We built Qwen3.8-27B in three more of our formats. Each line is one build per format, the one that measured best, with the build it was measured against in a session of its own: the same MMLU questions with the same settings for both. The sessions ran on different engine versions, so each line is read on its own, side by side and not ranked. The page draws all four on one scale, then zooms in on the three that hold. TWFP4 itself read 82.4% and 82.5% in two sessions, 20 answers apart. | Build | Weights, smaller is better | MMLU, higher is better | Measured against, same session | Points from it (95% range) | Reading | |---|---|---|---|---|---| | Tileward TWX4 | 30.96 GB | 82.3% | NVIDIA's NVFP4 build, 82.1% | +0.17 (−0.25 to +0.59) | Not separable, 0.17 points above: 469 right only for TWX4, 445 only for NVIDIA's build, p = 0.45; one RTX PRO 6000 Blackwell card, vLLM 0.25.1 | | Tileward TWFP4 | 20.05 GB | 82.4% | NVIDIA's NVFP4 build, 82.1% | +0.35 (−0.08 to +0.77) | Not separable, 0.35 points above: 487 and 438, p = 0.11; this page's main session, vLLM 0.30.0 | | Tileward TWX3 | 17.11 GB | 82.5% | Tileward TWFP4, 82.5% | −0.07 (−0.50 to +0.36) | Not separable, 0.07 points below: 469 right only for TWX3, 479 only for TWFP4, p = 0.77; one RTX PRO 6000 Blackwell card, vLLM 0.31.0 | | Tileward TWX2 | 14.97 GB | 74.4% | Tileward TWFP4, 82.5% | −8.11 (−8.77 to −7.46) | Behind by 8.1 points: 582 right only for TWX2, 1,721 only for TWFP4, p < 0.0001; the same session as TWX3 | TWX3 holds on the other tests too. In its session it made up 4.1% of the facts it wrote (70 of 1,715) against 4.4% for TWFP4 (71 of 1,627), no different by more than chance (p = 0.73), and it made 239 of 240 tool calls right against 238. It does follow the original less closely: 85.7% next-word agreement against 87.8%. TWX2 does not hold on this model. At 14.97 GB it answers 1,139 more MMLU questions wrong than TWFP4 out of 14,042, and it picks the original's next word 68.1% of the time. On Qwen3.6-35B-A3B, a mixture of experts, [our TWX2 build was 0.85 points behind TWX4](https://tileward.com/research/a-16gb-qwen3-6-35b-a3b/); on this dense model, TWX2 is 8.1 points behind TWFP4. ## Memory · Size TWFP4's weights are 20.05 GB, against 21.92 GB for NVIDIA's NVFP4 build. Six builds of the same model: [the original](https://huggingface.co/Qwen/Qwen3.8-27B), [NVIDIA's NVFP4 build](https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4), and four of ours. Size here is the weights files on disk; what the weights take once loaded, and what the server needs in all, follow in the next section. Each build's size sits beside its MMLU score and the score of the build it shared a session with; smaller leaves more of the card free. TWFP4 is 8.5% smaller than NVIDIA's build (1.09×); TWX3 is 22% smaller than it, and TWX2 32%. | Build | Weights on disk, smaller is better | Against the original, more is better | MMLU, against the build it shared a session with | |---|---|---|---| | Qwen3.8-27B as published | 55.56 GB | The original | – | | Tileward TWX4 | 30.96 GB | 44% smaller (1.79×) | 82.3% against NVIDIA's 82.1%: not separable | | NVIDIA's NVFP4 build | 21.92 GB | 61% smaller (2.53×) | 82.1% against TWFP4's 82.4%: not separable | | Tileward TWFP4 | 20.05 GB | 64% smaller (2.77×) | 82.4% against NVIDIA's 82.1%: not separable | | Tileward TWX3 | 17.11 GB | 69% smaller (3.25×) | 82.5% against TWFP4's 82.5%: not separable | | Tileward TWX2 | 14.97 GB | 73% smaller (3.71×) | 74.4% against TWFP4's 82.5%: 8.1 points behind | ## Memory · Cache 1.74 GiB less on the card, and 2.9% more room for the cache. The card has a fixed amount of memory. What the weights leave free, the server gives to the cache: the conversation the model holds while it answers. Less weight on the card is better, and a bigger cache means more people, or longer conversations, on the same card. Both builds were set up the way our hosted API serves the model, with a 262,144-token window and an FP8 cache. One RTX PRO 6000 Blackwell card is 95.59 GiB as its driver reports it: the weights, and the rest of the card for everything else, conversations included. Loaded, TWFP4's weights take 18.88 GiB of the card; on disk they are 20.05 GB. | Build | Weights on the card, smaller is better | The rest of the card, more is better | Tokens the cache holds, more is better | |---|---|---|---| | NVIDIA's NVFP4 build | 20.62 GiB | 74.97 GiB | 1,717,848 | | Tileward TWFP4 | 18.88 GiB | 76.71 GiB | 1,767,641 | The least of the card TWFP4 starts and answers with, under the same settings, is 35.0 GB: the weights, the cache and the server's own working memory together. ## Which build TWFP4 takes less of the card than NVIDIA's build; TWX3's weights take 17.11 GB. Choose TWFP4 when room on the card matters and the answers must hold: there is no measurable difference from NVIDIA's build on MMLU, MMLU-Redux, made-up facts or tool calls, and it takes 1.74 GiB less of the card, with 2.9% more room in the cache. NVIDIA's build stays closer to the original's wording: it picks the original's next word more often, 88.9% of the time against 87.8%. TWFP4 is the build our hosted API serves; try it from the [model library](https://tileward.com/model-library/). TWX3 and TWX2 are research builds. TWX3 keeps TWFP4's MMLU score, made-up facts and tool calls with 17.11 GB of weights, 14.7% less than TWFP4's, and follows the original's wording less closely. TWX2's weights are another 2.14 GB smaller, and it gives up 8.1 MMLU points for that. ## How it was measured Every comparison pairs two builds from one session, on the same card and engine. Within each session, both builds were served the same way on one card and asked the same questions. Figures from different sessions are shown side by side, but never compared. - **The card:** one RTX PRO 6000 Blackwell Server Edition (96 GB) in every session, with vLLM serving each build from its own weights: 0.30.0 for TWFP4 against NVIDIA's build on MMLU, made-up facts, tool calls and the cache, with the hosted API's settings; 0.31.0 for TWX3 and TWX2 against TWFP4; 0.25.1 for TWX4 against NVIDIA's build. - **Size:** the weights files, in decimal gigabytes. - **MMLU:** five-shot, all 14,042 questions, run by [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) with identical arguments for both builds of a pair and no sampling. Compared question by question with an exact McNemar test, and a 95% range for the paired difference (the normal approximation). - **MMLU-Redux:** 5,700 questions, one answer each, written with thinking on; an answer that does not finish counts as wrong. Compared question by question with an exact McNemar test. One reading. - **Made-up facts:** each build reads 360 passages from Tileward's own document set and writes out the facts they state; every fact is matched back to its passage, and one that no passage supports counts as made up. Compared with a two-sided Fisher test, treating facts as independent. One reading per build. - **Tool calls:** six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, with thinking off and on: 240 calls, scored mechanically on tool name, call count and argument shape. - **Agreement:** each build reads the same 300 passages (87,955 token positions), and we count where its top next token is the original's, with the mean KL distance from the original's word choices beside it. TWFP4 and NVIDIA's build ran in one session of their own, on vLLM 0.31.0; TWX3, TWX2 and TWX4 were read in the sessions of their MMLU pairs, on the same 300 passages. - **Other builds:** in the session with TWX3 and TWX2, made-up facts and tool calls ran only for a build with at least 80% next-word agreement and within one MMLU point of TWFP4, a rule written before the run; TWX2 met neither. The cache figures are the memory each build's weights took and the tokens the server set aside for the cache, as the server reported when it loaded them. - **Long request and images:** one prompt of 199,803 tokens with one fact planted in it, and one image with four digits to read, both sent to TWFP4 set up as the hosted API serves it. ## Related - [27B eval card](https://tileward.com/evals/#compressed-model-27b): the result in one screen - [Qwen3.6-35B-A3B at 16 GB](https://tileward.com/research/a-16gb-qwen3-6-35b-a3b/): TWX2 on a mixture of experts - [Qwen3.8-27B on Hugging Face](https://huggingface.co/Qwen/Qwen3.8-27B): the published model - [Talk to an engineer](https://calendar.app.google/hg2DzFTxKJgwFWm19): bring the questions, documents and tools you would give Qwen3.8-27B and the card you run on - [Research](https://tileward.com/research/): every report