Benchmark

We made Qwen3.8-27B 8.5% smaller than NVIDIA’s NVFP4 build, and beyond.

Qwen3.8-27B is Qwen’s dense model of 27 billion parameters, and it reads images as well as text. As Qwen publishes it, its weights take 55.56 GB, and NVIDIA publishes a smaller NVFP4 build of 21.92 GB. We made TWFP4 at 20.05 GB: 8.5% smaller again (1.09×), and the build our hosted API serves. Against NVIDIA’s build we found no measurable difference on MMLU’s 14,042 knowledge questions, MMLU-Redux, made-up facts or tool calls. NVIDIA’s build does follow the original model more closely, picking the same next word more often. Further down, our smaller research builds, TWX3 and TWX2, are measured against TWFP4.

See the matching eval card for the result in one screen.

At a glance

Against NVIDIA’s NVFP4: smaller, similar results on our tests, more room for the cache.

Each line is TWFP4 against NVIDIA’s NVFP4 build, the smaller public version of Qwen3.8-27B, measured in the same session. Every section below starts with its own line, then gives the full measurement.

8.5%
smaller than NVIDIA’s NVFP4 build
20.05 GB against 21.92 GB
100.4%
of the NVFP4 build’s MMLU score: no measurable difference
82.4% against 82.1%
3.8%
of facts made up: no measurable difference
NVFP4 build: 4.7%
100%
of the NVFP4 build’s right tool calls: the same
240 of 240 for both
2.9%
more room for the cache on the same card
1.74 GiB less weight on it
87.8%
next-word agreement with the original model
NVFP4 build: 88.9%, closer to the original
Benchmarks

No measurable difference from NVIDIA’s NVFP4 on four tests; NVFP4 stays closer to the original.

TWFP4 against NVIDIA’s build on every test on this page, with each build’s result drawn from zero. Each card links to the full result below.

MMLU

Knowledge: 14,042 multiple-choice questions across 57 subjects, with five worked examples before each. Higher is better.

NVIDIA’s NVFP4 build
82.1%
Tileward TWFP4
82.4%
The full result: no measurable difference

Made-up facts

Each build writes out the facts in 360 passages; a fact no passage supports is made up. Lower is better.

NVIDIA’s NVFP4 build
4.7%
Tileward TWFP4
3.8%
The full result, with the counts

Tool calls

240 calls, with thinking off and then on, scored on calling the right tool the right way. Higher is better.

NVIDIA’s NVFP4 build
240 of 240
Tileward TWFP4
240 of 240
The full result

Next-word agreement

How closely each build follows the original model: at 87,955 places in 300 passages, does it pick the original’s next word? Higher is closer.

NVIDIA’s NVFP4 build
88.9%
Tileward TWFP4
87.8%
The full result
Findings · TWFP4 against NVIDIA’s NVFP4

No measurable difference from NVFP4 on MMLU, MMLU-Redux, made-up facts or tool calls.

TWFP4 and NVIDIA’s build ran in the same session on the same card, set up the way our hosted API serves the model, and we asked three things of each. Does it still know what the original knows? MMLU is 14,042 multiple-choice questions across 57 subjects, and higher is better. Does it make things up? Each build wrote out the facts in 360 passages, and we counted the facts no passage supports, so lower is better. Can it still use tools? It made 240 calls, scored on calling the right tool the right way, and higher is better. MMLU-Redux, 5,700 questions answered in writing with thinking on, ran in a session of its own.

“No measurable difference” means the gap is smaller than chance alone produces. Tap a figure for the counts behind it.

TestNVIDIA NVFP4Tileward TWFP4Against NVIDIA’s build
MMLU, 14,042 questions, higher is better82.1% (11,522 right)82.4% (11,571 right)Compared question by question, 487 answers were right only for TWFP4 and 438 only for NVIDIA’s build (p = 0.11).
MMLU-Redux, 5,700 questions, higher is better89.4%89.4%Compared question by question, 133 answers were right only for TWFP4 and 129 only for NVIDIA’s build (p = 0.85), both builds in a session of their own.
Facts made up, lower is better4.7% (81 of 1,722)3.8% (63 of 1,638)63 of 1,638 facts from TWFP4 and 81 of 1,722 from NVIDIA’s build had no passage to support them. That is no different by more than chance (p = 0.23, treating facts as independent).
Tool calls right, higher is better240 of 240240 of 240The same

TWFP4 also answered one request of 199,803 tokens and found the one fact planted in it, and it read the digits in an image correctly: the build keeps the model’s long window and its image input.

Findings · Against the original

NVIDIA’s NVFP4 build follows the original more closely: 88.9% against 87.8%.

A test score can hide small changes in how a model writes. Next-word agreement measures how closely a build follows the original model: each build reads the same 300 passages, and at each of 87,955 places we ask whether it would pick the same next word as the original. Higher is closer to the original. NVIDIA’s build agrees 88.9% of the time and TWFP4 87.8%, both builds in a session of their own.

The chart adds our other builds, each read on the same passages in the session of its MMLU pair: TWX4 follows the original most closely of any build here, 93.4%, and TWX2 least, 68.1%. Drift, under each name, is how far a build’s word choices move from the original’s; lower is closer.

Tileward TWX4drift 0.023
93.4%
NVIDIA’s NVFP4 builddrift 0.043
88.9%
Tileward TWFP4drift 0.050
87.8%
Tileward TWX3drift 0.084
85.7%
Tileward TWX2drift 0.491
68.1%
Findings · Research builds against TWFP4

TWX3 holds at 17.11 GB; TWX2 is 8.1 points behind at 14.97 GB.

We built Qwen3.8-27B in three more of our formats. Each line is one build per format, the one that measured best, with the build it was measured against in a session of its own: the same MMLU questions with the same settings for both. The sessions ran on different engine versions, so each line is read on its own: the chart sets them side by side, and does not rank them. The first chart puts all four on one scale; the second zooms in on the three that hold. TWFP4 itself read 82.4% and 82.5% in two sessions, 20 answers apart.

Tileward TWX430.96 GB · 82.3% against NVIDIA’s 82.1%
Tileward TWX4: +0.17
Tileward TWFP420.05 GB · 82.4% against NVIDIA’s 82.1%
Tileward TWFP4: +0.35
Tileward TWX317.11 GB · 82.5% against TWFP4’s 82.5%
Tileward TWX3: −0.07
Tileward TWX214.97 GB · 74.4% against TWFP4’s 82.5%
Tileward TWX2: −8.11

Closer in, from one point behind to one point ahead, the three builds that hold can be told apart.

Tileward TWX430.96 GB · 82.3% against NVIDIA’s 82.1%
Tileward TWX4: +0.17
Tileward TWFP420.05 GB · 82.4% against NVIDIA’s 82.1%
Tileward TWFP4: +0.35
Tileward TWX317.11 GB · 82.5% against TWFP4’s 82.5%
Tileward TWX3: −0.07
Tileward TWX214.97 GB · 74.4% against TWFP4’s 82.5%
← −8.11 (−8.77 to −7.46): off the scale, eight points behind TWFP4

TWX3 holds on the other tests too. In its session it made up 4.1% of the facts it wrote (70 of 1,715) against 4.4% for TWFP4 (71 of 1,627), no different by more than chance (p = 0.73), and it made 239 of 240 tool calls right against 238. It does follow the original less closely: 85.7% next-word agreement against 87.8%.

TWX2 does not hold on this model. At 14.97 GB it answers 1,139 more MMLU questions wrong than TWFP4 out of 14,042, and it picks the original’s next word 68.1% of the time. On Qwen3.6-35B-A3B, a mixture of experts, our TWX2 build was 0.85 points behind TWX4; on this dense model, TWX2 is 8.1 points behind TWFP4.

Memory · Size

TWFP4’s weights are 20.05 GB, against 21.92 GB for NVIDIA’s NVFP4 build.

Six builds of the same model: the original, NVIDIA’s NVFP4 build, and four of ours. Size here is the weights files on disk; what the weights take once loaded, and what the server needs in all, follow in the next section. A shorter bar is a smaller build, and each label carries the build’s MMLU score beside the score of the build it shared a session with. TWFP4 is 8.5% smaller than NVIDIA’s build (1.09×); TWX3 is 22% smaller than it, and TWX2 32%.

Qwen3.8-27B as publishedThe original
55.56 GB
Tileward TWX444% smaller (1.79×) · MMLU 82.3%, NVIDIA’s 82.1%
30.96 GB
NVIDIA’s NVFP4 build61% smaller (2.53×) · MMLU 82.1%, TWFP4 82.4%
21.92 GB
Tileward TWFP464% smaller (2.77×) · MMLU 82.4%, NVIDIA’s 82.1%
20.05 GB
Tileward TWX369% smaller (3.25×) · MMLU 82.5%, TWFP4 82.5%
17.11 GB
Tileward TWX273% smaller (3.71×) · MMLU 74.4%, TWFP4 82.5%
14.97 GB
Memory · Cache

1.74 GiB less on the card, and 2.9% more room for the cache.

The card has a fixed amount of memory. What the weights leave free, the server gives to the cache: the conversation the model holds while it answers. Less weight on the card is better, and a bigger cache means more people, or longer conversations, on the same card. Both builds were set up the way our hosted API serves the model, with a 262,144-token window and an FP8 cache. This is one card for each build, as the server loaded it: the weights, and the rest of the card for everything else, conversations included. Loaded, TWFP4’s weights take 18.88 GiB of the card; on disk they are 20.05 GB.

NVIDIA’s NVFP4 buildLeaves 74.97 GiB of the card
The model: 20.62 GiBThe rest of the card: 74.97 GiB
Tileward TWFP4Leaves 76.71 GiB of the card
The model: 18.88 GiBThe rest of the card: 76.71 GiB
One RTX PRO 6000 Blackwell card, 95.59 GiB as its driver reports it, and the same server for both builds.
BuildWeights on the card smaller is betterThe rest of the card more is betterTokens the cache holds more is better
NVIDIA’s NVFP4 build20.62 GiB74.97 GiB1,717,848
Tileward TWFP418.88 GiB76.71 GiB1,767,641

The least of the card TWFP4 starts and answers with, under the same settings, is 35.0 GB: the weights, the cache and the server’s own working memory together.

Which build

TWFP4 takes less of the card than NVIDIA’s build; TWX3’s weights take 17.11 GB.

Choose TWFP4 when room on the card matters and the answers must hold: there is no measurable difference from NVIDIA’s build on MMLU, MMLU-Redux, made-up facts or tool calls, and it takes 1.74 GiB less of the card, with 2.9% more room in the cache. NVIDIA’s build stays closer to the original’s wording: it picks the original’s next word more often, 88.9% of the time against 87.8%. TWFP4 is the build our hosted API serves; try it from the model library.

TWX3 and TWX2 are research builds. TWX3 keeps TWFP4’s MMLU score, made-up facts and tool calls with 17.11 GB of weights, 14.7% less than TWFP4’s, and follows the original’s wording less closely. TWX2’s weights are another 2.14 GB smaller, and it gives up 8.1 MMLU points for that. To run Qwen3.8-27B on your own card, talk to an engineer.

How it was measured

Every comparison pairs two builds from one session, on the same card and engine.

Within each session, both builds were served the same way on one card and asked the same questions. Figures from different sessions are shown side by side, but never compared. The settings for each test follow.

The card. One RTX PRO 6000 Blackwell Server Edition (96 GB) in every session, with vLLM serving each build from its own weights: 0.30.0 for TWFP4 against NVIDIA’s build on MMLU, made-up facts, tool calls and the cache, with the hosted API’s settings; 0.31.0 for TWX3 and TWX2 against TWFP4; 0.25.1 for TWX4 against NVIDIA’s build.

Size. The weights files, in decimal gigabytes.

MMLU. Five-shot, all 14,042 questions, run by lm-evaluation-harness with identical arguments for both builds of a pair and no sampling. Compared question by question with an exact McNemar test, and a 95% range for the paired difference (the normal approximation).

MMLU-Redux. 5,700 questions, one answer each, written with thinking on; an answer that does not finish counts as wrong. Compared question by question with an exact McNemar test. One reading.

Made-up facts. Each build reads 360 passages from Tileward’s own document set and writes out the facts they state; every fact is matched back to its passage, and one that no passage supports counts as made up. Compared with a two-sided Fisher test, treating facts as independent. One reading per build.

Tool calls. Six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, with thinking off and on: 240 calls, scored mechanically on tool name, call count and argument shape.

Agreement. Each build reads the same 300 passages (87,955 token positions), and we count where its top next token is the original’s, with the mean KL distance from the original’s word choices beside it. TWFP4 and NVIDIA’s build ran in one session of their own, on vLLM 0.31.0; TWX3, TWX2 and TWX4 were read in the sessions of their MMLU pairs, on the same 300 passages.

Other builds. In the session with TWX3 and TWX2, made-up facts and tool calls ran only for a build with at least 80% next-word agreement and within one MMLU point of TWFP4, a rule written before the run; TWX2 met neither. The cache figures are the memory each build’s weights took and the tokens the server set aside for the cache, as the server reported when it loaded them.

Long request and images. One prompt of 199,803 tokens with one fact planted in it, and one image with four digits to read, both sent to TWFP4 set up as the hosted API serves it.

Bring your own workload

Bring the questions, documents and tools you would give Qwen3.8-27B and the card you run on, and we will measure the same things. Talk to an engineer, or compare the 27B eval card.