Benchmark

We shrank Qwen3.6-35B-A3B to 16 GB, 31% smaller than NVIDIA’s NVFP4 build.

Qwen3.6-35B-A3B is Qwen’s model of 35 billion parameters, about 3 billion of which work on each word. As Qwen publishes it, its weights take 71.90 GB, and our TWX4 build in the model library takes 24.10 GB. We made TWX2 at 16.05 GB: 31% smaller than NVIDIA’s NVFP4 build of the same model (23.42 GB), 33% smaller than TWX4, and 78% smaller than the original (4.5×). Against TWX4, in one session on one RTX PRO 6000 Blackwell card, it scored 81.9% on MMLU’s 14,042 questions against 82.8%: 0.85 points behind. It made up 3.2% of the facts it wrote, against 2.2%. On the card the model takes 13.5 GiB, where TWX4 takes 21.0. NVIDIA’s build was measured against TWX4 too, in its own session: 0.36 points below it on MMLU, 82.6% against 82.9%, too small a gap for the test to separate. Below, TWX2’s size and MMLU score sit beside it and three other builds of the same model, each measured against TWX4 in its own session.

See the matching eval card for the result in one screen.

At a glance

31% smaller than NVIDIA’s NVFP4 build, and 0.85 MMLU points behind TWX4.

Size is against NVIDIA’s NVFP4 build of Qwen3.6-35B-A3B. Quality is against TWX4, the build in our model library, in one session on one card. NVIDIA’s build was measured against TWX4 in a session of its own, so its line here is that pair. Every section below starts with its own line, then gives the full measurement.

31%
smaller than NVIDIA’s NVFP4 build
16.05 GB against 23.42 GB
98.9%
of the TWX4 build’s MMLU score: 0.85 points behind
81.9% against 82.8%
99.6%
of the TWX4 build’s MMLU score for NVIDIA’s NVFP4 build, in its own session: 0.36 points below
82.6% against 82.9%
3.2%
of the facts it wrote made up: 1.0 point more than TWX4
TWX4: 2.2%
97.0%
of the TWX4 build’s tool calls right, thinking off and on: 7 fewer
229 of 240 against 236 of 240
7.5 GiB
less of the card for the model than TWX4
13.5 GiB against 21.0 GiB
Benchmarks

Against TWX4: 0.85 MMLU points behind, 1.0 point more facts made up.

Every test on this page, with each build’s result drawn from zero: TWX2 also got 7 fewer of 240 tool calls right. Each card links to the full result below.

MMLU

Knowledge: 14,042 multiple-choice questions across 57 subjects, with five worked examples before each. Higher is better.

Tileward TWX4
82.8%
Tileward TWX2
81.9%
The full result, with the counts

Made-up facts

Each build writes out the facts stated in 360 passages, over three different draws of passages; a fact no passage supports counts as made up. Lower is better.

Tileward TWX4
2.2%
Tileward TWX2
3.2%
The full result, draw by draw

Tool calls

Six kinds of call, 20 tries each, with thinking off and then on. More is better.

Tileward TWX4Thinking off
118 of 120
Tileward TWX2Thinking off
117 of 120
Tileward TWX4Thinking on
118 of 120
Tileward TWX2Thinking on
112 of 120
The full result, kind of call by kind of call
Findings · Quality

TWX2 is 0.85 MMLU points behind TWX4, and makes up 1.0 point more facts.

We asked three things of TWX2, each against TWX4 on the same card in the same session. Does it still know as much? MMLU asks 14,042 multiple-choice questions across 57 subjects, and higher is better. Does it make facts up? Each build writes out the facts in 360 passages, three times over, and fewer made up is better. Can it still call tools? Six kinds of call, 20 tries each, and more right is better.

How far behind. On MMLU, 0.85 points is 119 more questions wrong out of 14,042: TWX2 answered 11,504 right and TWX4 11,623, so TWX2 keeps 98.9% of TWX4’s score, and the 95% range for the gap runs from 0.30 to 1.39 points. On made-up facts, 3.2% against 2.2% is 1.0 point more: about one fact in 31 made up, against one in 45 for TWX4. On tool calls, TWX2 got 229 of 240 right against 236, 7 fewer: one with thinking off and six with it on, most of them on nested arguments (14 of 20 against 19).

MMLU, compared question by question. Before the run we wrote down that a difference would be called behind or ahead only if the paired test separated the builds, and not separable otherwise. Tap the reading for the counts.

TestTileward TWX4Tileward TWX2Against TWX4
MMLU, 14,042 questions, higher is better82.8% (11,623 right)81.9% (11,504 right)823 questions right only for TWX4 and 704 only for TWX2 (exact McNemar p = 0.0025). The 95% range for the difference runs from 0.30 to 1.39 points behind.

Made-up facts, over three draws of 360 passages each. Both builds read the same passages in each draw, and the first draw is the one every earlier reading of this model used. TWX2 made up more on all three, by 0.8, 1.8 and 0.4 points; alone, only the second draw separates the builds, and the three pooled do, 1.0 point apart.

DrawTileward TWX4 lower is betterTileward TWX2 lower is betterTwo-sided Fisher test
First2.3% (39 of 1,672)3.1% (49 of 1,585)p = 0.20
Second2.2% (36 of 1,611)4.0% (62 of 1,551)p = 0.0053
Third2.1% (37 of 1,741)2.5% (40 of 1,616)p = 0.56
All three2.2% (112 of 5,024)3.2% (151 of 4,752)p = 0.0040

Tool calls, 20 tries of each kind, with thinking off and on. These are counts from one pass, with no test applied. Over both settings TWX2 made 229 of the 240 calls right, against 236 for TWX4, and most of its misses are on nested arguments.

Kind of callTileward TWX4, off more is betterTileward TWX2, off more is betterTileward TWX4, on more is betterTileward TWX2, on more is better
One call18202019
Two calls in one turn20201920
Nested arguments20171914
A tool that must not be called20202020
A two-turn loop20202020
The right tool out of twenty20202019
All six, out of 120118117118112
Findings · Other builds

TWX2 and four other builds on MMLU, each against TWX4 in its own session.

Four other builds of Qwen3.6-35B-A3B have an MMLU score from a session that measured TWX4 too, on the same questions with the same settings. Each line is one such pair, TWX2’s among them: the build’s size, its score, TWX4’s score in that session, and the difference with its 95% range. Across the four sessions TWX4 itself scored between 82.8% and 83.0%. The pairs ran on different cards and engine versions, so each line is read on its own: the chart sets them side by side, and does not rank them. Read against the same TWX4 build, TWX2 (0.85 points behind it) sits about half a point below NVIDIA’s NVFP4 build (0.36 points below it); no session ran those two side by side, so that half point has no test of its own.

Qwen’s FP8 build37.46 GB · 83.0% against 82.9%
Qwen’s FP8 build: +0.06
NVIDIA’s NVFP4 build23.42 GB · 82.6% against 82.9%
NVIDIA’s NVFP4 build: −0.36
Tileward TWX320.08 GB · 83.2% against 83.0%
Tileward TWX3: +0.24
Tileward TWX216.05 GB · 81.9% against 82.8%
Tileward TWX2: −0.85
ISTA-DASLab’s GSQ build12.57 GB · 81.4% against 82.8%
ISTA-DASLab’s GSQ build: −1.40
Memory · Size

TWX2 is 16.05 GB, 31% smaller than NVIDIA’s 23.42 GB NVFP4 build.

Eight builds of the same model: the original; three public builds, Qwen’s own FP8 build, NVIDIA’s NVFP4 build and ISTA-DASLab’s GSQ build; and four of ours, TWX4, TWFP4, TWX3 and TWX2. A shorter bar is a smaller build, and each label carries the build’s MMLU score beside TWX4’s from the same session. TWX2 is 31% smaller than NVIDIA’s build and 0.85 points behind TWX4, where NVIDIA’s build was 0.36 points below it, not separable; TWX3, at 20.08 GB, is 14% smaller than NVIDIA’s build and 0.24 points above TWX4, also not separable. The GSQ build is smaller than TWX2 and 1.40 points behind TWX4.

Qwen3.6-35B-A3B as publishedThe original
71.90 GB
Qwen’s FP8 build48% smaller (1.9×) · MMLU 83.0%, TWX4 82.9%
37.46 GB
Tileward TWX466% smaller (3.0×) · MMLU 82.8% to 83.0%
24.10 GB
NVIDIA’s NVFP4 build67% smaller (3.1×) · MMLU 82.6%, TWX4 82.9%
23.42 GB
Tileward TWFP468% smaller (3.1×)
23.08 GB
Tileward TWX372% smaller (3.6×) · MMLU 83.2%, TWX4 83.0%
20.08 GB
Tileward TWX278% smaller (4.5×) · MMLU 81.9%, TWX4 82.8%
16.05 GB
ISTA-DASLab’s GSQ build83% smaller (5.7×) · MMLU 81.4%, TWX4 82.8%
12.57 GB
Memory · One card

TWX2 takes 13.5 GiB of the card; TWX4 takes 21.0.

Both builds ran on one RTX PRO 6000 Blackwell card of 96 GB. This is the card for each build, as the server loaded it: the model, and the rest of the card for everything else, conversations included. TWX2 leaves 7.5 GiB more of the card free.

Tileward TWX4Leaves 74.6 GiB of the card
The model: 21.0 GiBThe rest of the card: 74.6 GiB
Tileward TWX2Leaves 82.1 GiB of the card
The model: 13.5 GiBThe rest of the card: 82.1 GiB
One RTX PRO 6000 Blackwell card, 95.59 GiB as its driver reports it, and the same server for both builds.
BuildThe model on the card smaller is betterThe rest of the card more is better
Tileward TWX421.0 GiB74.6 GiB
Tileward TWX213.5 GiB82.1 GiB
Which build

Choose TWX2 when the weights must fit in 16 GB.

TWX2 is 31% smaller than NVIDIA’s NVFP4 build and 33% smaller than TWX4. Against TWX4 it gives up 0.85 MMLU points and makes up 1.0 point more of the facts it writes, 3.2% against 2.2%, for 7.5 GiB more of the card. NVIDIA’s build, in its own session, was 0.36 points below TWX4 on MMLU, so read against the same TWX4, TWX2 sits about half a point below it. Where 20 GB fits, TWX3 (20.08 GB) is 14% smaller than NVIDIA’s build and was 0.24 points above TWX4 on MMLU in its own session, not separable; where the card has room for 24.10 GB, TWX4 keeps those points. Both are in the model library; to run either, talk to an engineer.

Our TWFP4 build of the same model, 23.08 GB, is measured against the original on its own report, with function calls and long documents.

How it was measured

Both builds ran one after the other on one card, the same way.

TWX2 and TWX4 ran in one session on one RTX PRO 6000 Blackwell card, with the same engine, settings and questions, TWX2 first. The settings for each test follow.

The card. RTX PRO 6000 Blackwell Server Edition (96 GB), with vLLM 0.31.0 serving each build from its own weights, the same engine files for both.

Size. The weights files, in decimal gigabytes.

MMLU. Five-shot, all 14,042 questions, run by lm-evaluation-harness with identical arguments for both builds and no sampling, each answer scored by the letter the model rates most likely. Compared question by question with an exact McNemar test, and a 95% range for the paired difference.

Made-up facts. Each build reads 360 passages from Tileward’s own document set and writes out the facts they state, taking its likeliest word every time; every fact is matched back to its passage, and one that no passage supports counts as made up. Three different draws of passages, the same for both builds, compared with a two-sided Fisher test, treating facts as independent.

Tool calls. Six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, with thinking off and on, one pass, scored mechanically on tool name, call count and argument shape.

The model on the card. The memory each build’s weights took, as its server reported when it loaded them.

Other builds. Each pair ran the same MMLU questions and arguments for both of its builds, in one session: Qwen’s FP8 and NVIDIA’s NVFP4 builds with TWX4 on one 80 GB Hopper card and vLLM 0.25.1; ISTA-DASLab’s GSQ build with TWX4 on one 80 GB Hopper card and vLLM 0.27.1, with the changes its model card asks for; TWX3 with TWX4 on one RTX PRO 6000 Blackwell card and vLLM 0.31.0. The 95% range is the normal approximation for a paired difference.

The model. Qwen3.6-35B-A3B is Qwen’s, released under the Apache 2.0 license.

Bring your own workload

Bring the questions and tools you would give Qwen3.6-35B-A3B and the card you run on, and we will measure the same things. Talk to an engineer, or compare the 35B-A3B TWX2 eval card.