MMLU
Knowledge: 14,042 multiple-choice questions across 57 subjects, with five worked examples before each. Higher is better.
The full result, with the countsQwen3.6-35B-A3B is Qwen’s model of 35 billion parameters, about 3 billion of which work on each word. As Qwen publishes it, its weights take 71.90 GB, and our TWX4 build in the model library takes 24.10 GB. We made TWX2 at 16.05 GB: 31% smaller than NVIDIA’s NVFP4 build of the same model (23.42 GB), 33% smaller than TWX4, and 78% smaller than the original (4.5×). Against TWX4, in one session on one RTX PRO 6000 Blackwell card, it scored 81.9% on MMLU’s 14,042 questions against 82.8%: 0.85 points behind. It made up 3.2% of the facts it wrote, against 2.2%. On the card the model takes 13.5 GiB, where TWX4 takes 21.0. NVIDIA’s build was measured against TWX4 too, in its own session: 0.36 points below it on MMLU, 82.6% against 82.9%, too small a gap for the test to separate. Below, TWX2’s size and MMLU score sit beside it and three other builds of the same model, each measured against TWX4 in its own session.
See the matching eval card for the result in one screen.
Size is against NVIDIA’s NVFP4 build of Qwen3.6-35B-A3B. Quality is against TWX4, the build in our model library, in one session on one card. NVIDIA’s build was measured against TWX4 in a session of its own, so its line here is that pair. Every section below starts with its own line, then gives the full measurement.
| Measure | Tileward TWX2 |
|---|---|
| Size of the weights, against NVIDIA’s NVFP4 build | Smaller by 31%: 16.05 GB against 23.42 GB |
| Knowledge on MMLU, against TWX4 in the same session | Behind by 0.85 points: 98.9% of TWX4’s score, 81.9% against 82.8% |
| NVIDIA’s NVFP4 build on MMLU, against TWX4 in its own session | 0.36 points below, not separable: 99.6% of TWX4’s score, 82.6% against 82.9% |
| Facts made up, against TWX4 | More, by 1.0 point: 3.2% against 2.2% |
| Tool calls right, thinking off and on, against TWX4 | 7 fewer: 97.0% of TWX4’s, 229 of 240 against 236 of 240 |
| The model’s share of one RTX PRO 6000 Blackwell card, against TWX4 | Less by 7.5 GiB: 13.5 GiB against 21.0 GiB |
Every test on this page, with each build’s result drawn from zero: TWX2 also got 7 fewer of 240 tool calls right. Each card links to the full result below.
Knowledge: 14,042 multiple-choice questions across 57 subjects, with five worked examples before each. Higher is better.
The full result, with the countsEach build writes out the facts stated in 360 passages, over three different draws of passages; a fact no passage supports counts as made up. Lower is better.
The full result, draw by drawSix kinds of call, 20 tries each, with thinking off and then on. More is better.
The full result, kind of call by kind of callWe asked three things of TWX2, each against TWX4 on the same card in the same session. Does it still know as much? MMLU asks 14,042 multiple-choice questions across 57 subjects, and higher is better. Does it make facts up? Each build writes out the facts in 360 passages, three times over, and fewer made up is better. Can it still call tools? Six kinds of call, 20 tries each, and more right is better.
How far behind. On MMLU, 0.85 points is 119 more questions wrong out of 14,042: TWX2 answered 11,504 right and TWX4 11,623, so TWX2 keeps 98.9% of TWX4’s score, and the 95% range for the gap runs from 0.30 to 1.39 points. On made-up facts, 3.2% against 2.2% is 1.0 point more: about one fact in 31 made up, against one in 45 for TWX4. On tool calls, TWX2 got 229 of 240 right against 236, 7 fewer: one with thinking off and six with it on, most of them on nested arguments (14 of 20 against 19).
MMLU, compared question by question. Before the run we wrote down that a difference would be called behind or ahead only if the paired test separated the builds, and not separable otherwise. Tap the reading for the counts.
| Test | Tileward TWX4 | Tileward TWX2 | Against TWX4 |
|---|---|---|---|
| MMLU, 14,042 questions, higher is better | 82.8% (11,623 right) | 81.9% (11,504 right) | 823 questions right only for TWX4 and 704 only for TWX2 (exact McNemar p = 0.0025). The 95% range for the difference runs from 0.30 to 1.39 points behind. |
Made-up facts, over three draws of 360 passages each. Both builds read the same passages in each draw, and the first draw is the one every earlier reading of this model used. TWX2 made up more on all three, by 0.8, 1.8 and 0.4 points; alone, only the second draw separates the builds, and the three pooled do, 1.0 point apart.
| Draw | Tileward TWX4 lower is better | Tileward TWX2 lower is better | Two-sided Fisher test |
|---|---|---|---|
| First | 2.3% (39 of 1,672) | 3.1% (49 of 1,585) | p = 0.20 |
| Second | 2.2% (36 of 1,611) | 4.0% (62 of 1,551) | p = 0.0053 |
| Third | 2.1% (37 of 1,741) | 2.5% (40 of 1,616) | p = 0.56 |
| All three | 2.2% (112 of 5,024) | 3.2% (151 of 4,752) | p = 0.0040 |
Tool calls, 20 tries of each kind, with thinking off and on. These are counts from one pass, with no test applied. Over both settings TWX2 made 229 of the 240 calls right, against 236 for TWX4, and most of its misses are on nested arguments.
| Kind of call | Tileward TWX4, off more is better | Tileward TWX2, off more is better | Tileward TWX4, on more is better | Tileward TWX2, on more is better |
|---|---|---|---|---|
| One call | 18 | 20 | 20 | 19 |
| Two calls in one turn | 20 | 20 | 19 | 20 |
| Nested arguments | 20 | 17 | 19 | 14 |
| A tool that must not be called | 20 | 20 | 20 | 20 |
| A two-turn loop | 20 | 20 | 20 | 20 |
| The right tool out of twenty | 20 | 20 | 20 | 19 |
| All six, out of 120 | 118 | 117 | 118 | 112 |
Four other builds of Qwen3.6-35B-A3B have an MMLU score from a session that measured TWX4 too, on the same questions with the same settings. Each line is one such pair, TWX2’s among them: the build’s size, its score, TWX4’s score in that session, and the difference with its 95% range. Across the four sessions TWX4 itself scored between 82.8% and 83.0%. The pairs ran on different cards and engine versions, so each line is read on its own: the chart sets them side by side, and does not rank them. Read against the same TWX4 build, TWX2 (0.85 points behind it) sits about half a point below NVIDIA’s NVFP4 build (0.36 points below it); no session ran those two side by side, so that half point has no test of its own.
| Build | Weights smaller is better | MMLU higher is better | TWX4, same session | Points from TWX4 95% range | Reading |
|---|---|---|---|---|---|
| Qwen’s FP8 build | 37.46 GB | 83.0% | 82.9% | +0.06 (−0.28 to +0.41) | 312 questions right only for the FP8 build and 303 only for TWX4 (p = 0.75). One 80 GB Hopper card, vLLM 0.25.1. |
| NVIDIA’s NVFP4 build | 23.42 GB | 82.6% | 82.9% | −0.36 (−0.75 to +0.03) | 366 questions right only for the NVFP4 build and 416 only for TWX4 (p = 0.080). The same session as the FP8 build. |
| Tileward TWX3 | 20.08 GB | 83.2% | 83.0% | +0.24 (−0.14 to +0.63) | 401 questions right only for TWX3 and 367 only for TWX4 (p = 0.23). One RTX PRO 6000 Blackwell card, vLLM 0.31.0. |
| Tileward TWX2 | 16.05 GB | 81.9% | 82.8% | −0.85 (−1.39 to −0.30) | 704 questions right only for TWX2 and 823 only for TWX4 (p = 0.0025). This page’s session: one RTX PRO 6000 Blackwell card, vLLM 0.31.0. |
| ISTA-DASLab’s GSQ build | 12.57 GB | 81.4% | 82.8% | −1.40 (−1.96 to −0.83) | 721 questions right only for the GSQ build and 917 only for TWX4 (p < 0.0001). One 80 GB Hopper card, vLLM 0.27.1. |
Eight builds of the same model: the original; three public builds, Qwen’s own FP8 build, NVIDIA’s NVFP4 build and ISTA-DASLab’s GSQ build; and four of ours, TWX4, TWFP4, TWX3 and TWX2. A shorter bar is a smaller build, and each label carries the build’s MMLU score beside TWX4’s from the same session. TWX2 is 31% smaller than NVIDIA’s build and 0.85 points behind TWX4, where NVIDIA’s build was 0.36 points below it, not separable; TWX3, at 20.08 GB, is 14% smaller than NVIDIA’s build and 0.24 points above TWX4, also not separable. The GSQ build is smaller than TWX2 and 1.40 points behind TWX4.
| Build | Weights smaller is better | Against the original more is better | MMLU, against TWX4 in the same session higher is better |
|---|---|---|---|
| Qwen3.6-35B-A3B as published | 71.90 GB | The original | – |
| Qwen’s FP8 build | 37.46 GB | 48% smaller (1.9×) | 83.0% against 82.9%: 0.06 points above, not separable |
| Tileward TWX4 | 24.10 GB | 66% smaller (3.0×) | The reference: 82.8% to 83.0% across four sessions |
| NVIDIA’s NVFP4 build | 23.42 GB | 67% smaller (3.1×) | 82.6% against 82.9%: 0.36 points below, not separable |
| Tileward TWFP4 | 23.08 GB | 68% smaller (3.1×) | – |
| Tileward TWX3 | 20.08 GB | 72% smaller (3.6×) | 83.2% against 83.0%: 0.24 points above, not separable |
| Tileward TWX2 | 16.05 GB | 78% smaller (4.5×) | 81.9% against 82.8%: 0.85 points behind |
| ISTA-DASLab’s GSQ build | 12.57 GB | 83% smaller (5.7×) | 81.4% against 82.8%: 1.40 points behind |
Both builds ran on one RTX PRO 6000 Blackwell card of 96 GB. This is the card for each build, as the server loaded it: the model, and the rest of the card for everything else, conversations included. TWX2 leaves 7.5 GiB more of the card free.
| Build | The model on the card smaller is better | The rest of the card more is better |
|---|---|---|
| Tileward TWX4 | 21.0 GiB | 74.6 GiB |
| Tileward TWX2 | 13.5 GiB | 82.1 GiB |
TWX2 is 31% smaller than NVIDIA’s NVFP4 build and 33% smaller than TWX4. Against TWX4 it gives up 0.85 MMLU points and makes up 1.0 point more of the facts it writes, 3.2% against 2.2%, for 7.5 GiB more of the card. NVIDIA’s build, in its own session, was 0.36 points below TWX4 on MMLU, so read against the same TWX4, TWX2 sits about half a point below it. Where 20 GB fits, TWX3 (20.08 GB) is 14% smaller than NVIDIA’s build and was 0.24 points above TWX4 on MMLU in its own session, not separable; where the card has room for 24.10 GB, TWX4 keeps those points. Both are in the model library; to run either, talk to an engineer.
Our TWFP4 build of the same model, 23.08 GB, is measured against the original on its own report, with function calls and long documents.
TWX2 and TWX4 ran in one session on one RTX PRO 6000 Blackwell card, with the same engine, settings and questions, TWX2 first. The settings for each test follow.
The card. RTX PRO 6000 Blackwell Server Edition (96 GB), with vLLM 0.31.0 serving each build from its own weights, the same engine files for both.
Size. The weights files, in decimal gigabytes.
MMLU. Five-shot, all 14,042 questions, run by lm-evaluation-harness with identical arguments for both builds and no sampling, each answer scored by the letter the model rates most likely. Compared question by question with an exact McNemar test, and a 95% range for the paired difference.
Made-up facts. Each build reads 360 passages from Tileward’s own document set and writes out the facts they state, taking its likeliest word every time; every fact is matched back to its passage, and one that no passage supports counts as made up. Three different draws of passages, the same for both builds, compared with a two-sided Fisher test, treating facts as independent.
Tool calls. Six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, with thinking off and on, one pass, scored mechanically on tool name, call count and argument shape.
The model on the card. The memory each build’s weights took, as its server reported when it loaded them.
Other builds. Each pair ran the same MMLU questions and arguments for both of its builds, in one session: Qwen’s FP8 and NVIDIA’s NVFP4 builds with TWX4 on one 80 GB Hopper card and vLLM 0.25.1; ISTA-DASLab’s GSQ build with TWX4 on one 80 GB Hopper card and vLLM 0.27.1, with the changes its model card asks for; TWX3 with TWX4 on one RTX PRO 6000 Blackwell card and vLLM 0.31.0. The 95% range is the normal approximation for a paired difference.
The model. Qwen3.6-35B-A3B is Qwen’s, released under the Apache 2.0 license.
Bring the questions and tools you would give Qwen3.6-35B-A3B and the card you run on, and we will measure the same things. Talk to an engineer, or compare the 35B-A3B TWX2 eval card.