Tileward / Evals
Evals

See what Tileward changes before you deploy it.

Every test we have run, with its result in one line. The tables come first. The cards under them show the detail, and each one links to its write-up.

Tileward Models Each model, its size, and what it was compared with.

ModelBuildFile sizeCompared with
Tileward DeepSeek-V4-ProLanguageTWX3892.7 → 675.2 GB24% smaller (1.3×)
  • The original, already mostly 4-bit Within half a point on MMLU, 89.6% against 90.0%, level on made-up facts and tool calls, and found all 80 facts hidden in passages up to 1,040,000 tokens.
Tileward GLM-5.3LanguageTWFP41,506.7 → 464.8 GB69% smaller (3.2×)
  • Z.ai’s FP8 release Level on MMLU, 86.2% against 86.1%, and on made-up facts, tool calls and next-word agreement. Ours is 38% smaller (1.6×), and it also runs from two of the cards. As a coding agent, in one run each, it solved 86 of 113 DeepSWE tasks against 74, and was level on Terminal-Bench 2.0, 56 of 77 against 58.
  • The 16-bit original Level on MMLU, 86.2% against 86.3%, and found all 10 passkeys hidden in 130,063 tokens.
Tileward Kimi K3.1 ProLanguageTWX2.51,561.3 → 1,008.3 GB35% smaller (1.5×)
  • The original, already 4-bit Within a point on MMLU, 90.5% against 91.1%, and level on made-up facts and tool calls. As a coding agent, in one run each, within two tasks of it on SWE-bench Verified (381 of 500 against 383), DeepSWE (87 of 113 against 89) and Terminal-Bench 2.0 (62 of 78 against 60). Our TWX3 build (1,178.4 GB) scores the same 90.5%.
Tileward Kimi K3LanguageTWX2.51,560.9 → 1,007.9 GB35% smaller (1.5×)
  • The original, already 4-bit Within a point on MMLU, 90.5% against 91.1%, and level on made-up facts and tool calls. Our TWX3 build (1,178.1 GB) is level on MMLU, 90.9%.
Tileward GLM-5.3-FlashLanguageTWFP4642.7 → 204.4 GB68% smaller (3.1×)
  • DeepSWE Solved 64.6% of 113 long coding tasks as a coding agent, in line with the 63.4% Z.ai reports for the original.
  • The original Picks its next word 88.8% of the time.
Tileward Qwen3-VL-32BVision and textTWFP466.7 → 20.0 GB70% smaller (3.3×)
  • Red Hat’s NVFP4 build Level on MMLU, 82.9% against 82.5%. Ours is 8.7% smaller. Red Hat’s stays closer to the original on next-word agreement.
Tileward Qwen3.8-27BLanguageTWFP455.6 → 20.1 GB64% smaller (2.8×)
  • NVIDIA’s NVFP4 build Level on MMLU, 82.4% against 82.1%. Ours is 8.5% smaller. NVIDIA’s stays closer to the original on next-word agreement.
  • The original, in an agent loop Our earlier TWX4 build (31.0 GB) finished 43.7% of 142 tasks against 45.1%, a gap inside the test’s margin.
Tileward 35B-A3BLanguageTWFP471.9 → 23.1 GB68% smaller (3.1×)
  • The original Level on BFCL’s function calls, 75.9% against 76.3%, and on AA-LCR’s long documents, 59.6% against 60.8%. On one card the model takes 19.1 GiB against 64.6.
Tileward 35B-A3BLanguageTWX271.9 → 16.0 GB78% smaller (4.5×)
  • NVIDIA’s NVFP4 build, 23.4 GB TWX2 is 31% smaller. On MMLU each was measured against TWX4 in its own session: NVIDIA’s build 0.36 points below it, not separable, and TWX2 0.85 points behind.
  • Our TWX4 build, 24.1 GB 0.85 points behind on MMLU, 81.9% against 82.8%, and 1.0 point more facts made up, 3.2% against 2.2%. TWX2 is 33% smaller, and takes 13.5 GiB of one card against 21.0.
Tileward 35B-A3BLanguageTWX471.9 → 24.1 GB66% smaller (3.0×)
  • The publisher’s FP8 build Level on MMLU, 82.8% against 82.9%. Ours is 36% smaller (1.55×).
  • NVIDIA’s NVFP4 build Level on MMLU, 82.9% against 82.6%, and closer to the original. NVIDIA’s is 2.8% smaller (1.03×) and caches 12% more tokens at our hosted setting.
Tileward PersonaPlex 7BVoiceTWX416.7 → 6.3 GB62% smaller (2.7×)
  • The original Chooses its next word 96% of the time. On one laptop it makes each 80 ms frame in 20 ms against about 40.
Tileward VoiceChat 11BVoiceTWX444.4 → 11.5 GB74% smaller (3.9×)
  • The original Picks the right tool as often as the original, 55 of 80 against 52. On the card it runs at the original’s speed.
Whisper large-v3Speech to textTWX43.09 → 0.98 GB68% smaller (3.2×)
  • The original At most 0.3 points more error than the original on nine of ten test sets, and 1.06 on Danish. 152 live calls on one 20 GB card, against 140.
Whisper large-v3-turboSpeech to textTWX41.62 → 0.56 GB65% smaller (2.9×)
  • Whisper large-v3-turbo Level with turbo on six of ten test sets and behind on four, by 0.09 to 0.68 points. 331 live calls on one 20 GB card, against 301.

Tileward Context How much less was sent, and what happened to the answers.

WorkloadSent to the modelWhat held
Long conversationsLOCOMO95% less context (18.9×)55% of questions answered, against 63–67% with the whole transcript.
One long conversation, asked back97–99% fewer tokens (38–82×)20 of 20 questions answered, the same as resending the whole transcript.
An agent with a long backlogAgentBench, 104 tasks68% less context (3.2×)37 tasks solved against 24 with full history. The lead moves by 14–19 tasks between identical runs.
Agents in sequence7 agents49% fewer tokens (2.0×)86% of the material each step needed was delivered.
Agents in parallel8 agents78% fewer tokens (4.5×)66% of the material each agent needed was delivered.
A recall upgradeLongMemEval-S, our own run on a 150-question sample, self-judged47.9% → 57.9% right14 answers fixed and none broken, on the 140 questions scored.

Tileward Governance A locked tile, tested by trying to get past it.

TestResultWhat held
Requests written to get past a lock2,201 of 2,251 stoppedStopped before the model saw them. 50 got through.
Ordinary requests45 of 50 allowed5 were blocked that should not have been.
Workload
Benchmark
Measurement

Tileward Models Smaller builds, measured against the original.

Model compressionMMLUQuality

24% smaller, and within half a point of the original on MMLU Tileward DeepSeek-V4-Pro against DeepSeek-V4-Pro as DeepSeek publishes it, already mostly 4-bit

DeepSeek-V4-Pro as published892.7 GB · 90.0% MMLU
Tileward DeepSeek-V4-ProTWX3675.2 GB · 89.6% MMLU
24%
Smaller than DeepSeek-V4-Pro as published (1.3×)
89.6%
MMLU score, the original’s 90.0%: inside the one-point bar
1.9%
Facts it made up, the original’s 1.8%
1,200 of 1,200
Tool calls right, the original’s 1,200
80 of 80
Hidden facts found, out to 1,040,000 tokens, the original’s 80
6,942,711
Tokens cached, the original’s 5,707,924

Nearly a quarter smaller, within half a point on MMLU, and level on made-up facts, tool calls and million-token passages. It holds 1.2× the original’s cache on the same eight cards, and picks the original’s next word 87.6% of the time, 92.8% past the first 4,096 tokens of a long passage.

  • The original is the release DeepSeek publishes, with most of its weights already stored in 4 bits, so the 24% is against a mostly 4-bit model, not a 16-bit one.
  • MMLU: five-shot, 14,042 questions, no sampling, scored by the answer letter the model rates most likely. The bar, set before the run: no more than one point below the original on the paired test. 247 answers right only for the original and 192 only for TWX3 (p = 0.0099; at most 0.64 points below).
  • Fabrication: each build reads 360 passages from Tileward’s own document set and writes out the facts they state; every fact is matched back to its passage, and one that no passage supports counts as made up. The original wrote 1,782 facts and 32 were not supported; TWX3 wrote 1,825 and 34 were not (p = 0.90, treating facts as independent).
  • Tool calls: six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, run five times and scored mechanically on tool name, call count and argument shape: 600 calls at each thinking effort. Both builds made 600 right at low effort and 600 at high.
  • Hidden facts: ten short facts in each of two passages at 32,768, 131,072, 524,288 and 1,040,000 tokens, asked for in one answer; both builds found all 80.
  • Agreement: teacher-forced on 300 corpus chunks (82,492 positions), top-1 agreement with the original 87.6%, mean KL 0.079; past the first 4,096 tokens of 28 long passages (1,196,032 positions), 92.8%, KL 0.044.
  • Bars are weights on disk; loaded, 83.5 GiB on each card for TWX3 and 108.9 for the original. With 90% of each card the cache holds 6,942,711 and 5,707,924 tokens. Every figure on this card is from one session on eight B300 cards and vLLM. DeepSeek-V4-Pro is DeepSeek’s, under the MIT License.
Report: We shrank DeepSeek-V4-Pro by 24%, within half a point of the original on MMLU →
Agent workflowsModel compressionMMLUQuality

38% smaller than Z.ai’s FP8 release, at the same quality Tileward GLM-5.3 against Z.ai’s FP8 release and the 16-bit original

GLM-5.3 in 16 bits1,506.7 GB · 86.3% MMLU
Z.ai’s FP8 release755.6 GB · 86.1% MMLU
Tileward GLM-5.3TWFP4464.8 GB · 86.2% MMLU
38%
Smaller than Z.ai’s FP8 release (1.6×)
86.2%
MMLU score, the FP8 release’s 86.1%: level
1.4%
Facts it made up, the FP8 release’s 2.2%
1,200 of 1,200
Tool calls right, as for both releases
10 of 10
Passkeys found in 130,063 tokens
4,047,872
Tokens cached, the FP8 release’s 3,267,776
86 of 113
DeepSWE tasks solved as a coding agent, the FP8 release’s 74, in one run each
56 of 77
Terminal-Bench 2.0 tasks solved, the FP8 release’s 58: level

Smaller than the FP8 release, and level with it and with the 16-bit original on every quality test here. It holds 24% more cache than the FP8 release on the same eight cards, and both pick the 16-bit original’s next word 91.9% of the time. As a coding agent it is level with the FP8 release on Terminal-Bench 2.0, and solved more DeepSWE tasks in one run.

  • MMLU: five-shot, 14,042 questions, no sampling, scored by the answer letter the model rates most likely. The bar, set before the run: no more than one point below the 16-bit original on the paired test. 296 answers right only for the original and 283 only for TWFP4 (p = 0.62; at most 0.37 points below); 339 and 315 for the FP8 release (p = 0.37; at most 0.47 points below). TWFP4 against the FP8 release: 340 and 329 (p = 0.70).
  • Fabrication: each build reads 360 passages from Tileward’s own document set and writes out the facts they state; every fact is matched back to its passage, and one that no passage supports counts as made up. The 16-bit original wrote 1,361 facts and 23 were unsupported; the FP8 release wrote 1,399 and 31 were (p = 0.34), TWFP4 1,381 and 20 (p = 0.65), treating facts as independent.
  • Tool calls: six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, run five times and scored mechanically on tool name, call count and argument shape: 600 calls at each thinking effort. All three made all 600 at both efforts.
  • Agreement: teacher-forced on 300 corpus chunks (83,139 positions), top-1 agreement with the 16-bit original; the mean KL to it is 0.030 for the FP8 release and 0.031 for TWFP4.
  • Long passages: 24 passages of 32,768 tokens, agreement with the 16-bit original in the first 2,048 tokens and after them: 90.5% then 90.1% for the FP8 release, 90.6% then 90.5% for TWFP4. Passkeys: ten, each hidden at a different depth of a 130,063-token prompt.
  • Coding agent: TWFP4 and Z.ai’s FP8 release, one run each on the same tasks, compared task by task. DeepSWE: all 113 tasks, worked with mini-SWE-agent, with the task’s network open only to the model; 21 solved only by TWFP4 and 9 only by the FP8 release (p = 0.043; 1.3 to 19.8 points above). One run of each detects only large gaps, so we read it as TWFP4 holding up over long work, not as a better coder. Terminal-Bench 2.0: 77 of its 89 tasks, leaving out, before either release ran, the 12 whose own reference solution failed on this machine; 6 and 8 (p = 0.79).
  • Bars are the weights files; loaded, 55.3 GiB on each card for TWFP4, 89.4 for the FP8 release and 176.5 for the 16-bit original. With 90% of each card and an 8-bit cache, the cache holds 4,047,872, 3,267,776 and 1,307,648 tokens. The quality and memory figures on this card are from one session on eight B300 cards and stock vLLM 0.32.0; the coding runs used stock vLLM on eight B300 cards, with 95% of each card. GLM-5.3 is Z.ai’s, under the GLM-5.3 License.
  • Fewer cards: served from four of the cards, 106.8 GiB on each and a cache of 2,891,456 tokens; from two, with the server allowed 95% of each card, 209.6 GiB on each and 862,272 tokens. Both loaded and answered; the quality figures are from the eight-card run.
Report: We made GLM-5.3 38% smaller than Z.ai’s FP8 release, at the same quality →
Agent workflowsModel compressionMMLUQuality

35% smaller, and within a point of the original on MMLU Tileward Kimi K3.1 Pro against Kimi K3.1 Pro as Moonshot publishes it, already 4-bit

Kimi K3.1 Pro as published1,561.3 GB · 91.1% MMLU
Tileward Kimi K3.1 ProTWX31,178.4 GB · 90.5% MMLU
Tileward Kimi K3.1 ProTWX2.51,008.3 GB · 90.5% MMLU
35%
Smaller than Kimi K3.1 Pro as published (1.5×)
90.5%
MMLU score, the original’s 91.1%: inside the one-point bar
0.1%
Facts it made up, the original’s 0.0%
1,198 of 1,200
Tool calls right, the original’s 1,200
64 GiB
Less on each of the eight cards
6,086,715
Tokens cached, the original’s 2,135,421
381 of 500
SWE-bench Verified issues solved as a coding agent, the original’s 383
87 of 113
DeepSWE tasks solved, the original’s 89
62 of 78
Terminal-Bench 2.0 tasks solved, the original’s 60

A third smaller, within a point on MMLU, and level on made-up facts and tool calls. It holds 2.9× the original’s cache on the same eight cards. TWX3 scores the same on MMLU and keeps closer to the original on next-word agreement, 91.2% against 88.4% for TWX2.5. As a coding agent, TWX2.5 came within two tasks of the original on SWE-bench Verified, DeepSWE and Terminal-Bench 2.0.

  • The original is the one release Moonshot publishes, already stored in 4 bits, so the 35% is against a 4-bit model, not a 16-bit one.
  • MMLU: five-shot, 14,042 questions, no sampling, scored by the answer letter the model rates most likely. The bar, set before the run: no more than one point below the original on the paired test. 301 answers right only for the original and 218 only for TWX2.5 (p = 0.0003; at most 0.86 points below); 267 and 187 for TWX3 (p = 0.0002; at most 0.82 points below). Between the two builds, 290 and 287 (p = 0.93).
  • Fabrication: each build reads 360 passages from Tileward’s own document set and writes out the facts they state; every fact is matched back to its passage, and one that no passage supports counts as made up. The original wrote 1,666 facts and every one was supported; TWX2.5 wrote 1,630 and 2 were not (p = 0.24), TWX3 1,565 and 2 (p = 0.23), treating facts as independent.
  • Tool calls: six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, run five times and scored mechanically on tool name, call count and argument shape: 600 calls at each thinking effort. Low effort: 600, 600 and 599 right for the original, TWX2.5 and TWX3; high effort: 600, 598 and 598.
  • Agreement: teacher-forced on 300 corpus chunks (82,712 positions), top-1 agreement with the original; the mean KL to it is 0.062 for TWX2.5 and 0.038 for TWX3.
  • Coding agent: the original and TWX2.5, one run each on the same tasks, compared task by task. SWE-bench Verified: all 500 issues, worked with mini-SWE-agent, no network inside the task containers, graded by the official harness; 13 issues solved only by TWX2.5 and 15 only by the original (p = 0.85). DeepSWE: all 113 tasks, with the task’s network open only to the model; 9 and 11 (p = 0.82). Terminal-Bench 2.0: 78 of its 89 tasks, leaving out, before any build ran, the 11 whose own reference solution failed on this machine; 5 and 3 (p = 0.73). One run of each detects only large gaps.
  • Bars are weights on disk; loaded, 128.1 GiB on each card for TWX2.5, 147.9 for TWX3 and 192.5 for the original. With 90% of each card the cache holds 6,086,715, 4,915,678 and 2,135,421 tokens. The quality and memory figures on this card are from one session on eight B300 cards and vLLM’s own Kimi K3.1 image; the coding runs used the same image on eight B300 cards, with 95% of each card. Kimi K3.1 Pro is Moonshot AI’s, under the Kimi K3.1 Pro License.
Report: We shrank Kimi K3.1 Pro by 35%, within a point of the original on MMLU →
Model compressionMMLUQuality

35% smaller, and within a point of the original on MMLU Tileward Kimi K3 against Kimi K3 as Moonshot publishes it, already 4-bit

Kimi K3 as published1,560.9 GB · 91.1% MMLU
Tileward Kimi K3TWX31,178.1 GB · 90.9% MMLU
Tileward Kimi K3TWX2.51,007.9 GB · 90.5% MMLU
35%
Smaller than Kimi K3 as published (1.5×)
90.5%
MMLU score, the original’s 91.1%: inside the one-point bar
0.9%
Facts it made up, the original’s 1.0%
596 of 600
Tool calls right, the original’s 598
64 GiB
Less on each of the eight cards
3,945,603
Tokens cached, the original’s 1,467,558

A third smaller, within a point on MMLU, and level on made-up facts and tool calls. It holds 2.7× the original’s cache on the same eight cards. TWX3 keeps closer to the original: level on MMLU, and 91.4% next-word agreement against 88.6% for TWX2.5.

  • The original is the one release Moonshot publishes, already stored in 4 bits, so the 35% is against a 4-bit model, not a 16-bit one.
  • MMLU: five-shot, 14,042 questions, no sampling, scored by the answer letter the model rates most likely. The bar, set before the run: no more than one point below the original on the paired test. 290 answers right only for the original and 207 only for TWX2.5 (p = 0.0002; at most 0.85 points below); 242 and 209 for TWX3 (p = 0.13).
  • Fabrication: each build reads passages from Tileward’s own document set and writes out the facts they state; every fact is matched back to its passage, and one that no passage supports counts as made up. The original wrote 1,350 facts and 13 were unsupported; TWX2.5 wrote 1,384 and 12 were (p = 0.84), TWX3 1,285 and 12 (p = 1.0), treating facts as independent.
  • Tool calls: six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, run five times and scored mechanically on tool name, call count and argument shape: 600 calls at each thinking effort. Low effort: 598, 596 and 597 right for the original, TWX2.5 and TWX3; high effort: 597, 595 and 590.
  • Agreement: teacher-forced on 300 corpus chunks (82,712 positions), top-1 agreement with the original; the mean KL to it is 0.060 for TWX2.5 and 0.037 for TWX3.
  • Bars are weights on disk; loaded, 127.3 GiB on each card for TWX2.5, 147.2 for TWX3 and 191.7 for the original. With 90% of each card the cache holds 3,945,603, 3,195,766 and 1,467,558 tokens. Every figure on this card is from one session on eight B300 cards and stock vLLM 0.31.0.
Report: We shrank Kimi K3 by 35%, within a point of the original on MMLU →
Agent workflowsModel compressionQuality

68% smaller, and in line with the original on DeepSWE Tileward GLM-5.3-Flash as a coding agent, and against the 16-bit original

GLM-5.3-Flash in 16 bits642.7 GB
Tileward GLM-5.3-FlashTWFP4204.4 GB · 64.6% DeepSWE
68%
Smaller than the 16-bit original (3.14×)
64.6%
DeepSWE tasks solved; Z.ai reports 63.4% for the original
88.8%
Next-word agreement with the original
2
Cards to run it, every weight on the cards

A third of the original’s size, in line with it on DeepSWE. TWFP4 solved 64.6% of 113 long coding tasks against the 63.4% Z.ai reports for the original, and 88.8% of the time it picks the original’s next word.

  • DeepSWE v1.1: all 113 tasks, one attempt each, mini-SWE-agent 2.4.6 under Datacurve’s own harness, the model’s full 1,048,576-token window, three hours a task, temperature 1.0, top-p 0.95, on two B200 cards: 73 solved (64.6%; 95% interval 55.4% to 72.8%). Z.ai reports 63.4% for the original from its own run (mini-SWE-agent, a 400K window, six hours a task).
  • Agreement: teacher-forced on 32 passages (16,352 positions), top-1 agreement with the BF16 original; the mean KL to it is 0.088. Our earlier TWX4 build (205.1 GB) agrees 87.3% of the time, level with TWFP4 on the same run.
  • Bars are weights on disk: 204.45 GB against 642.65 GB. Every figure on this card is from stock vLLM 0.32.0 on two cards: two B300 cards, and two B200 cards for DeepSWE.
Report: We shrank GLM-5.3-Flash by 68%, in line with the original on DeepSWE →
Model compressionMMLUQuality

Level with Red Hat’s NVFP4 build, and 8.7% smaller Tileward Qwen3-VL-32B against Red Hat’s NVFP4 build of the same base

Qwen3-VL-32B as published66.7 GB · 83.7% MMLU
Red Hat NVFP421.9 GB · 82.5% MMLU
Tileward Qwen3-VL-32BTWFP420.0 GB · 82.9% MMLU
8.7%
Smaller than Red Hat’s build (1.09×)
82.9%
MMLU score, Red Hat’s 82.5%: level
70.5%
MMStar score, Red Hat’s 70.7%: level
5.8%
Facts it made up, Red Hat’s 5.6%
600 of 600
Tool calls right, Red Hat’s 600
514,992
Tokens cached at serving settings, Red Hat’s 500,688

Smaller, and level on MMLU, MMStar, made-up facts and tool calls. It holds 2.9% more tokens in its cache. Red Hat’s build stays closer to the original on next-word agreement, 89.7% against 87.7%.

  • MMLU: five-shot, 14,042 questions, lm-evaluation-harness, no sampling. The bars are one session on one card, same base model; the original scored 83.7% there. The paired test counts the answers that differ: 371 right only for ours, 319 only for Red Hat’s (p = 0.052).
  • MMStar: 1,500 image questions, greedy, replies capped at 1,024 tokens: 60 right only for ours and 64 only for Red Hat’s (p = 0.79).
  • Fabrication: each build reads passages from Tileward’s own document set and writes out the facts they state; every fact is matched back to its passage, and one that no passage supports counts as made up. Ours wrote 1,680 facts and 98 were unsupported; Red Hat’s wrote 1,799 and 101 were (p = 0.83, treating facts as independent).
  • Tool calls: six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, run five times and scored mechanically on tool name, call count and argument shape: 600 calls. Both builds made all 600.
  • Agreement: teacher-forced on 300 corpus chunks (86,589 positions), top-1 agreement with the BF16 original, in a separate session where every build ran under the same settings; the mean KL to it is 0.056 for ours and 0.046 for Red Hat’s.
  • Bars are weights on disk (19.96 against 21.86 GB); loaded on the card, 19.1 against 20.8 GiB. At a 262,144-token window, an FP8 cache and 90% of the card, the cache holds 514,992 tokens for ours and 500,688 for Red Hat’s. Every figure on this card is from one RTX PRO 6000 Blackwell card and stock vLLM 0.31.0.
Report: We made Qwen3-VL-32B 8.7% smaller than NVFP4, at the same quality →
Model compressionMMLUQuality

Level with NVIDIA’s build, and 8.5% smaller Tileward Qwen3.8-27B against NVIDIA’s NVFP4 build

NVIDIA NVFP421.9 GB · 82.1% MMLU
Tileward Qwen3.8-27BTWFP420.1 GB · 82.4% MMLU
8.5%
Smaller than NVIDIA’s build
82.4%
MMLU score, NVIDIA’s 82.1%: level
3.8%
Facts it made up, NVIDIA’s 4.7%
240 of 240
Tool calls right, NVIDIA’s 240

Smaller, and level on MMLU, MMLU-Redux, fabrication and tool calls. MMLU-Redux 89.4% against 89.4% (p = 0.85), fabrication 3.8% against 4.7% of facts (p = 0.23), and all 240 tool calls right for both. On next-word agreement with the original model NVIDIA’s build stays closer, 88.9% against 87.8%.

  • MMLU: five-shot, 14,042 questions, lm-evaluation-harness, no sampling. The two bars are one session on one machine, same base model. The paired test counts the answers that differ: 487 right only for ours, 438 only for NVFP4 (p = 0.11).
  • MMLU-Redux: 5,700 questions, answers generated with thinking on, both builds in one session of their own: 133 right only for ours and 129 only for NVFP4.
  • Fabrication: each build reads passages from Tileward’s own document set and writes out the facts they state; every fact is matched back to its passage, and one that no passage supports counts as fabricated. Ours wrote 1,638 facts and 63 were unsupported; NVFP4 wrote 1,722 and 81 were (p = 0.23, treating facts as independent).
  • Tool calls: six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, scored mechanically on tool name, call count and argument shape, once with thinking off and once on: 240 calls. Both builds made all 240.
  • Agreement: teacher-forced on 300 corpus chunks (87,955 positions), top-1 agreement with the BF16 original, both builds in one session of their own; the mean KL to it is 0.050 for ours and 0.043 for NVFP4.
  • Bars are weights on disk (20.05 against 21.92 GB); loaded on the card, 18.9 against 20.6 GiB. NVIDIA’s build is its public NVFP4 checkpoint of the same base.
Report: We made Qwen3.8-27B 8.5% smaller than NVIDIA’s NVFP4 build, and beyond →
Model compressionQuality

Level with the original on tool calls and long documents, and 68% smaller Tileward 35B-A3B TWFP4 against Qwen3.6-35B-A3B as Qwen publishes it

Qwen3.6-35B-A3B as published71.9 GB · 76.3% BFCL
Tileward 35B-A3BTWFP423.1 GB · 75.9% BFCL
68%
Smaller than the original (3.1×)
75.9%
BFCL function calls, the original’s 76.3%: level
59.6%
AA-LCR long documents, the original’s 60.8%: level
96.5%
Of the original’s next words, at 45,430 places
19.1 GiB
Of one card for the model, the original’s 64.6 GiB

Smaller, and level with the original on function calls and long documents. BFCL is 75.9% against 76.3% (99.5% of the original’s score, 95% range 96.5% to 102.5%) and AA-LCR 59.6% against 60.8% (98.0%, 90.3% to 105.8%). It is also 1.5% smaller than NVIDIA’s NVFP4 build.

  • BFCL: 500 tasks of BFCL v4, three runs for each build, native function calling, scored by BFCL’s own checker: 1,144 of 1,500 for the original and 1,138 for TWFP4. Task by task, 55 went better for TWFP4 and 58 for the original (p = 0.85).
  • AA-LCR: 25 of its questions over long sets of documents, ten runs for each build, every answer graded against its reference by the same judge model for both: 152 of 250 and 149. Question by question, 5 went better for TWFP4 and 6 for the original (p = 1.0).
  • Agreement: 100 replies the original wrote, read back by both builds on one card; TWFP4 picks the original’s likeliest next word at 96.5% of 45,430 places, with a mean KL of 0.019 over the original’s 16 likeliest words.
  • Bars are weights on disk (71.90 against 23.08 GB); loaded on the card, 64.6 against 19.1 GiB. Every figure on this card is from RTX PRO 6000 Blackwell cards and vLLM 0.32.0, with Qwen’s own sampling settings and thinking on; BFCL and AA-LCR ran each build on its own card of the same model.
  • Qwen3.6-35B-A3B is Qwen’s, released under the Apache 2.0 license.
Report: We shrank Qwen3.6-35B-A3B by 68%, level with the original on tool calls and long documents →
Model compressionMMLUQuality

31% smaller than NVIDIA’s NVFP4 build, and 0.85 points behind our TWX4 on MMLU Tileward 35B-A3B TWX2 against NVIDIA’s NVFP4 build on size, and against Tileward 35B-A3B TWX4 on quality

Tileward 35B-A3BTWX424.1 GB · 82.8% MMLU
Tileward 35B-A3BTWX216.0 GB · 81.9% MMLU
31%
Smaller than NVIDIA’s NVFP4 build (23.42 GB); 33% smaller than TWX4, 78% than the original (4.5×)
81.9%
MMLU score, TWX4’s 82.8%: 0.85 points behind
3.2%
Facts it made up, TWX4’s 2.2%
112 of 120
Tool calls right with thinking on, TWX4’s 118; with it off, 117 against 118
13.5 GiB
Of one card for the model, TWX4’s 21.0 GiB

31% smaller than NVIDIA’s NVFP4 build, for 0.85 MMLU points against TWX4 and 1.0 point more facts made up. MMLU is 81.9% against 82.8% (823 questions right only for TWX4 and 704 only for TWX2, p = 0.0025), and made-up facts 3.2% against 2.2% over three draws of passages (p = 0.0040). Beside it on the report are four other builds of the same model, each against TWX4 in its own session: NVIDIA’s NVFP4 build was 0.36 points below TWX4 and our TWX3 build, at 20.08 GB, 0.24 points above it, neither separable from it, and ISTA-DASLab’s GSQ build, at 12.57 GB, was 1.40 points behind. Against the same TWX4, TWX2 sits about half a point below NVIDIA’s build, a gap no session measured directly.

  • MMLU: five-shot, 14,042 questions, lm-evaluation-harness, no sampling, identical arguments for both builds. The reading was set before the run: behind or ahead only where the exact paired test separates the builds (p < 0.05).
  • Fabrication: each build reads 360 passages from Tileward’s own document set and writes out the facts they state, on three draws of passages; a fact no passage supports counts as made up. TWX2 wrote 4,752 facts and 151 were not supported; TWX4 5,024 and 112, treating facts as independent.
  • Tool calls: six cases, 20 trials each, with thinking off and on, one pass, scored mechanically; counts only.
  • Bars are weights on disk (24.10 against 16.05 GB); loaded on the card, 21.0 against 13.5 GiB. Every figure on this card is from one session on one RTX PRO 6000 Blackwell card and vLLM 0.31.0. Qwen3.6-35B-A3B is Qwen’s, released under the Apache 2.0 license.
Report: We shrank Qwen3.6-35B-A3B to 16 GB, 31% smaller than NVIDIA’s NVFP4 build →
Model compressionMMLUQuality

Closer to the original than NVIDIA’s build, and level on MMLU Tileward 35B-A3B against NVIDIA’s NVFP4 build, whose file is 2.8% smaller (1.03×)

NVIDIA NVFP423.4 GB · 82.6% MMLU
Tileward 35B-A3BTWX424.1 GB · 82.9% MMLU
0.023
Distance from the original model, NVIDIA’s 0.040: lower is closer
82.9%
MMLU score, NVIDIA’s 82.6%: level
2.3%
Facts it made up, NVIDIA’s 2.1%
238 of 240
Tool calls right, NVIDIA’s 240
179
Tokens a second for one caller, NVIDIA’s 197; ours leads at 64
508,031
Tokens cached at our hosted setting, NVIDIA’s 571,026

Closer to the original model, and level on MMLU and fabrication. Its next-word probabilities sit nearer the original’s than NVIDIA’s do (a KL distance of 0.023 against 0.040), MMLU is 82.9% against 82.6% (p = 0.08), and fabrication 2.3% against 2.1% of facts (p = 0.82). It leads on MMLU-Redux, 90.7% against 90.0%, mostly on questions NVIDIA’s build did not finish within the 12,288-token limit. NVIDIA’s file is 2.8% smaller (1.03×). NVIDIA’s build decodes faster for 1 and 16 callers while ours is faster for 64, has room for 12% more cached tokens at our hosted memory setting, and missed none of the 240 tool calls where ours missed two.

  • Closeness: teacher-forced on 300 corpus chunks (87,955 positions), one request at a time. The mean KL divergence of each build’s next-word probabilities (its top 20) from the BF16 original’s is 0.023 for ours and 0.040 for NVFP4. Run twice in the same boot on 40 of the chunks, ours moves 0.003 from itself and NVFP4 0.008, so the gap is larger than either build’s own noise.
  • MMLU: five-shot, 14,042 questions, lm-evaluation-harness, no sampling. The two bars are one session on one machine, same base model. The paired test counts the answers that differ: 416 right only for ours, 366 only for NVFP4 (p = 0.08).
  • MMLU-Redux: 5,700 questions, answers generated with thinking on and capped at 12,288 tokens, the same session: 126 right only for ours and 86 only for NVFP4 (p = 0.007). NVFP4 reached the cap on 107 questions and ours on 78; on the 5,569 both finished, 92.0% against 91.8% (p = 0.37).
  • Speed: output tokens a second without speculative decoding, random 1,024-token prompts and 256-token replies, 1, 16 and 64 simultaneous callers: 179, 1,625 and 3,442 for ours against 197, 1,682 and 3,316. With 8,192-token prompts the two are within 1% at 4 and 16 callers.
  • Memory: at our hosted service’s memory setting and its full 262,144-token window, the cache holds 508,031 tokens for ours and 571,026 for NVFP4. At the engine’s default setting the two are within 1.3%.
  • Fabrication: each build reads passages from Tileward’s own document set and writes out the facts they state; every fact is matched back to its passage, and one that no passage supports counts as fabricated. Ours wrote 1,683 facts and 38 were unsupported; NVFP4 wrote 1,819 and 38 were (p = 0.82, treating facts as independent).
  • Tool calls: six cases (one call, two calls in one turn, nested arguments, a tool that must not be called, the right tool out of twenty, a two-turn loop), 20 trials each, scored mechanically on tool name, call count and argument shape, once with thinking off and once on: 240 calls. Ours missed two, both in the one-call case with thinking off, where it answered in text instead of calling the tool; NVFP4 missed none (p = 0.50).
  • Bars are weights on disk (24.10 against 23.42 GB); loaded on the card, 21.0 against 20.4 GiB. NVIDIA’s build is its public NVFP4 checkpoint of the same base. Every figure on this card is from one 80 GB Hopper-generation card.
All write-ups →
Model compressionMMLUQuality

Level with the publisher’s build, and 36% smaller (1.55×) Tileward 35B-A3B on MMLU

The publisher’s own FP8 build37.5 GB · 82.9% MMLU
Tileward 35B-A3BTWX424.1 GB · 82.8% MMLU
Community GPTQ build24.4 GB · 83.3% MMLU
Community AWQ build25.5 GB · 83.2% MMLU
36%
Smaller than the FP8 build (1.55×)
±0.3
Points either way on each score

Level with FP8. Both community 4-bit builds score 0.4 to 0.5 points higher, a direction rather than a size.

  • Five-shot, 14,042 questions, lm-evaluation-harness, no sampling. The four bars are one run, one machine, same base model.
  • Quoted to the tenth: our own build, run twice in one session, scores 82.8 and 82.9, and 181 of 14,042 answers change between the runs.
  • Bars are weights on disk; serving VRAM is 25.6 GB against FP8's 39.1 GB. MMLU is multiple choice and we lead on two of its four categories; matching FP8 is consistent with the builds behaving alike, not proof.
Report: A smaller model should not become a different model →
Model compressionAgent workflowsAgentBenchQuality

In an agent loop, level with the uncompressed model four builds of Qwen3.8-27B on 142 tasks

Qwen3.8-27B, uncompressed64 + 64 of 142 · 45.1%
Tileward Qwen3.8-27B, TWX461 + 63 of 142 · 43.7%
The publisher’s own FP8 build60 + 65 of 142 · 44.0%
Bonsai 2 27B, a ternary build57 + 64 of 142 · 42.6%
43.7%
Tasks finished, uncompressed 45.1%
142
Tasks, each run twice per build

A tie on finished tasks: all three compressed builds finish within noise of the base. The ternary build is the smallest file and works hardest for it. Paired by task over two runs, the TWX4 build of Tileward-Qwen3.8-27b wins 10 and loses 13 against the BF16 base, a gap smaller than the test can see. The publisher’s FP8 build finishes 1.1 points fewer (11 won, 16 lost) and the ternary build 2.5 fewer (13 won, 19 lost); neither gap is one this test can call. Every compressed build spent more prompt tokens than the base to get there: ours 7% more, the ternary build 28% more.

  • AgentBench OS: Linux tasks in a real container, pass or fail decided by the environment; the 142 tasks every arm got a turn on. Greedy decoding, thinking off, 32k context, one NVIDIA 80 GB GPU, the same tasks and settings for all four.
  • An agent loop does not repeat itself exactly, so 142 tasks resolve a gap of about ten points; every gap on this card is smaller than that, a direction rather than a size.
  • The three Qwen-format builds ran on vLLM; the ternary build ran on its publisher’s own server, the only one that loads it, with reasoning off like the others, so its publisher’s thinking-on figures do not compare to this row. On this Hopper-generation card the FP8 build ran its native 8-bit path.
All write-ups →
Model compressionQuality

62% smaller, and half the time for each frame Tileward PersonaPlex 7B on one laptop

PersonaPlex as published16.7 GB · 37 to 42 ms a frame
Standard 8-bit conversion8.2 GB · 23 ms a frame
Tileward TWX46.3 GB · 20 ms a frame
62%
Smaller file
20 ms
To make each 80 ms frame
96%
Same next word as the original

Smaller and faster than both, and behind the 8-bit conversion on agreement. The standard 8-bit conversion picks the original’s next word 98.1% of the time; a standard conversion at our size, 92.8%.

  • One laptop, every build in the same runtime. Compute is the model alone for one conversation; the audio codec is not counted.
  • Agreement is over 375 word moments in four conversations recorded from the published model, so a point either way is within chance.
  • In 20 unscripted conversations a build, a speech recogniser followed the TWX4 build as well as the original.
Report: We shrank PersonaPlex by 62% to run at 20ms a frame →
Model compressionQuality

74% smaller, and it still picks the right tool Tileward VoiceChat 11B against VoiceChat as published

VoiceChat as published44.4 GB · right tool 52 of 80
The same weights, stored in 16 bits24.2 GB · identical to the original
Tileward TWX411.5 GB · right tool 55 of 80
74%
Smaller file than as published
55 of 80
Right tool, the original 52
95%
Same next word as the original

A quarter of the file and no loss this test can see; on the card it keeps the original’s size and speed. The file is 53% smaller than the lossless form. The build is at 71 ms of each 80 ms frame, as the original is, and 24 GiB in memory.

  • 100 spoken requests in six synthetic voices on a clean line, scored on the first tool call. A gap of three or four calls in 80 is within chance, so the two builds are level.
  • One NVIDIA 96 GB card, one conversation at a time. Across 100 answers offline and 20 through the streaming path, an answer repeated itself twice for the original and once for the build.
  • VoiceChat is NVIDIA’s model under the OpenMDW License Agreement, version 1.1 (OpenMDW-1.1); the build is a modified version of it under the same agreement.
Report: We compressed VoiceChat by 74% and made it run in real time on one card →
Model compressionQuality

68% smaller, and at most 0.3 points more error on nine of ten sets Whisper large-v3 and large-v3-turbo, each in TWX4, on one 20 GB card

Whisper large-v3 as published3.09 GB · 140 live calls
Whisper large-v3-turbo as published1.62 GB · 301 live calls
Tileward TWX4 of large-v30.98 GB · 152 live calls
Tileward TWX4 of turbo0.56 GB · 331 live calls
68%
Smaller file than large-v3 (3.2×)
0.3 pts
Most error added on nine of ten sets
1.06 pts
Error added on Danish, the tenth
8%
More live calls than large-v3 on one 20 GB card
4 of 10
Sets where the turbo build trails turbo, by 0.09 to 0.68 pts

Smaller than turbo, and level with it or ahead on every set. The large-v3 build is 40% smaller than large-v3-turbo (1.7×); its error rate is lower than turbo’s on four sets and level on six. Turbo carries about twice the live calls.

  • One 20 GB card, every build on vLLM at 90% of the card’s memory. Word error rate, or character error rate for Mandarin and Japanese, with a paired bootstrap 95% range for each change.
  • Ten sets: LibriSpeech test-clean and test-other, FLEURS in seven languages, and 384 synthetic call-centre clips. Japanese turns on one clip where the original repeats itself.
  • A live call is one second of audio transcribed per second, with 512 windows of 28 seconds sent at once. Large-v3 fills this card’s memory, so a larger card holds more.
Report: We shrank Whisper large-v3 by 68%, below the size of turbo →

Tileward Context Less sent to the model, and what happened to the answers.

Conversation memoryLOCOMOToken costQuality

Most of the answers, on 95% less context (19×) LOCOMO long conversations, judged by a separate model

95%
Less context than the whole transcript (18.9×)
55%
Questions answered with Tileward Context
63–67%
Answered with the whole transcript

Tileward sent 1,499 tokens per question against 28,365 for the full transcript. This keeps most, not all, of the answer rate; a separate model judged every answer.

Report: Keep most of the answer rate on 95% less context (19×) →
Conversation memoryToken costQuality

Every question answered, on 97–99% fewer tokens (38–82×) one 142-turn conversation, asked back

The whole transcript, resent3,122 tokens a question
With Tileward Context38–83 tokens a question
97–99%
Fewer tokens (38–82×)
20 of 20
Questions answered, either way
0 of 20
Answered with no conversation at all

12 facts mentioned once each, 20 questions, 8 needing two or three facts at once. Qwen3.5-9B, exact substring match.

All write-ups →
Agent workflowsAgentBenchToken costQuality

More tasks solved, on 68% less context (3.2×) 104 AgentBench tasks with earlier work in history

68%
Less context than full history (3.2×)
37 against 24
Tasks solved, with Context and with full history
14–19
Tasks that change between identical runs

Tileward sent 523,028 context tokens against 1,651,372 for full histories. The context reduction is stable; the 13-task lead is not, so the repeated ordering matters more than the margin.

Report: Give agents useful context, not the whole backlog →
Agent workflowsToken costQuality

Agents in sequence, on half the tokens seven agents handing work down a line

planner → builder-migration → builder-api → validator → tester → reviewer → release
Without Tileward80,031 tokens/run
Routing set up by hand35,226 tokens/run
With Tileward Context40,486 tokens/run
49%
Fewer tokens (2.0×)
39.5M
Tokens saved per 1,000 runs
86%
Of what each step needed, delivered

11,667 tokens of shared state over 21 artifacts. Tileward delivered 86% of the material each step declared it needed; hand-configured routing, which is a person tagging artifacts per problem, delivered 95%. This flow was graded twice, four hours apart: Tileward 0.881 then 0.952, the uncompressed control 0.952 then 1.000. The scores move further between runs than between arms, so treat the token figures as the reliable half. The fuller run.

Write-up: Compression was the easy part →
Agent workflowsToken costQuality

Agents in parallel, on 78% fewer tokens (4.5×) eight agents gathering at once

five parallel gatherers → synthesizer → critic → editor
Without Tileward364,306 tokens/run
Routing set up by hand65,513 tokens/run
With Tileward Context81,388 tokens/run
78%
Fewer tokens (4.5×)
283M
Tokens saved per 1,000 runs
66%
Of what each agent needed, delivered

This is the flow we do worst on, and the saving is the largest. 45,712 tokens of shared state over 32 artifacts. Hand-configured routing delivered every artifact each agent declared it needed; Tileward delivered 66%. Answers were not graded on this flow, but a later run judged them on a blind panel and Tileward came last of four. The whole result.

Write-up: Compression was the easy part, including this loss →
Conversation memoryLongMemEvalQuality

A recall upgrade fixed 14 answers and broke none LongMemEval-S, our own run on a 150-question sample, self-judged

Before the upgrade67 of 140 · 47.9%
After the upgrade81 of 140 · 57.9%
+10.0
Points gained, same questions before and after
14 and 0
Answers fixed, answers broken
9→16
LongMemEval-S temporal-reasoning, of 40

The arms differ by one upgrade to how Tileward Context recalls conversations; store, questions, model and judge are identical. No question that was right before went wrong after. Paired 95% interval on the overall gain: +5.0 to +15.0 points. The upgrade spends a little of the recall budget on itself, so the after arm sometimes carries slightly less material and still wins.

  • Our own harness run of LongMemEval-S on Tileward’s recall path at default settings: a single run of a 150-question sample, 140 scored identically in both arms.
  • Tileward 35B-A3B wrote and judged every answer.
Report: Improve recall without losing correct answers →

Tileward Governance A locked tile, tested by trying to get past it.

GovernanceQuality

2,201 of 2,251 attempts to get past a lock, stopped the same locked request, reworded every way we could

Requests written to get past a lock2,251 tested
Stopped before the model saw them2,201
Got through50
2,251
Requests tested
50
Got through
45 of 50
Ordinary requests allowed
Report: Put policy in front of the model—not inside the prompt →

Agent flows measured on a 120B open-weight model, tokens counted by the serving model rather than estimated. A provider that discounts a repeated prefix narrows what the saving is worth.

Bring your own workload. Point the harness at something you actually run. Talk to an engineer.