# See what Tileward changes before you deploy it

Every test Tileward has run, each with its result in one line: compressed model builds measured against the original, how much less Tileward Context sends and what happened to the answers, and a locked tile tested by trying to get past it. Each result below links to its card on the page, and each card links to its write-up.

Page: https://tileward.com/evals/

## Tileward Models: smaller builds, measured against the original

- [Tileward 35B-A3B on MMLU](https://tileward.com/evals/#compressed-model): 71.9 → 24.1 GB, 66% smaller (3.0×). Level with the publisher's own FP8 build on MMLU, 82.8% against 82.9%, and 36% smaller than that build (1.55×). Five-shot, 14,042 questions, lm-evaluation-harness, no sampling; one run, one machine, same base model.
- [Tileward 35B-A3B against NVIDIA's NVFP4 build](https://tileward.com/evals/#compressed-model-a3b-nvfp4): 71.9 → 24.1 GB, 66% smaller (3.0×). Closer to the original than NVIDIA's NVFP4 build of the same base, a KL distance of 0.023 against 0.040 on next-word probabilities, and level with it on MMLU, 82.9% against 82.6%, and fabrication (2.3% of facts made up against 2.1%); ahead on MMLU-Redux, 90.7% against 90.0%, mostly on questions NVIDIA's build did not finish within the 12,288-token limit. NVIDIA's file is 2.8% smaller (1.03×); NVIDIA's build decodes faster for 1 and 16 callers while ours is faster for 64, has room for 12% more cached tokens at our hosted memory setting, and made 240 of 240 tool calls against our 238. One session on one 80 GB Hopper-generation card.
- [Tileward Qwen3.8-27B against NVIDIA's NVFP4 build](https://tileward.com/evals/#compressed-model-27b): 55.6 → 20.1 GB, 64% smaller (2.8×). Level with NVIDIA's NVFP4 build of the same base on MMLU, 82.4% against 82.1%, and 8.5% smaller; level on MMLU-Redux, fabrication (4.0% of facts made up against 4.8%) and speed; 238 of 240 tool calls right against 240, and NVIDIA's build stays closer to the original on next-word agreement, 89.0% against 87.8%. One session on one machine.
- [Qwen3.8-27B in an agent loop](https://tileward.com/evals/#compressed-model-agents): the earlier TWX48 build finished 43.7% of 142 AgentBench OS tasks against 45.1% for the uncompressed model, a gap inside the test's margin. Each task run twice per build, greedy decoding, thinking off, the same tasks and settings for all four builds.
- [Tileward PersonaPlex 7B](https://tileward.com/evals/#compressed-voice-model): 16.7 → 6.3 GB, 62% smaller (2.7×). On one laptop it makes each 80 ms frame in 20 ms against about 40, and chooses the original's next word 96% of the time, over 375 word moments in four conversations.
- [Tileward VoiceChat 11B](https://tileward.com/evals/#compressed-voice-agent): 44.4 → 11.5 GB, 74% smaller (3.9×). Picks the right tool 55 of 80 times against 52 for the original, on spoken requests in six synthetic voices scored on the first tool call; a gap of three or four calls in 80 is within chance. On one NVIDIA 96 GB card it runs at the original's speed. VoiceChat is NVIDIA's model under OpenMDW-1.1; the build is a modified version of it under the same agreement.
- [Whisper large-v3 and large-v3-turbo](https://tileward.com/evals/#compressed-speech-recognition): a TWX4 build of each, measured on one 20 GB card. Large-v3: 3.09 → 0.98 GB, 68% smaller (3.2×), at most 0.3 points more error than the original on nine of ten test sets and 1.06 on Danish, and 152 live calls against 140. Turbo: 1.62 → 0.56 GB, 65% smaller (2.9×), level with turbo on six sets and behind on four by 0.09 to 0.68 points, and 331 live calls against 301. Word error rate, or character error rate for Mandarin and Japanese, on LibriSpeech, FLEURS in seven languages and 384 synthetic call-centre clips; a live call is one second of audio transcribed per second.

## Tileward Context: less sent to the model, and what happened to the answers

- [Long conversations, LOCOMO](https://tileward.com/evals/#locomo-long-conversation): 95% less context than the whole transcript (18.9×), 1,499 tokens per question against 28,365. 55% of questions answered, against 63–67% with the whole transcript, judged by a separate model.
- [One long conversation, asked back](https://tileward.com/evals/#long-conversation): on one 142-turn conversation, 97–99% fewer tokens (38–82×), and 20 of 20 questions answered, the same as resending the whole transcript. Qwen3.5-9B, exact substring match.
- [An agent with a long backlog](https://tileward.com/evals/#agent-backlog): on 104 AgentBench tasks, 68% less context than full history (3.2×), and 37 tasks solved against 24 with full history. The context reduction is stable; the lead moves by 14–19 tasks between identical runs.
- [Agents in sequence](https://tileward.com/evals/#engineering-pipeline): seven agents handing work down a line, 49% fewer tokens (2.0×). Tileward delivered 86% of the material each step declared it needed, against 95% for routing set up by hand. Graded scores moved further between runs than between arms, so the token figures are the reliable half.
- [Agents in parallel](https://tileward.com/evals/#research-fan-out): eight agents gathering at once, 78% fewer tokens (4.5×). Tileward delivered 66% of the material each agent needed, where hand-configured routing delivered all of it. This is the flow Tileward does worst on: a later run judged the answers on a blind panel, and Tileward came last of four.
- [A recall upgrade](https://tileward.com/evals/#recall-upgrade): 47.9% → 57.9% right on LongMemEval-S, with 14 answers fixed and none broken on the 140 questions scored. Our own run on a 150-question sample; Tileward 35B-A3B wrote and judged every answer.

## Tileward Governance: a locked tile, tested by trying to get past it

- [Requests written to get past a lock](https://tileward.com/evals/#policy-bypass): 2,201 of 2,251 stopped before the model saw them, 50 got through; 45 of 50 ordinary requests allowed. Measured as coverage of a hand-authored English corpus across direct locks and prohibited material placed beside allowed work.

## How it was measured

Agent flows were measured on a 120B open-weight model, with tokens counted by the serving model rather than estimated. A provider that discounts a repeated prefix narrows what the saving is worth.

## Related

- [A smaller model should not become a different model](https://tileward.com/research/what-compression-inherits/): the Tileward 35B-A3B compression report
- [Shrinking PersonaPlex](https://tileward.com/research/a-smaller-voice-model/) and [compressing VoiceChat](https://tileward.com/research/a-smaller-voice-agent/): the voice model reports
- [Shrinking Whisper](https://tileward.com/research/a-smaller-speech-recogniser/): the speech recognition report
- [Long conversations](https://tileward.com/research/a-nineteenth-of-the-context/) and [agent backlogs](https://tileward.com/research/a-third-of-the-context/): the Tileward Context reports
- [Compression was the easy part](https://tileward.com/research/where-context-loses/): the multi-agent flows, including the loss
- [Improve recall without losing correct answers](https://tileward.com/research/recall-upgrade-ab/): the recall upgrade report
- [Put policy in front of the model](https://tileward.com/research/governance-security-control/): the Tileward Governance report
- [All research write-ups](https://tileward.com/research/)
- [Talk to an engineer](https://calendar.app.google/hg2DzFTxKJgwFWm19): point the harness at a workload you actually run