A smaller model should not become a different model.
Compression only creates deployment value if the behavior you selected the model for survives. We shrank two model families and tested a subtle operational risk: whether sensitive values still survive ordinary copy, extraction, and summarization tasks. Across 120 checks per model, compression did not create a directional change. The base model mattered far more, and the paired knowledge eval measured the trade-off that did appear: 0.40 MMLU points for Tileward 27B, a research build, against its uncompressed original.
For the currently served build's size-and-quality comparison, see the compressed-model eval.
This work extends a finding published by Ingot. We reproduce their result independently, then test whether it survives compression and whether it shows up in a model they never looked at.
Compress the deployment, then re-test the behavior.
Compression makes a model smaller and cheaper to run. The obvious worry is that it also changes how the model behaves. We tested two models, each in its Both compressed builds were made the way the Tileward 35B-A3B we serve today is made. The original models are Qwen3.8-27B and Qwen3.6-35B-A3B, published by Qwen. Tileward 27B is a research build..
| Model | Size | Why it is here |
|---|---|---|
| Qwen3.8-27B | Full | The model Ingot examined |
| Tileward 27B | Compressed | Our version of it |
| Qwen3.6-35B-A3B | Full | A second, unrelated model |
| Tileward 35B-A3B | Compressed | The model we serve today |
Each one was handed 40 realistic identifiers, the sort any business holds: card numbers, tax IDs, patient record numbers, passwords, account details. Each identifier went through three ordinary jobs. Copy this record exactly. Pull the number out of it. Summarise the ticket. Forty identifiers across three jobs is Scored with Ingot's own published script, run unmodified so our numbers and theirs mean the same thing. Anything other than the value surviving intact counts as a loss: refused outright, partly masked, or silently omitted., and we ran the whole thing twice, once straight through and once Asking a model the same question once tells you very little, because a single answer can go either way by chance. Repeating each of the 120 checks eight times turns a yes-or-no into a rate. We committed to this second pass before seeing any compressed result..
Compression preserved this behavior across both model families.
| Model | Full size | Compressed | Difference |
|---|---|---|---|
| Qwen3.8-27B to Tileward 27B | 26 of 120 | 26 of 120 | none at all |
| Qwen3.8-27B, repeated | 188 of 960 | 216 of 960 | A 2.9 point gap. Two-proportion z = +1.57, and because repeats of the same check are correlated that is already a generous reading. |
| Qwen3.6-35B-A3B to Tileward 35B-A3B | 4 of 120 | 1 of 120 | A 2.5 point gap in the opposite direction. Two-proportion z = −1.36. |
| Qwen3.6-35B-A3B, repeated | 32 of 960 | 19 of 960 | A 1.4 point gap, again in the opposite direction to the other model. Two-proportion z = −1.85. |
In the first row the same 26 checks failed before and after, not merely the same number of them.
Base-model selection mattered more than compression.
Handed the same 120 checks, one of these models quietly withheld 21.7% of the records it was asked to write down. The other withheld 3.3%.
That is a 21.7% against 3.3% on a single pass, and 19.6% against 3.3% when repeated. It is the largest effect in the study by a wide margin, and the only one in this battery that no reasonable reading calls chance. The compression cost on the knowledge test below is separately real, and much smaller. in how often a model decides, without being asked, to hold back information you gave it. Neither model announces it. The other model's refusals are Under repeated testing the A3B holds back eight items. All eight are items Qwen3.8 also holds back. Not one is unique to it., the same items, just far rarer.
Some operational risks arrive with the model family.
Every model is built on a fixed list of word fragments. Some are so rare that the model barely learns them, and a model cannot reliably repeat back a fragment it never learned. Ingot found this in one model. We found it in two.
Working from the models themselves rather than Ingot's list, we picked out the twelve least-learned fragments and asked each model to repeat them. Qwen3.8-27B managed The same figure Ingot reports, reached independently from our own candidate list, at a mean embedding norm of 0.8998 against their quoted 0.899., which is exactly what Ingot reported.
These two models were built separately and released apart, but they share Both use the same 248,044 vocabulary entries with the same identifiers, compared entry by entry. Same list, different training runs., and the same fragments are broken in both: Ranked against each model's own average rather than in absolute terms. The two models are different widths, and an absolute cutoff would unfairly flag the narrower one. are common to the pair, and the single worst one is the same in each. The flaw came with the word list they were both built on.
| Model | Repeated back correctly | Survived being copied into a record |
|---|---|---|
| Qwen3.8-27B | 5 of 12 | 7 of 12 |
| Tileward 27B | 5 of 12 | 5 of 12 |
| Qwen3.6-35B-A3B | 4 of 12 | 7 of 12 |
| Tileward 35B-A3B | 4 of 12 | 7 of 12 |
The part of the model that holds the word list comes through compression The embedding matrices are byte for byte identical between each original model and our compressed build of it. Maximum difference: exactly 0.0, in both cases., so compression neither adds this problem nor removes it.
We put a number on the trade-off that did move.
On a Full MMLU: 14,042 multiple-choice questions across 57 subjects, five examples given before each, run on the pinned public harness with both models measured in one session under identical settings. of just over fourteen thousand questions, the original Qwen3.8-27B scored 82.63% and Tileward 27B, our compressed research build of it, scored 82.23%.
Four tenths of a point. The two models sat the same exam, and Paired McNemar on the per-question outcomes: 311 questions the original got right and ours got wrong, against 255 the other way. Exact p = 0.021. shows it is a real difference rather than luck.
Choose the model for behavior, then verify the compressed build.
A model that quietly declines to write something down does not tell you it declined: the output is well formed and the record reconciles. For anything that archives, logs or reconciles, that is the failure worth checking for, and it has almost nothing to do with compression.
Identifier set. On Ingot's original ten identifiers the two models appeared to protect completely different things; at forty they did not, so the result depends on which identifiers are tested, and this run used forty.
Your traffic. The broken fragments are evidence of a risk, not a measure of how often they turn up in your traffic.
Point the same battery at the model you are actually deploying, against the records you actually handle, and we will publish what it does. Talk to an engineer, compare the current size-and-quality eval, or read what we serve and what it costs.