A smaller model should not become a different model.

Compression makes a model smaller and cheaper to run. The obvious worry is that it also changes how the model behaves. Testing one model cannot answer that, because anything you find might just be a quirk of that model. So we tested two, each in its original and compressed formBoth compressed builds use the same recipe we ship. The original models are Qwen3.8-27B and Qwen3.6-35B-A3B, published by Qwen. Tileward 27B is a research build and is not on our API..

Each one was handed 40 realistic identifiers, the sort any business holds: card numbers, tax IDs, patient record numbers, passwords, account details. Each identifier went through three ordinary jobs. Copy this record exactly. Pull the number out of it. Summarise the ticket. Forty identifiers across three jobs is 120 checks per modelScored with Ingot's own published script, run unmodified so our numbers and theirs mean the same thing. Anything other than the value surviving intact counts as a loss: refused outright, partly masked, or silently omitted., and we ran the whole thing twice, once straight through and once repeated eight times overAsking a model the same question once tells you very little, because a single answer can go either way by chance. Repeating each of the 120 checks eight times turns a yes-or-no into a rate. We committed to this second pass before seeing any compressed result..

This is the result we went looking for, and it is a flat no. In both models, under both ways of asking, compression left the behaviour where it found it.

The first row is the cleanest evidence: the same 26 checks failed before and after, not merely the same number of them. And where the figures do wobble, the two models wobble in opposite directions, one slightly up and one slightly down. A real effect points one way. This one points both.

Compression moved nothing. Swapping the model moved almost everything. Handed the same 120 checks, one of these models quietly withheld 21.7% of the records it was asked to write down. The other withheld 3.3%.

That is a six-fold difference21.7% against 3.3% on a single pass, and 19.6% against 3.3% when repeated. It is the largest effect in the study by a wide margin, and the only one in this battery that no reasonable reading calls chance. The compression cost on the knowledge test below is separately real, and much smaller. in how often a model decides, without being asked, to hold back information you gave it. Neither model announces it. The record still balances. If you are choosing a model for anything that archives, logs or reconciles, this difference matters more than most of what appears on a leaderboard.

Every model is built on a fixed list of word fragments it is allowed to use. Some of those fragments are so rare that the model barely learns them, and it turns out a model cannot reliably repeat back a fragment it never learned. Ingot found this in one model. We found it in two.

Working from the models themselves rather than Ingot's list, we picked out the twelve least-learned fragments and asked each model to repeat them. Qwen3.8-27B managed 5 of 12The same figure Ingot reports, reached independently from our own candidate list, at a mean embedding norm of 0.8998 against their quoted 0.899., which is exactly what Ingot reported.

Here is the part a single model cannot show. These two models were built separately and released apart, but they share the identical word listBoth use the same 248,044 vocabulary entries with the same identifiers, compared entry by entry. Same list, different training runs., and the same fragments are broken in both: 26 of the worst 30Ranked against each model's own average rather than in absolute terms. The two models are different widths, and an absolute cutoff would unfairly flag the narrower one. are common to the pair, and the single worst one is the same in each. The flaw is not really a property of either model. It came with the word list they were both built on.

Our compressed builds score exactly what the originals score, and they have to. The part of the model that holds the word list comes through compression completely untouchedThe embedding matrices are byte for byte identical between each original model and our compressed build of it. Maximum difference: exactly 0.0, in both cases.. That is the useful part for anyone buying a compressed model. You cannot compress your way into this problem, and you cannot compress your way out of it either.

Compression is not free, and we would rather publish the price than let anyone assume there wasn't one. On a standard knowledge testFull MMLU: 14,042 multiple-choice questions across 57 subjects, five examples given before each, run on the pinned public harness with both models measured in one session under identical settings. of just over fourteen thousand questions, the original scored 82.63% and our compressed build scored 82.23%.

Four tenths of a point. Compared loosely that looks like rounding, but the two models sat the same exam, and comparing question by questionPaired McNemar on the per-question outcomes: 311 questions the original got right and ours got wrong, against 255 the other way. Exact p = 0.021. shows it is a real difference rather than luck. So: a measured 0.4 points. Not a rounding error, not a catastrophe, and entirely separate from the record-keeping behaviour above, which did not move at all.

Our first run used Ingot's original ten identifiers, and it produced a much neater conclusion. The two models appeared to protect completely different things. One guarded an API key and shrugged at credit cards. The other did the reverse. No overlap at all. Two models with incompatible instincts about what is sensitive is a genuinely interesting claim, and we had the numbers to make it.

It fell apart at forty identifiers. The second model's refusals turned out to be a smaller version of the first model'sUnder repeated testing the A3B holds back eight items. All eight are items Qwen3.8 also holds back. Not one is unique to it., the same items, just far rarer. The striking difference was never a property of the models. It was a property of which ten identifiers happened to be in the test.

We are including this because it is the entire argument for widening a test before publishing from it. A small test that produces a clean, quotable, wrong answer is worth more as a warning than the answer would ever have been as a headline.

A model that quietly declines to write something down does not tell you it declined. The output is well formed, the field is filled in, the record reconciles against everything else. Nothing downstream raises a hand. That is the failure worth caring about here, and it turns out to have almost nothing to do with compression.

What this run cannot tell you: whether a compression method designed deliberately to change this behaviour could manage it, because ours was not designed that way. How often these broken fragments turn up in your particular traffic, since a rarely-learned fragment is evidence of a risk rather than proof of one. Or whether forty identifiers is enough, when we have already shown that ten was not.