# A smaller model should not become a different model

Research report R-003, by Anantha Sharma. Tileward compressed two model families and re-tested
whether sensitive values still survive ordinary copy, extraction and summarization tasks. Across
120 checks per model, compression did not create a directional change; the base model mattered far
more, and the trade-off that did appear was 0.40 MMLU points for Tileward 27B, a research build, against its
uncompressed original.

Page: https://tileward.com/research/what-compression-inherits/

The work extends a finding published by
[Ingot](https://ingot.tools/reports/qwen3-8-27b-glitch-tokens), reproduced independently and then
tested through compression and on a second model.

## How it was measured

- Models: Qwen3.8-27B (the model Ingot examined) and Tileward 27B, a compressed research build of
  it; Qwen3.6-35B-A3B and Tileward 35B-A3B, the compressed model Tileward serves today.
- Battery: 40 realistic identifiers (card numbers, tax IDs, patient record numbers, passwords,
  account details), each through three jobs: copy the record exactly, pull the number out,
  summarise the ticket. That is 120 checks per model, run once straight through and once with
  every check repeated eight times.
- Scoring: Ingot's own published script, run unmodified. Anything other than the value surviving
  intact counts as a loss: refused outright, partly masked, or silently omitted.

## Finding one: compression preserved this behavior

Losses, full size against compressed:

| Model | Full size | Compressed | Difference |
|---|---|---|---|
| Qwen3.8-27B to Tileward 27B | 26 of 120 | 26 of 120 | none at all |
| Qwen3.8-27B, repeated | 188 of 960 | 216 of 960 | too small to call (2.9 point gap, z = +1.57) |
| Qwen3.6-35B-A3B to Tileward 35B-A3B | 4 of 120 | 1 of 120 | too small to call (2.5 point gap in the opposite direction, z = -1.36) |
| Qwen3.6-35B-A3B, repeated | 32 of 960 | 19 of 960 | too small to call (1.4 point gap in the opposite direction, z = -1.85) |

In the first row the same 26 checks failed before and after, not merely the same number of them.

## Finding two: base-model selection mattered more

Handed the same 120 checks, Qwen3.8-27B quietly withheld 21.7% of the records it was asked to write
down and Qwen3.6-35B-A3B withheld 3.3%: a six-fold difference, the largest effect in the study.
Neither model announces it, and under repeated testing all eight items the A3B holds back are items
Qwen3.8 also holds back.

## Finding three: some risks arrive with the model family

- Tileward picked the twelve least-learned word fragments from the models themselves and asked each
  model to repeat them. Qwen3.8-27B managed 5 of 12, the figure Ingot reports.
- The two families share the identical word list (the same 248,044 vocabulary entries); 26 of the
  worst 30 fragments are common to the pair.
- Repeated back correctly: Qwen3.8-27B 5 of 12, Tileward 27B 5 of 12, Qwen3.6-35B-A3B 4 of 12,
  Tileward 35B-A3B 4 of 12.
- The embedding matrices are byte for byte identical between each original model and its compressed
  build (maximum difference exactly 0.0), so compression neither adds this problem nor removes it.

## The cost

On full MMLU (14,042 multiple-choice questions across 57 subjects, five examples before each, both
models measured in one session under identical settings), the original Qwen3.8-27B scored 82.63% and
Tileward 27B, the compressed research build of it, 82.23%.

## Choose the model for behavior, then verify the compressed build

- A model that quietly declines to write something down does not say so: the output is well formed
  and the record reconciles. For anything that archives, logs or reconciles, that is the failure to
  check for.
- The result depends on which identifiers are tested; this run used forty.
- The broken fragments are evidence of a risk, not a measure of how often they turn up in your
  traffic.

## Related

- [Compressed-model eval](https://tileward.com/evals/#compressed-model): size-and-quality comparison for the currently served build
- [Ingot's report](https://ingot.tools/reports/qwen3-8-27b-glitch-tokens): the finding this work reproduces and extends
- [Tileward Models](https://tileward.com/models/): what Tileward serves and what it costs
- [Talk to an engineer](https://calendar.app.google/hg2DzFTxKJgwFWm19): point the same battery at the model and records you handle
- [Research](https://tileward.com/research/): all research reports