# Keep most of the answer rate on 95% less context (19×)

Tileward research report R-006. On LOCOMO, a benchmark of long conversations with questions about what was said in them, a model given the whole transcript answered 63–67% of the answerable questions on 28,365 tokens each. Given what Tileward Context returned, it answered 55% on 1,499 tokens each; given only the newest turns, 8%. When the answer is not in the conversation, Context made one up 40–59% less often.

Page: https://tileward.com/research/a-nineteenth-of-the-context/

## How it was measured

- **Benchmark:** [LOCOMO (Maharana et al.)](https://arxiv.org/abs/2402.17753): ten conversations that run for hundreds of turns over several sittings, and 1,986 questions about them. The first table counts the 1,540 the conversations can answer; the 446 unanswerable ones are scored on whether the model stated an answer anyway.
- **What changed:** only what the model was given (the whole transcript, the newest turns that fit the same budget, what Tileward Context returned, or no conversation at all). The questions, model and judge were the same for every row.
- **Judge:** a separate model, Claude Sonnet 4.6, graded every answer against the reference. The model that answered never graded itself.
- **Models:** a 120B open-weight model answered for the first table. The unanswerable-question comparison adds Tileward 35B-A3B.

## Answerable questions

| What the model was given | Questions answered | Context per question | Against sending everything |
|---|---|---|---|
| Everything, as-is | 63% | 28,365 tokens | Baseline |
| Tileward Context | 55% | 1,499 tokens | 8 points fewer, on 95% less context (19×) |
| The newest turns that fit the same budget | 8% | 2,165 tokens | 55 points fewer, on 92% less (13×) |
| Nothing (no conversation at all) | 0% | 138 tokens | 63 points fewer, on 99.5% less (206×) |

- The whole transcript ran out of room on 99 of the longest questions; leaving those out, it scores 67%.
- Against the newest turns, Tileward Context won 769 questions and lost 41.

## When there is no answer

About a fifth of LOCOMO's questions ask about something the conversation never says; the right response is to say so.

- On Tileward 35B-A3B, Context made an answer up 26% of the time, against 63% with the whole transcript: 59% fewer (95% interval 53–65%).
- On the 120B model it was 31% against 52%: 40% fewer.
- Sending less makes a model say "not mentioned" more often, which also costs answers it could have given. On Tileward 35B-A3B, answerable questions right were 72% with the whole transcript, 54% with Context and 7% with the newest turns.

## Deployment meaning

Use selective context when the whole transcript is too expensive, or too large for the window. Here Tileward Context sent 95% less (18.9×) and finished eight to twelve points behind the whole transcript. Test that trade on your own conversations.

## Related

- [LOCOMO eval card](https://tileward.com/evals/#locomo-long-conversation): the result and limits in one screen
- [LOCOMO paper](https://arxiv.org/abs/2402.17753): the benchmark
- [Tileward Context](https://tileward.com/context/): send only the conversation that matters
- [Research](https://tileward.com/research/): every report