Improve recall without losing correct answers.
In one Both arms answer the identical questions from the identical store, so each question is its own control. The result is a set of per-question flips, not two separate scores that happen to differ. on LongMemEval-S, a public benchmark for long-term conversational memory (Wu et al., ICLR 2025), one recall upgrade moved Tileward Context from 67 to 81 correct answers out of 140. Fourteen questions flipped from wrong to right; none flipped back. Both arms used the same store, questions, model, and judge. This is one self-judged run, and the upgrade itself is still undisclosed.
Change one thing, then compare every question with itself.
LongMemEval (Wu et al., ICLR 2025) tests whether a system can find and use an answer buried in a long, multi-session conversation. We ran its S variant against the product's own recall path at default settings (the same store and retrieval a customer gets), with nothing hand-configured per question.
For each question, the conversation history is ingested into a fresh store once. Both arms then recall from that same store and answer the same question. The arms differ by exactly one upgrade to how Tileward Context recalls conversations. Answers are written and judged by Tileward 35B-A3B, the model we serve. The sample is 150 questions; 140 scored, because the other 10 carry no answer-bearing turn in the dataset, so neither arm saw them.
Fourteen answers improved; none regressed in this run.
The two arms differ by 14 correct answers, or +10.0 percentage points.
| Arm | Correct | Accuracy |
|---|---|---|
| Before the upgrade | 67 of 140 | 47.9% |
| After the upgrade | 81 of 140 | 57.9% |
Because the arms are paired, the gain decomposes into flips: 14 questions fixed, 0 broken. The paired 95% interval on the overall gain is +5.0 to +15.0 points, computed over per-question flips within the run. See the matching eval card.
The upgrade spends a little of the recall budget on itself, so the after arm sometimes carries slightly less material into the answer and still wins. How much it spends depends on how much you recall: about 3.4% more on the several-thousand-token contexts this benchmark pulls, but +31% and +22% on the much smaller payloads of the long-conversation workload on Evals, where a per-item cost is a large share of a small total.
Where it moved.
LongMemEval labels every question with one of its own categories.
| LongMemEval-S category | Before | After |
|---|---|---|
| knowledge-update | 15 of 19 | 17 of 19 |
| multi-session | 12 of 38 | 14 of 38 |
| single-session-assistant | 16 of 17 | 17 of 17 |
| single-session-preference | 1 of 9 | 1 of 9 |
| single-session-user | 14 of 17 | 16 of 17 |
| temporal-reasoning | 9 of 40 | 16 of 40 |
Trust the paired gain, not 57.9% in isolation.
The supported claim is narrow: on the same store, questions, model, and judge, this change added 14 correct answers and removed none. The absolute scores (47.9% and 57.9%) belong to this harness and should not travel without the setup.
Same judge on both sides. Tileward 35B-A3B graded every answer and has not been checked against a human grader on this run, so the comparison and the absolute level may both move.
Not comparable to published LongMemEval results. Those use other answering models, other judges, and the full question set. This is a 150-question sample.
One run, one seed. A second seed is on the list under Next.
What we are running next.
A second seed and a larger slice of the benchmark before the level is quoted anywhere. And a third-party judge arm, to put a bound on how much self-grading flatters.
Already done. Because the upgrade spends recall budget, we expected the long-conversation figures on Evals to get worse and re-measured them to find out how much. They got better: that card now reads 38–83 tokens per question where it read 103–127, because unrelated engine work had moved it further than this upgrade costs. The control arm shows the same thing. The gain belongs to that engine work, not to this upgrade.
Run the default recall path on a conversation history you actually have and see what comes back. Talk to an engineer, or read what else we measured, and its limits.