Improve recall without losing correct answers.

LongMemEval (Wu et al., ICLR 2025) tests whether a system can find and use an answer buried in a long, multi-session conversation. We ran its S variant against the product's own recall path at default settings (the same store and retrieval a customer gets), with nothing hand-configured per question. That distinction matters to us.

For each question, the conversation history is ingested into a fresh store once. Both arms then recall from that same store and answer the same question. The arms differ by exactly one upgrade to how Tileward Context recalls conversations. Nothing else in the path changes. Answers are written and judged by Tileward 35B-A3B, the model we serve, with the same judge grading both arms. The sample is 150 questions; 140 scored, because the other 10 carry no answer-bearing turn in the dataset, so neither arm saw them. One run, one seed.

The two arms differ by 14 correct answers, or +10.0 percentage points. Because every question is paired, we can also see the direction of every change.

Arm	Correct	Accuracy
Before the upgrade	67 of 140	47.9%
After the upgrade	81 of 140	57.9%

Because the arms are paired, the gain decomposes into flips: 14 questions fixed, 0 broken. The paired 95% interval on the overall gain is +5.0 to +15.0 points, computed over per-question flips within the run. The gain is one-directional: nothing the product already answered correctly was lost to the change. The matching eval card keeps the same denominator and interval in a compact view.

The upgrade is not free. It spends a little of the recall budget on itself, so the after arm sometimes carries slightly less material into the answer and still wins. How much it spends depends on how much you recall: about 3.4% more on the several-thousand-token contexts this benchmark pulls, but +31% and +22% on the much smaller payloads of the long-conversation workload on Evals, where a per-item cost is a large share of a small total. A percentage overhead quoted without the payload size it was measured on is not a number worth carrying.

LongMemEval labels every question with one of its own categories. Here is the full split, weak rows included.

LongMemEval-S category	Before	After
knowledge-update	15 of 19	17 of 19
multi-session	12 of 38	14 of 38
single-session-assistant	16 of 17	17 of 17
single-session-preference	1 of 9	1 of 9
single-session-user	14 of 17	16 of 17
temporal-reasoning	9 of 40	16 of 40

The movement is concentrated: one category contributes exactly half of the net gain, three move by one or two questions, and the category we score worst on does not move at all (1 of 9 in both arms). An upgrade that helps one kind of question does nothing for our weakest kind, and the table says so. The two large categories that stay hard, at 14 of 38 and 1 of 9 after the upgrade, are where the product's work is, not where its story is.

The supported claim is narrow: on the same store, questions, model, and judge, this change added 14 correct answers and removed none. The absolute scores (47.9% and 57.9%) belong to this harness and should not travel without the setup.

Same judge on both sides. Tileward 35B-A3B graded every answer and has not been checked against a human grader on this run. Using the same judge reduces one source of asymmetry, but it does not prove that a human grader would preserve the gain. The comparison and the absolute level may both move.

Not comparable to published LongMemEval results. Those use other answering models, other judges, and the full question set. This is a 150-question sample.

One run. The interval is across questions within it. A second seed is on the list under Next.

Name the upgrade, when the review closes, and update this post rather than replace it. A second seed and a larger slice of the benchmark before the level is quoted anywhere. And a third-party judge arm, to put a bound on how much self-grading flatters.

One item from this list is already done, and it went against our own prediction. Because the upgrade spends recall budget, we expected the long-conversation figures on Evals to get worse and re-measured them to find out how much. They got better: that card now reads 38–83 tokens per question where it read 103–127, because months of unrelated engine work had moved it further than this upgrade costs. The control arm shows the same thing. The gain belongs to that engine work, not to this upgrade, and saying so is the only way the two numbers can sit on one site honestly. Two lessons we would rather publish than keep: a figure measured on an old build is not a claim about the current one, and predicting the direction of a change is not a substitute for measuring it.