# Improve recall without losing correct answers

Research report R-004, by Anantha Sharma. In one paired A/B on LongMemEval-S, a public benchmark for
long-term conversational memory, one recall upgrade moved Tileward Context from 67 to 81 correct
answers out of 140. Fourteen questions flipped from wrong to right and none flipped back, in one
self-judged run.

Page: https://tileward.com/research/recall-upgrade-ab/

## How it was measured

- Benchmark: the S variant of [LongMemEval](https://arxiv.org/abs/2410.10813) (Wu et al., ICLR
  2025), which tests whether a system can find and use an answer buried in a long, multi-session
  conversation.
- System: the product's own recall path at default settings (the same store and retrieval a
  customer gets), with nothing hand-configured per question.
- Pairing: for each question the conversation history is ingested into a fresh store once; both
  arms recall from that same store and answer the same question, so each question is its own
  control. The arms differ by exactly one upgrade to how Tileward Context recalls conversations.
- Model and judge: answers are written and judged by Tileward 35B-A3B, the model Tileward serves,
  on both arms.
- Sample: 150 questions; 140 scored, because the other 10 carry no answer-bearing turn in the
  dataset. One run, one seed.

## Results

| Arm | Correct | Accuracy |
|---|---|---|
| Before the upgrade | 67 of 140 | 47.9% |
| After the upgrade | 81 of 140 | 57.9% |

- The arms differ by 14 correct answers, or +10.0 percentage points: 14 questions fixed, 0 broken.
- The paired 95% interval on the overall gain is +5.0 to +15.0 points, computed over per-question
  flips within the run.
- The upgrade spends some of the recall budget on itself, and how much depends on how much is
  recalled: about 3.4% more on the several-thousand-token contexts this benchmark pulls. The after
  arm sometimes carries slightly less material into the answer and still wins.

By LongMemEval-S category:

| Category | Before | After |
|---|---|---|
| knowledge-update | 15 of 19 | 17 of 19 |
| multi-session | 12 of 38 | 14 of 38 |
| single-session-assistant | 16 of 17 | 17 of 17 |
| single-session-preference | 1 of 9 | 1 of 9 |
| single-session-user | 14 of 17 | 16 of 17 |
| temporal-reasoning | 9 of 40 | 16 of 40 |

## Reading the result

The supported claim is the paired gain: on the same store, questions, model and judge, this change
added 14 correct answers and removed none. The absolute scores (47.9% and 57.9%) belong to this
harness, a 150-question sample answered and judged by Tileward 35B-A3B, and should not be set beside
published LongMemEval results, which use other answering models, other judges and the full question
set.

Next: a second seed, a larger slice of the benchmark, and a third-party judge arm.

## Related

- [Recall upgrade eval card](https://tileward.com/evals/#recall-upgrade): the matching one-card result
- [LongMemEval](https://arxiv.org/abs/2410.10813): the benchmark paper (Wu et al., ICLR 2025)
- [Evals](https://tileward.com/evals/): what else Tileward measured, and its limits
- [Stop paying every agent to reread everything](https://tileward.com/research/where-context-loses/): the multi-agent study with hand-configured routing
- [Talk to an engineer](https://calendar.app.google/hg2DzFTxKJgwFWm19): run the default recall path on a conversation history you have
- [Research](https://tileward.com/research/): all research reports