Korely on LoCoMo and LongMemEval
This page reports the latest full measurements of Korely on two public benchmarks. On
LoCoMo, with the protocol of the Mem0 paper, answering from Korely's memory
scores 86.9%, against 79.9% when the same model reads the whole
conversation. Reader and judge are gpt-4o-mini in both. On
LongMemEval, oracle split, Korely scores 87.2%. The 86.9% is a
reread: context blocks saved by an earlier full run were read again and graded by
gpt-4o-mini (details below).
These are measures from October 2026, not of the current version. The last full run of both benchmarks was on version 0.1.36 (15 and 16 October 2026). No later release, 1.0.1 to 1.0.3 included, has been run on either benchmark in full, and the code that stores and reads memories has changed since.
gpt-4o-mini.gpt-4o-2024-08-06.Terms
The Glossary defines the words used across the docs. This page uses memory, end user, context block and token budget as defined there. Eight terms are local to this page:
- Mem0: another memory system for agents. Its 2025 paper (arXiv 2504.19413) is the protocol this page follows, and its numbers are the comparison.
- Reread: the context blocks saved by a finished run are given to a reader again, and graded again. Nothing is written to Korely a second time.
- 95% interval: the range in which the true score probably lies, given that only some questions were asked.
- LoCoMo: a public benchmark of ten very long conversations between two people, each with questions about what was said (Maharana et al., ACL 2024).
- LongMemEval: a public benchmark of 500 questions over long chat histories of one user, in six question types (Wu et al., ICLR 2025, arXiv 2410.10813). The oracle split gives each question only the sessions that hold the evidence; the harder splits bury them among unrelated sessions. Some questions are abstentions: the right answer is that the history does not say.
- Protocol: the fixed recipe of a benchmark: which questions, which model reads the memory and writes the answer (the reader), which model grades it (the judge), and which judge prompt. Scores are comparable only under the same protocol.
- LLM-as-a-judge: a language model compares each answer with the reference answer and marks it correct or wrong. The score is the share marked correct.
- Category: LoCoMo sorts questions into five kinds. This page uses four: multi-hop (combine several statements), temporal (ask about when), open-domain (need knowledge beyond the conversation), single-hop (one statement). The fifth, adversarial, is left out, as Mem0 and Zep do.
LoCoMo
How it was run. Each person in a conversation is an end user of their own, and
every turn is written once as a memory, with the session's date as its timestamp: 5,882
memories. For each of the 1,540 questions of categories 1 to 4, Korely builds a context block of
2,000 tokens per person (4,000 in all), and the reader answers from it. The judge prompt is
the one published by Mem0, word for word, at temperature 0 (the model's randomness switched off). The reader prompt is ours. "Mem0's
protocol" here means that judge prompt, gpt-4o-mini as reader and judge (the model
of the Mem0 paper's experiments,
arXiv 2504.19413), and categories 1 to 4.
| Setup | Reader and judge | Correct |
|---|---|---|
Original run: Korely memory, graded by gpt-oss-120b. The first row reads these same context blocks again | gpt-oss-120b | 80.8% (1,245) |
| Korely memory, version 0.1.36, 16 October 2026 | gpt-4o-mini | 86.9% (1,338) |
| Whole conversation, no Korely in the loop, 7 October 2026 | gpt-4o-mini | 79.9% (1,231) |
With the same reader and the same judge, the memory answers better than handing the model the whole history, with about a fifth of the tokens: the prompt with the memory blocks averaged about 3,900 tokens, a whole conversation is about 19,000.
Where the gap comes from
| Category | Questions | Korely memory | Whole conversation |
|---|---|---|---|
| Temporal | 321 | 253 | 167 |
| Multi-hop | 282 | 233 | 228 |
| Single-hop | 841 | 789 | 775 |
| Open-domain | 96 | 63 | 61 |
Most of the distance comes from the questions about time: 86 of the 107 extra correct answers. Korely stores each memory with its date and prints the context block under those days, so "when did she go to the support group?" is answered from a dated memory instead of from a long text where the date sits in a session header. On the other three categories the two setups are close.
The judge moves the score by about 7 points
Read every LLM-judged number with its judge next to it. In an earlier full run
(version 0.1.20-dev), gpt-oss-120b wrote the answers. We graded those same
answers twice: 81.0% with gpt-oss-120b as judge, 88.4% with
gpt-4o-mini, 7.4 points higher with no answer changed. The same effect is behind
the 80.8% and 86.9% in the table above: most of that distance is the judge, not the reader.
gpt-4o-mini is a lenient judge: an independent audit of
LoCoMo by a third party,
locomo-audit, measures that it accepts
62.81% of answers that are vague but wrong. A score graded by it compares only with other
scores graded by it.
LongMemEval
How it was run. The benchmark's own protocol, with its reader prompt and its
judge prompts (one per question type, and one for abstentions). Each question is an end user;
every turn of 12 characters or more is a memory: 10,949 memories. Korely returns a context block
of 4,000 tokens, and gpt-4o-2024-08-06 reads and judges. Result: 436 of 500,
87.2% (95% interval 84.0 to 89.8).
| Question type | Correct |
|---|---|
| single-session-user | 69 of 70 |
| single-session-assistant | 55 of 56 |
| single-session-preference | 23 of 30 |
| multi-session | 108 of 133 |
| knowledge-update | 65 of 78 |
| temporal-reasoning | 116 of 133 |
Tracing each of the 64 wrong answers to its evidence: in 49 the evidence was in the context block and the reader got it wrong; in 8 the search found it but it did not fit in the block; in 1 the memory was never written; in 1 the search did not find it; and 5 have no evidence marked in the dataset.
The oracle split is the easiest of the three. The LongMemEval paper reports 87.0% for GPT-4o when it is given the oracle sessions themselves. So on this split the memory matches handing the reader the evidence; it does not show how well Korely finds evidence among unrelated sessions. Many published LongMemEval numbers are on the harder splits, and ours is not comparable with them. We have not run those splits in full.
What these numbers do not show
- Not the current version. The full runs are of 0.1.36. Releases since then changed how memories are written and how the context block is built, and neither benchmark has been run in full on them.
- A reread, not a fresh run. The 86.9% is a second reading: the context blocks saved by the
gpt-oss-120brun (the 80.8%) were read again and judged bygpt-4o-mini. In the reread, the instruction note for the reader goes in the user message instead of the system prompt, so it is not the same measurement as a fresh run withgpt-4o-mini. - Variance. Three full LoCoMo runs on two versions gave 81.0%, 81.9% and 80.8% with
gpt-oss-120b, and 86.2%, 86.4% and 86.9% when reread withgpt-4o-mini. The paper of Mem0 averages ten runs; each row here is one run, and LongMemEval has one run with no repeat. - Prompts. Our reader prompt gives the date of the question and the dates of the sessions, in the memory setup and in the whole-conversation baseline. That is why our whole-conversation baseline (79.9%) is above the 72.9% of the Mem0 paper with the same model. The clean comparison is the internal one: same prompt, reader and judge.
- A ceiling below 100. locomo-audit finds that 6.4% of LoCoMo's reference answers are wrong (99 of 1,540), which puts the honest ceiling at 93.57%.
- Not point-in-time reads. The time questions above ask when something happened. Neither benchmark asks what was true on a given date or what the store knew then (the
as_ofreads); those are covered by tests, not by a public benchmark. Both benchmarks are in English, and no read latency figure is published here.
Published scores under a comparable protocol
For context, these are scores other systems report on LoCoMo under a comparable protocol:
categories 1 to 4, accuracy graded by an LLM, gpt-4o-mini in the setup. They were
measured by their authors, with their own prompts and runs, and are not a ranking. Korely sits
in the leading group of these, with the caveats above.
| System | Reported | Source and what it states (read 1 November 2026) |
|---|---|---|
| Mem0 | 66.88% | arXiv 2504.19413, Table 2. The text names gpt-4o-mini for the pipeline and does not name the judge model. |
| Mem0 with graph | 68.44% | Same table. |
| Zep, as run in the Mem0 paper | 65.99% | Same table. |
| Whole conversation, as run in the Mem0 paper | 72.90% | Same table. |
| MemOS | 75.80% | arXiv 2507.03724, Table 3. The same gpt-4o-mini foundation model for all methods. |
| EverMemOS | 86.76% | arXiv 2601.02163, Table 1, gpt-4o-mini backbone; judged by gpt-4o-mini and two auxiliary judges. |
| SmartSearch | 91.9% | arXiv 2603.15599: gpt-4o-mini as answer model and judge, four categories, the setup of the MemOS paper. |
| Korely | 86.9% | This page. gpt-4o-mini reader and judge, Mem0's judge prompt, one reread, version 0.1.36, 16 October 2026. |
Mem0 is another system, and its paper used its own prompts. It reports 66.9% there; with the same reader model and Mem0's judge prompt we measure 86.9%, with a lenient judge and our own reader prompt, so the two numbers are not directly comparable. The two papers also differ on the whole-conversation baseline (72.9% there, 79.9% here), for the prompt reasons above. That compares our October 2026 run against a number from Mem0's 2025 paper and setup. We did not re-run Mem0. Scores above 90% on LoCoMo mostly come from stronger readers, several searches per question or a lenient judge, which is why the reader and the judge are named next to every number on this page.
What we say, and what we do not
- With the same reader and the same judge, Korely's memory answers better than giving the model the whole history, with about a fifth of the tokens, and mostly on time questions.
- Korely scores 86.9% on LoCoMo with the protocol of the Mem0 paper (
gpt-4o-minias reader and judge), where Mem0 reports 66.9%. - We do not claim first place on either benchmark, and we do not quote a number without its reader and judge.
See also
- Research & architecture, the papers behind the design
- Temporal facts, how dates are stored
- Glossary, shared terms