Does a memory layer still beat naive context?

I build a small memory engine for AI agents. And I keep running into the same wall. Every memory product quotes one accuracy number. 90 percent. 94 percent. 96 percent. Every time I try to put two of them side by side, I can't, because each one was measured with a different model grading the answers, on a different version of the test. Some of those numbers aren't even "did it answer correctly". They are "did it find the right document", which is a different thing wearing the same costume.

So I stopped trying to win the number game, and I asked a smaller, more honest question. It is the only question where a memory layer actually has to earn its keep.

Picture a library. You walk up with a question. A librarian goes to the shelves, brings you back a few pages, you read them, and you answer. Almost everything in agent memory is some version of that librarian. There are two librarians I care about here.

The first one is lazy. When you ask a question, she just hands you the last few pages you happened to read. No thought about whether they are relevant. That is a "recency window": keep the most recent slice of the conversation and hope the answer is in there.

The second one is smart. She has actually read your whole history, and she brings you the few pages that actually answer the question, even if you talked about it months ago. That is what our memory layer does: it selects.

Now the trick that makes the test fair. I let both librarians bring you the same number of pages. Same size stack. If the smart librarian could only win by handing you a thicker pile, she wouldn't have proven anything, she'd just be spending more. So I fix the budget and the only thing that can differ is which pages they picked.

The person reading those pages and writing the answer is an AI model. I call it the student. A second AI, the judge, checks the student's answer against the correct one. Same student, same judge, for both librarians. That way, if the student or the judge has any quirk, it hits both sides equally and cancels out.

I ran this on a public benchmark called LongMemEval, 178 questions about long, multi-session chats. Same budget for both librarians (about 1,900 tokens, roughly a couple of pages).

The smart librarian's pages let the student answer 76 percent correctly. The lazy librarian's recent pages: 42 percent. That is plus 34 points. Almost twice as many right answers, for the same number of pages.

And it makes sense where the gap comes from. The lazy librarian does fine when the answer is in something you just talked about. She falls apart when the answer lives in an old session (how many times did this happen, what did you say three months ago), because the recent pages simply don't reach back that far. She guesses, or she shrugs.

178 questions is a small class. Maybe the smart librarian just got lucky on a handful of them. Before I claim anything, I have to rule that out. There are three standard tools for it. Let me explain each one as if you have never seen it, then show you our number.

Tool 1: the p-value

Imagine the two librarians were secretly equally good. Even then, luck alone would sometimes make one look a bit better, the same way flipping two coins won't always give the same number of heads. The p-value is the probability that luck alone, between two librarians who are really the same, would produce a gap at least as big as the one I actually saw. A tiny p-value means luck basically never does this, so the gap is real. One way to write it is to add up the chance of every outcome at least as lopsided as ours, treating each disagreement like a coin flip:

Our number comes out to 0.00000000000078, about one in a trillion. In plain words: if the two librarians were actually equally good, I would have to re-run this whole test about a trillion times to see a 34 point gap show up even once by accident. So no, we did not get lucky.

Tool 2: McNemar's test

Both librarians answered the exact same 178 questions. Some questions are just easy (both get them right) and some are just hard (both miss them). Those tell you nothing about who is better. So McNemar's test throws away every question the two agreed on, and looks only at the disagreements. If the two were truly equal, each disagreement should be a 50/50 coin flip. In our data they disagreed on 79 questions, and the smart librarian won 70 of them, the lazy one only 9. That is like getting 70 heads in 79 tosses. A fair coin never does that. Here b is the smart librarian's wins, c the lazy one's:

The top is the size of the gap between 70 and 9, minus 1, squared; the bottom is the total disagreements. A value of 45.6 is enormous for this test. The smart librarian is genuinely better, not luckier.

Tool 3: the confidence interval

"76 percent" is my best guess from these particular 178 questions. A different 178 would wobble a little. A confidence interval is the honest range around the guess: I'm 95 percent sure the true number sits between these two values. The standard recipe for a percentage is the Wilson interval. It looks busy, but every piece is just "the measured rate, nudged toward the middle and widened a bit because the sample is finite":

The rate is the measured percentage, n the number of questions, and z a fixed constant (1.96) that pins it to 95 percent sure. Run it on our results: the smart librarian lands in 69.7 to 82.0 percent, the lazy one in 35.1 to 49.5 percent. Read those two ranges again. They don't even touch. There is no version of this where the two are close. And all of it recomputes from the published data with no API key, in a second, on anyone's laptop.

You might be thinking: forget selecting, just give the student every page. We measured that too. Hand the student the entire history and it scores 83 percent. Our smart librarian's two pages reach 76. So the selected block keeps about 92 percent of the full library's accuracy, on roughly a third of the pages. The point was never that selecting beats reading everything. It's that you almost never can read everything (there's too much) or want to (it's slow and expensive), and the right two pages get you almost all the way there for a fraction of the cost.

Here is the reason I lead with the gap and not with a hero number. I changed the student. I swapped Gemini Flash for GPT-4o, a stronger and more careful model, and re-ran the exact same test. The smart librarian's score dropped from 76 to 63 percent.

Did the memory get worse? No. The pages were identical. What changed is the student. GPT-4o, under the instruction "say I don't know if the answer isn't clearly there", is more cautious. It declines on 32 percent of answerable questions, where Flash declines on only 10 percent. A more cautious student raises its hand less, so it scores lower, on the same pages. But watch what happens to the gap:

The absolute number moved by 13 points. The gap barely moved, and it stayed overwhelmingly real under both students. The absolute is a property of the student you happened to use. The gap is a property of the memory. This is exactly why the 90 something numbers floating around the field aren't comparable. A number with no student, no judge, and no version of the test attached can't be checked, so it can't really be believed.

You don't need anyone to name names. You just need three questions to audit any number:

1. Which student read the pages? A stronger reader inflates every score.
2. Which judge graded it? A lenient grader, or one grading its own answers, quietly lifts the number.
3. Which version of the test? Handing the student the right session up front is a much easier exam than burying it in hundreds of irrelevant ones. And some headline numbers aren't "did it answer", they're "did it retrieve", which is not the same exam at all.

Ask those three and most leaderboard numbers turn into "measured under conditions nobody wrote down". Ours are written down, and you can run them.

Finding the right pages is one job. Knowing what's related to what is another. Our librarian keeps an index of which pages mention which people, tools, and topics. So when you ask "what else touches this?", she follows the index, instead of squinting at titles and searching again.

And the thing that makes her smart isn't a bigger pile of paper. It's that she keeps a notebook of facts, and every fact knows when it became true and when she learned it. When a new fact contradicts an old one (you used to charge 9.99, now you charge 19), she crosses out the old one instead of keeping both and confusing you. It runs on plain Postgres, it can be hosted in the EU, and the notebook is yours, on your machine. It's the difference between a drawer stuffed with every receipt you ever got, and a notebook someone actually kept up to date.

Everything above lives in one open repository. Most of it needs no API key at all: the significance test and the page counts recompute straight from the saved, per-question results. If you want to see the student effect with your own eyes, swap the reader to GPT-4o and watch the absolute fall to 63 while the gap holds. I'd much rather you find that number in our repo, with my explanation right next to it, than discover it yourself and think we hid it.

github.com/verdana86/korely-graphrag

The selected pages that score 76 are produced by Korely's hosted memory, the same memory the API and SDK give your agents. In agent memory, the honest unit of progress isn't a big accuracy number. It's the gap at an equal budget, measured with a stated student, a stated judge, and a stated test, and reproducible by a stranger. Same number of pages, the right ones instead of the recent ones, nearly twice the right answers. That's the claim. You can run it yourself.