Better Agent Memory Starts Before Retrieval

A useful agent remembers across conversations, not just within them. What a user explained last week should still shape the answer it gives today, and that continuity is worth more the longer the relationship runs.

It is also where the cost hides. The simplest way to carry the past forward is to keep resending it, feeding prior turns and prior sessions back to the model so nothing is forgotten. But that history only grows. The tokens spent on a single turn come to track how long the agent has known the user rather than how hard the current question is. Latency and inference cost climb with the length of the relationship, so the users who engage the most, the ones a product most wants to keep, become the most expensive and the slowest to serve.

The tokens also buy less than they appear to. Most of what an agent has seen has nothing to do with the next question, so each turn asks the model to find a shrinking fraction of signal in a growing volume of text. Past a certain length, more context stops improving the answer. You continue paying for every token even as the marginal value of that context falls.

The standard response is memory. Distill what matters from each conversation so it carries forward without carrying every token forward. Retrieval is often treated as the central optimization problem: given a large collection of memories, how do you find the right ones at inference time?

We started from a different hypothesis. Perhaps retrieval isn't the primary bottleneck. The bottleneck is what gets written into memory in the first place.

Retrieval can only search memories that exist. If important information is never extracted, no retrieval strategy can recover it later. Conversely, if memory captures the right information up front, retrieval becomes a much simpler problem.

The highest-leverage optimization happens at write time, when information is extracted into memory rather than searched for.

Not all knowledge should be stored the same way. Some of it is narrative. An account switched from monthly to annual billing after a new VP consolidated its vendors, and the value is in the reasoning, not the switch itself. Some of it is atomic. A renewal date of March 31, a contract worth $240,000, a hard requirement that customer data stay in the EU. Forcing both into a single representation always cost us something: rich summaries lost the precise figures, and bare facts lost the context that made them matter.

So we stopped choosing. Our memory system writes in two forms. Narrative memories preserve causal context, the reasoning behind a decision or a change. Atomic memories preserve exact facts, the kind that has to survive verbatim. Retrieval draws on whichever the question needs.

Memory also has to change without losing its past. Deleting or overwriting an old value discards the context that often explains a later decision, so we default to appending. When something genuinely evolves, a renewal date that slips, the primary contact on an account, a security requirement that tightens, we add the new state and mark what it superseded, so retrieval can favor what is true now while the earlier version stays on record.

We expected to need more structure than that, knowledge graphs included, and found we didn't. With the two representations in place, improving what we wrote mattered more than improving how we searched it.

We evaluated on LongMemEval, LoCoMo, and BEAM, all multi-session benchmarks, against two baselines.

The first is full-history inference: put the whole conversation in the prompt and let the model find what matters. It is the brute-force alternative to memory, and it only runs while the history fits, so we use it as a reference on LongMemEval but not on long conversations like BEAM.

The second is our own evaluation of Mem0 OSS, run under the same per-benchmark answering model and evaluation harness as our system. Extraction and judging use the same model throughput, with judge prompts matching Mem0’s. Holding these fixed makes the comparison more attributable to the memory system itself.

Even where it fits, full history is not the ceiling you might expect. On LongMemEval it scored 60.6% against our 90.6% with the same model, while we processed a fraction of the context. Having every fact in the prompt is not the same as using it, and a focused memory beat the whole conversation.

Across benchmarks, the answering model processed far less: 97% fewer tokens on long conversations, 57% fewer on standard ones, with mean latency down 45% and p95 down 67% on long conversations.

Memory breaks the link between history length and answer cost

Against our Mem0 OSS baseline, the gains were in accuracy. On LongMemEval we scored 90.6 against 71.6, on LoCoMo 88.2 against 82.2, and on BEAM 61.3 against 39.7.

The largest gains appeared in knowledge updates and preference retention, categories that depend heavily on what the system chooses to preserve and how it represents the current state. Knowledge-update accuracy improved from 75.6% to 94.9%, and preference retention from 76.7% to 96.7%.

These results pointed us toward a problem upstream of retrieval. The information had been shared, but it still needed to be represented precisely enough to remain useful sessions later.

Long context is a brute-force way to preserve experience. Adaptive memory offers a different trade-off: spend a small amount once to shape what was learned, then reuse it without repeatedly processing the full history.

Full-history performance does not require full-history inference. Our results suggest that matching full-history performance does not require full-history inference. The largest gains came not from retrieving more, but from writing better representations of what mattered.

An agent should become more useful the longer it knows you, not slower and more expensive.