Long context vs fact-based memory: the design fork for persistent AI agents

When you build a persistent AI agent, the first fork is whether to stuff the whole conversation history into context or extract only the facts into a memory store. The March 2026 paper Beyond the Context Window settles this by numbers rather than intuition. Its core result is a break-even point: at 100k tokens, the memory approach becomes cheaper after about 10 turns. ASAP summarizes the paper from the original.

Why this paper matters now

As context windows stretched to hundreds of thousands of tokens, the industry's default answer became "just put everything in." No matter how long the conversation, you could re-inject the full history every turn, so a separate memory design seemed unnecessary. But that premise lumps accuracy and cost into a single lump. This paper's contribution is to separate the two axes and compare long-context GPT-5-mini against the Mem0 memory system under matched conditions. It reframes the question from "which is better" to "when is which better."

Accuracy and cost trace different curves

On accuracy, long context leads. Long-context GPT-5-mini leads on factual recall, scoring higher on the LongMemEval and LoCoMo benchmarks. The Mem0 memory system, by contrast, is competitive on PersonaMemv2, where persona consistency rests on stable factual attributes. In other words, long context holds the raw text and does well on fine-grained recall, while memory retains extracted core facts and does well at holding a persona.

The cost curve runs the other way entirely. Long-context inference charges more per turn as the context grows, even with caching enabled: a rising profile. The memory system holds a roughly fixed per-turn read cost after the initial write phase: a flat profile. The key lens for reading this paper is that the accuracy curve and the cost curve move in opposite directions.

How to read the break-even number

The break-even is where the two curves cross. At 100k tokens, the memory system becomes cheaper after roughly 10 interaction turns, and the break-even point arrives sooner as the context grows longer. The caution here is that "10 turns" is not a constant but a value tied to the 100k-token condition. At 200k the crossing arrives earlier, and for a very short conversation the break-even may never arrive at all. Read the number as a function, not a rule.

Implications for practitioners

Many production services are sensitive to per-token pricing and run workloads that stay with the same user for a long time, such as support bots or repetitive task agents. Plugging in this paper's criteria, the longer and more repetitive the session, the more fact-based memory pays off. Conversely, for a short QA-style feature where precise fine-grained recall is the whole game, long context is still reasonable. In practice, rather than picking one, it is worth considering a hybrid: use long context for the first few turns to secure accuracy, then switch to memory once the conversation crosses the threshold turn.

Limitations and open questions (ASAP's view)

These results come from a specific model (GPT-5-mini), a specific memory implementation (Mem0), and the 100k-token condition. Change the model or the memory system and the break-even location can shift. Benchmark scores also do not always track real service quality, and controlling which facts get discarded during memory extraction needs separate attention. Even so, the framing, separating accuracy from cost to set a quantitative rule for when to use which, should stay relevant in agent-design discussions for a long time.

Source: ASAP summary of "Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents" (arXiv:2603.04814, March 5, 2026; Natchanon Pollertlam, Witchayut Kornsuwannawit).