1
00:00:01,000 --> 00:00:08,244
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:08,244 --> 00:00:42,424
[Hal Turing] Today's paper is "Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads." The first author is Yasmine Omri, with Ziyu Gan as equal-contribution co-lead. They're at Stanford, along with colleagues from KU Leuven's MICAS group and MIT. It's arXiv 2606.06448, version two, September 2026. The question: a long-lived agent has to remember across sessions, so who pays for the remembering, and when?

3
00:00:42,424 --> 00:01:01,743
[Dr. Ada Shannon] The answer is that the bill mostly arrives at write time, and almost nobody was measuring it. Memory systems have been compared on accuracy. This paper profiles ten of them as systems workloads and splits the cost into construction, retrieval and generation. Where the cost lands changes how you schedule and choose them.

4
00:01:01,743 --> 00:01:08,337
[Hal Turing] Start with the obvious objection. Context windows are enormous now. Why not paste the whole history into the prompt?

5
00:01:08,337 --> 00:01:38,059
[Dr. Ada Shannon] Multi-session history eventually exceeds any budget. Prefill cost grows quadratically with history length. Prefix caching mostly fails across sessions, because the cached state gets evicted or the prompt changes. And recall degrades for material in the middle of long inputs. That's the Lost in the Middle paper by Nelson Liu at Stanford, 2023, and the RULER benchmark from NVIDIA in 2024 showed effective context is often shorter than the advertised window.

6
00:01:38,059 --> 00:01:42,610
[Hal Turing] So you need external memory. Isn't that just retrieval-augmented generation, RAG?

7
00:01:42,610 --> 00:02:19,019
[Dr. Ada Shannon] RAG attaches a model to a mostly static corpus, such as documentation. Agent memory is a corpus the agent writes itself from its own interaction stream, often one per user. Take an assistant that learns you prefer morning meetings and are vegetarian. That has to be written, retrieved weeks later, and possibly updated when you change your mind. So there's a write path, a read path and a maintenance policy. The paper frames it as a seven-stage pipeline: ingestion, construction, storage, retrieval, prompt assembly, generation and maintenance.

8
00:02:19,019 --> 00:02:21,852
[Hal Turing] Construction is the write step. What actually happens in it?

9
00:02:21,852 --> 00:02:48,183
[Dr. Ada Shannon] It turns raw history into stored records, and it comes in four forms. Absent means long context, with nothing built. Deterministic means chunk the text and index it, with no LLM involved. LLM-mediated means a model extracts facts, and maybe decides whether to add, update or delete against existing memories. Agentic means the model itself decides when to write.

10
00:02:48,183 --> 00:02:49,576
[Hal Turing] Give me one chat turn.

11
00:02:49,576 --> 00:03:11,124
[Dr. Ada Shannon] The user says, "I moved to Lisbon last month." Deterministic construction stores that sentence in a chunk. LLM-mediated construction calls a model to produce a fact like "user lives in Lisbon," then checks whether an older "lives in Berlin" record should be replaced. The second costs extra model calls on every turn, and that's the cost the paper is chasing.

12
00:03:11,124 --> 00:03:12,425
[Hal Turing] And the ten systems?

13
00:03:12,425 --> 00:03:53,571
[Dr. Ada Shannon] Four paradigms. One is long-context passthrough, the baseline. Two is flat RAG: BM25, a classic lexical index, and embedRAG, a dense vector index. Three-a is append-only structure: GraphRAG and HippoRAG v2 extract entities and relations but never rewrite. Three-b is consolidating: Mem0 and SimpleMem merge and update records. Four is agentic control flow: A-Mem, Letta, which descends from the MemGPT paper by Charles Packer at Berkeley in 2023, and MIRIX. I won't narrate the table. Just hold the gradient: more LLM involvement at write time as you go down the list.

14
00:03:53,571 --> 00:03:56,775
[Hal Turing] Why is that a systems question rather than just a leaderboard question?

15
00:03:56,775 --> 00:04:13,772
[Dr. Ada Shannon] Because an accuracy score can't tell you whether cost moved from the read path to the write path. The paper asks three things. Where does the cost go? What do the shapes of those costs imply for scheduling on shared hardware? And how do they behave as histories grow and fleets scale?

16
00:04:13,772 --> 00:04:17,534
[Hal Turing] I need two terms pinned down before numbers show up. Prefill and decode?

17
00:04:17,534 --> 00:04:45,165
[Dr. Ada Shannon] Prefill reads the whole input prompt in one parallel pass, and it's compute- and bandwidth-heavy. Decode generates output one token at a time. Time to first token, TTFT, is the wait before an answer starts, which is mostly prefill. The paper argues construction is long-read, short-write: big prompts in, small structured outputs out. That has implications for GPU utilization, which we'll get to.

18
00:04:45,165 --> 00:04:46,419
[Hal Turing] And the main benchmark?

19
00:04:46,419 --> 00:05:16,791
[Dr. Ada Shannon] LongMemEval_S_*, a conversational-memory question-answering task inside MemoryAgentBench. There are five samples, each about 360,000 tokens of history, fed in 4,096-token chunks, with 60 queries each, so 300 questions. An LLM judge scores correctness, and higher is better. MemoryAgentBench as a whole tests four competencies: accurate retrieval, test-time learning, long-range understanding and selective forgetting.

20
00:05:16,791 --> 00:05:17,952
[Hal Turing] Where's this heading?

21
00:05:17,952 --> 00:05:35,738
[Dr. Ada Shannon] Three beats. Construction is the hidden bill. The cheapest design turns out to be a surprisingly strong baseline. And we should test whether the harness comparing these systems is fair, because the authors had to adapt several of them to fit one setup. I'd hold off on numbers until we've covered how they measured.

22
00:05:35,738 --> 00:05:41,776
[Hal Turing] Then let's cover the how. How do they measure cost per phase on a GPU that's shared by everything?

23
00:05:41,776 --> 00:06:10,661
[Dr. Ada Shannon] They tag every chat call and every embedding call with its phase: construction, retrieval or generation. Then they line those tags up on one timeline with GPU telemetry from NVML and DCGM, and integrate the energy per phase. There are two regimes. A remote one uses OpenAI models. A local one uses vLLM on a single H100 with a Qwen3 ladder from 1.7 billion to 32 billion parameters, thinking disabled.

24
00:06:10,661 --> 00:06:12,844
[Hal Turing] And to make ten different systems comparable?

25
00:06:12,844 --> 00:06:37,178
[Dr. Ada Shannon] They standardize the generation model, the embedder, 4096-token chunks, and top-10 retrieval, cut to top-5 on overflow. Then they adapt individual systems. Mem0 and SimpleMem get changed extraction prompts, A-Mem's notes are aggregated, and Letta gets 512-token chunks and a cap on tool calls. I'm stating those as method facts now, because they matter when we judge the results later.

26
00:06:37,178 --> 00:06:38,618
[Hal Turing] Fine. Give me the headline.

27
00:06:38,618 --> 00:07:04,206
[Dr. Ada Shannon] The cheapest design wins or ties on accuracy. On the main benchmark, BM25 gets 47 percent of questions right. Letta gets 27.7. To get one correct answer, BM25 spends about four kilojoules and Letta about 186. That's roughly 45 times the energy for a lower score. Letta also needs about fourteen hours to build memory that BM25 builds in minutes.

28
00:07:04,206 --> 00:07:05,925
[Hal Turing] Is that just this one benchmark?

29
00:07:05,925 --> 00:07:25,662
[Dr. Ada Shannon] Partly. Averaged across all the MemoryAgentBench datasets, BM25 is still first at 55.8 percent. But those are recall-heavy tasks, which favor lexical matching. On harder categories the gap narrows or reverses, and the authors say BM25 shouldn't be read as a universal winner.

30
00:07:25,662 --> 00:07:29,888
[Hal Turing] So where does all that energy go in the elaborate systems? Answering questions?

31
00:07:29,888 --> 00:07:50,182
[Dr. Ada Shannon] No, construction. Take Mem0. It answers a query in under a tenth of a second, where stuffing the full history into the prompt takes about 38. But on one sample, Mem0's build costs about nine times the energy of BM25's entire run, build plus sixty questions. The fast answer was paid for earlier.

32
00:07:50,182 --> 00:07:53,061
[Hal Turing] Does it matter when it's paid, if the joules are the same?

33
00:07:53,061 --> 00:08:02,117
[Dr. Ada Shannon] Yes. Energy is invariant, but latency is schedulable. You can push construction into the background so users never feel it. You can't make the energy disappear.

34
00:08:02,117 --> 00:08:07,225
[Hal Turing] Why is construction so wasteful on a GPU, though? Reading a lot should keep it busy.

35
00:08:07,225 --> 00:08:31,792
[Dr. Ada Shannon] It reads a lot and writes little. The median decode share of construction tokens is 4.6 percent. So it's mostly prefill and embedding. The waste comes from the loop. Mem0 makes small serialized calls, one embedding per call, and single-tenant it uses about 12 percent tensor-core activity. The GPU waits between calls. GraphRAG and HippoRAG v2 batch hundreds of sequences per embedding call, so they look different.

36
00:08:31,792 --> 00:08:33,928
[Hal Turing] So share the GPU across users.

37
00:08:33,928 --> 00:09:02,582
[Dr. Ada Shannon] They did. Ten independent Mem0 builds on one backend reach 9.6 times the single-build throughput. HippoRAG v2 gets only 1.4 times, since it's already batched. They also checked a Qwen3-235B model across eight H100s, where Mem0's build took 13.4 times longer than HippoRAG v2's. But co-tenancy covers only those two systems, and interference with live question traffic is argued from mechanism, not measured.

38
00:09:02,582 --> 00:09:06,390
[Hal Turing] What if the construction model is small? Cheaper extraction, same memory?

39
00:09:06,390 --> 00:09:28,774
[Dr. Ada Shannon] Systems differ in how much they need. GraphRAG stays around 47 to 48 percent from the 1.7B model up to GPT-4o-mini. MIRIX fails entirely at 1.7B, because the tool-call and JSON formats break and the store gets corrupted. That's a capability floor. Operators need to validate it before deployment.

40
00:09:28,774 --> 00:09:31,560
[Hal Turing] And if sessions arrive faster than the write finishes?

41
00:09:31,560 --> 00:10:10,477
[Dr. Ada Shannon] That's the MemoryArena test, where later sessions depend on earlier ones. Staleness is how many earlier sessions haven't been written when a query arrives. With a five-second gap between sessions, BM25 and embedRAG stay fresh. Mem0, MIRIX, Letta, SimpleMem and A-Mem fall behind by one or two sessions in async mode. Per-session construction runs from about a millisecond to about ten seconds, with tails over a hundred. You either block the user or answer from stale memory. One limit: they count stale sessions, not the accuracy lost, and timing was replayed from traces.

42
00:10:10,477 --> 00:10:13,821
[Hal Turing] What about growth? Histories don't stay at 360K tokens.

43
00:10:13,821 --> 00:10:38,898
[Dr. Ada Shannon] Storage at one million tokens varies about ninefold across systems. The bigger problem is that Letta, A-Mem and MIRIX scale super-linearly in token cost, and Letta diverges beyond 256K. Retrieval latency stays nearly flat. No evaluated system forgets or prunes by default. Their 100,000-user storage figures are a projection from a single-user curve, not a measured fleet.

44
00:10:38,898 --> 00:10:39,966
[Hal Turing] Tail behavior?

45
00:10:39,966 --> 00:11:01,607
[Dr. Ada Shannon] Algorithm-bounded phases stop where the design says, like one BM25 lookup. LLM-bounded phases run until the model decides it's done, so only an iteration cap bounds the worst case. The p95 to median ratio is about 1.3 for BM25 and embedRAG, 3.9 for Letta, and 5.9 for GraphRAG.

46
00:11:01,607 --> 00:11:05,647
[Hal Turing] So does anything win on build cost, serve latency and accuracy together?

47
00:11:05,647 --> 00:11:21,437
[Dr. Ada Shannon] No. Mem0 has the lowest per-query latency, about 2.2 seconds in that setup, but 26.8 percent macro accuracy. SimpleMem is slowest at query time, about 18 seconds. It's a frontier, not a leaderboard.

48
00:11:21,437 --> 00:11:22,691
[Hal Turing] The recommendations?

49
00:11:22,691 --> 00:11:54,688
[Dr. Ada Shannon] They cluster. For selection, don't choose on accuracy alone, and match the system to query volume. For scheduling, treat construction as a rate-limited background job with admission control, and keep it off latency-sensitive endpoints, especially shared embedding servers, where head-of-line blocking is the hazard. Then validate the construction model's floor, cap iterations on LLM-bounded systems, and add forgetting or compaction yourself. For freshness, check that construction plus retrieval fits inside the gap between sessions.

50
00:11:54,688 --> 00:12:02,211
[Hal Turing] BM25 beat the elaborate systems. My suspicion is the harness. Can the data separate bad design from bad tuning?

51
00:12:02,211 --> 00:12:18,047
[Dr. Ada Shannon] No. Beyond the prompt changes, note aggregation and Letta's small chunks and tool-call cap, everyone was forced to top-10 retrieval. There's no sensitivity or tuning study, so we can't say how much of the gap is design and how much is harness.

52
00:12:18,047 --> 00:12:20,137
[Hal Turing] Which direction do those changes push?

53
00:12:20,137 --> 00:12:34,487
[Dr. Ada Shannon] Mostly against the agentic systems. Small chunks and tight tool-call caps likely hurt Letta in particular. A top-10 cap may hurt designs that rely on bigger retrievals or multi-step search. That's reasoning about direction, not a measurement.

54
00:12:34,487 --> 00:12:36,577
[Hal Turing] Does the same doubt apply to the energy gap?

55
00:12:36,577 --> 00:13:04,627
[Dr. Ada Shannon] Less so. Energy follows call counts, and Letta made 18,394 calls where BM25 made 300. The caps already limit those calls, so the energy gap probably survives tuning better than the accuracy gap does. But the 4.1 versus 186 kilojoule ratio has accuracy in the denominator, so it inherits the accuracy doubt. What would settle it is per-system tuning sweeps, unconstrained retrieval budgets, and variance across seeds.

56
00:13:04,627 --> 00:13:05,834
[Hal Turing] And the statistics?

57
00:13:05,834 --> 00:13:32,351
[Dr. Ada Shannon] Five samples, 300 queries, no confidence intervals. GraphRAG at 46.0 against BM25 at 47.0 is not a reliable win for either. The big gaps, BM25 near 47 against Letta under 28, are probably real, though no uncertainty is reported. LongMemEval_S_* also favors lexical recall, and the authors themselves say BM25 isn't a universal winner.

58
00:13:32,351 --> 00:13:37,692
[Hal Turing] Then the amortization claim. Construction dominates, but only for sixty queries per memory.

59
00:13:37,692 --> 00:13:56,918
[Dr. Ada Shannon] Right, and the paper never sweeps that number. The break-even is construction energy divided by the per-query energy saved. As an anchor, Mem0's one-sample build was about 1,050 kilojoules. Spread over sixty queries, that's roughly 17.5 kilojoules each. That's our arithmetic, not a paper figure.

60
00:13:56,918 --> 00:13:59,379
[Hal Turing] So at a thousand queries per user it could flip?

61
00:13:59,379 --> 00:14:15,865
[Dr. Ada Shannon] Against long-context prompting, possibly. But that energy isn't in Table 3. Against BM25 there's no break-even on this benchmark, because the consolidating systems also score lower. Recommendation 6 gives the direction but no threshold.

62
00:14:15,865 --> 00:14:17,398
[Hal Turing] What does the nearby work say?

63
00:14:17,398 --> 00:14:44,194
[Dr. Ada Shannon] Sleep-time Compute, by Kevin Lin and colleagues, 2025, is the closest relative. It defers computation to idle time and amortizes it across queries on the same context, which is the scheduling idea behind Recommendation 3. Zep, from Rasmussen and colleagues in 2025, is cited but not evaluated. Its bi-temporal invalidation, tracking when a fact was true and when it was recorded, would test the staleness model properly.

64
00:14:44,194 --> 00:14:45,355
[Hal Turing] And the KV side?

65
00:14:45,355 --> 00:15:12,708
[Dr. Ada Shannon] That's the blind spot. LMCache, by Yihua Cheng and colleagues, 2025, offloads and shares KV cache across GPU, CPU and disk. The paper says prefix caching fails across sessions, and that is exactly what LMCache targets. Persisted KV is a rival to distilled text memory, and nothing here compares the two. Restore latency from SSD against repeated extraction prefill is simply unmeasured.

66
00:15:12,708 --> 00:15:14,473
[Hal Turing] Where is the paper actually solid?

67
00:15:14,473 --> 00:15:40,293
[Dr. Ada Shannon] The single-node mechanism: construction is long-read, short-write, serialized loops leave the GPU idle, the construction-model floor exists, token cost grows with history, and nothing forgets by default. Those hold up well. The extrapolated parts are the 100,000-user storage figure from one user's curve, freshness from replayed traces at a fixed five-second gap, and co-tenancy on two systems.

68
00:15:40,293 --> 00:15:45,634
[Hal Turing] The co-tenancy result measures build throughput, though. It doesn't measure interference with question answering.

69
00:15:45,634 --> 00:15:58,823
[Dr. Ada Shannon] Exactly. Recommendation 3 rests on construction prefill competing with latency-sensitive traffic, and no QA tail latency under mixed load is shown. The mechanism is plausible, but the size of the effect is unknown.

70
00:15:58,823 --> 00:16:03,327
[Hal Turing] Likewise the MIRIX cliff. Is that a memory floor or a JSON failure?

71
00:16:03,327 --> 00:16:16,934
[Dr. Ada Shannon] Closer to a JSON and tool-call failure at Qwen3-1.7B. Whether constrained decoding would remove it is untested, and it's one model family. So I'd call it a deployment-validation warning, not proof the algorithm needs a big model.

72
00:16:16,934 --> 00:16:18,838
[Hal Turing] What should an operator do differently?

73
00:16:18,838 --> 00:16:41,919
[Dr. Ada Shannon] Run construction as a rate-limited background job with admission control, kept off shared embedding paths. Validate the smallest construction model your system tolerates. Cap iterations for LLM-bounded systems. Add your own forgetting and compaction, since none comes built in, and deletion requests will make that matter. And pick by query arrival pattern, not accuracy alone.

74
00:16:41,919 --> 00:16:43,173
[Hal Turing] And the open experiments?

75
00:16:43,173 --> 00:16:59,287
[Dr. Ada Shannon] The paper names multi-node and multi-agent consistency, and multimodal memory. I'd add a queries-per-memory sweep, per-system tuning, variance across seeds, a bi-temporal comparison, and persisted-KV baselines against text memory.

76
00:16:59,287 --> 00:17:16,238
[Hal Turing] So, the takeaway. Remembering has a write bill, and it arrives before anyone asks a question. That bill is unavoidable in energy, though it can be hidden in latency by running it in the background. Accuracy alone is a poor way to choose a memory system.

77
00:17:16,238 --> 00:17:31,006
[Dr. Ada Shannon] Agreed. The boundary is one benchmark focus, five samples, and harness adaptations nobody tested. It doesn't show that expensive memory designs are worse in general. It shows that their costs deserve measuring before you adopt them.

78
00:17:31,006 --> 00:17:32,817
[Hal Turing] That's our time. Thanks for listening.