1
00:00:01,000 --> 00:00:08,058
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:08,058 --> 00:00:35,411
[Hal Turing] Today's paper is Multi-Agent Transactive Memory, by To Eun Kim and colleagues at Carnegie Mellon University, with a co-author at UC Berkeley. It's on arXiv, June 2026. The question is simple to state. If many different agents drop their task logs into one shared searchable pile, can other agents pull from it and succeed more often, in fewer steps, with no coordination and no joint training?

3
00:00:35,411 --> 00:00:48,090
[Dr. Ada Shannon] The pitch is that every new agent today re-discovers solutions that already exist somewhere. Those logs get thrown away, or the producing agent keeps them private. Whether sharing actually fixes that is what we need to test.

4
00:00:48,090 --> 00:00:53,430
[Hal Turing] Let's build up to it. Start with retrieval-augmented generation, since everything here sits on top of that.

5
00:00:53,430 --> 00:01:13,957
[Dr. Ada Shannon] RAG, from Lewis and colleagues at Facebook AI Research in 2020. Before the model answers, a retriever fetches relevant text from an outside collection and puts it in the prompt. The collection is usually human-written: Wikipedia, docs, papers. The knowledge lives outside the weights, so you can update or delete it without retraining.

6
00:01:13,957 --> 00:01:15,396
[Hal Turing] And the agent version of that?

7
00:01:15,396 --> 00:01:43,957
[Dr. Ada Shannon] Single-agent experience memory. Reflexion, by Shinn and colleagues in 2023, stores verbal self-critiques after failed attempts. Synapse, by Zheng and colleagues in 2024, retrieves stored successful trajectories as exemplars for computer-control tasks. ExpeL, by Zhao and colleagues at Tsinghua, AAAI 2024, collects trajectories, extracts insights, and retrieves successful ones at inference time. All of them keep memory inside one agent's loop.

8
00:01:43,957 --> 00:01:48,043
[Hal Turing] You keep saying trajectory. People hear that and think of a neat summary.

9
00:01:48,043 --> 00:02:04,948
[Dr. Ada Shannon] It's not neat. It's the raw log of one task: each step pairs an action with the observation it produced. A click and the resulting page HTML. Or 'open fridge' and the text reply. It's a record of what happened, not a document written for a reader.

10
00:02:04,948 --> 00:02:07,455
[Hal Turing] So the new step in this paper is sharing the pool.

11
00:02:07,455 --> 00:02:32,440
[Dr. Ada Shannon] Right. A shared repository of agent-generated trajectories serving a whole population. Producers contribute trajectories. Consumers retrieve from the repository during their own task. One agent can be both. And ExpeL matters beyond being prior work. It's the control this paper's own claim needs: private memory versus shared memory. The paper cites it but doesn't run it. Hold that thought.

12
00:02:32,440 --> 00:02:33,694
[Hal Turing] Where does the name come from?

13
00:02:33,694 --> 00:03:00,629
[Dr. Ada Shannon] Daniel Wegner's transactive memory, 1987, from group psychology. A couple or a team remembers more than any member, because they keep a shared way of knowing who knows what, and how to look it up. The paper borrows the shared locate-and-retrieve mechanism. The boundary matters, though. Wegner's version is human groups with social awareness of each other's expertise. Here it's a shared index. The agents aren't modeling each other.

14
00:03:00,629 --> 00:03:02,765
[Hal Turing] So closer to a search engine than a team.

15
00:03:02,765 --> 00:03:15,165
[Dr. Ada Shannon] Closer to Stack Overflow for agents. Producers are the people who answered, consumers are the ones searching. The analogy's limit: Stack Overflow answers were written to be read. These trajectories weren't.

16
00:03:15,165 --> 00:03:19,205
[Hal Turing] Why not just fine-tune on the logs, or distill, or run a central orchestrator?

17
00:03:19,205 --> 00:03:49,670
[Dr. Ada Shannon] Those each need something. Fine-tuning needs weights and a training loop. Distillation and multi-agent training need joint setups. Orchestration needs agents that know about each other. Here the consumer is a frozen model reading examples in context, whatever model it is. Nothing is shared except the repository, and you can delete a bad trajectory, which you can't do with weights. The costs are context tokens, and a dependence on retrieval quality and on the model's in-context learning.

18
00:03:49,670 --> 00:03:51,667
[Hal Turing] Now the testbeds. ALFWorld first.

19
00:03:51,667 --> 00:04:14,004
[Dr. Ada Shannon] A text-only household simulator. The agent reads descriptions and issues commands from a small action set, like 'open fridge' or 'heat mug', toward a goal such as 'put a cleaned apple in the fridge'. Success is binary per episode, and you report the fraction completed. Higher is better. There are six task types and 120 rooms. Worth noticing: that's a small, fixed world.

20
00:04:14,004 --> 00:04:15,026
[Hal Turing] And WebArena?

21
00:04:15,026 --> 00:04:38,849
[Dr. Ada Shannon] Self-hosted realistic websites, e-commerce, forums, and others, where the agent browses to do things like find an order or post a comment. Tasks are long and hard, and success for this model population is around 20%. But the tasks are templated over those few sites, so the variety is narrower than it feels. Remember both of those facts. They set up a question we'll come back to.

22
00:04:38,849 --> 00:04:41,125
[Hal Turing] Give me the paper's claim in one breath, then.

23
00:04:41,125 --> 00:04:55,661
[Dr. Ada Shannon] Retrieval over the shared pool helps. A learned reranker helps more. The benefit spreads across the population, and it grows with the size of the memory. Those are claims to test, not conclusions. Next, how they actually build the index.

24
00:04:55,661 --> 00:05:01,512
[Hal Turing] So there's a big pile of trajectories. What does an agent actually do mid-task to find the useful one?

25
00:05:01,512 --> 00:05:18,416
[Dr. Ada Shannon] The retrieval key is the task description plus the last five steps the agent just took, embedded with E5-Base. The stored value is the next five steps from some past trajectory. So the agent isn't asking which old task looks like mine. It's asking what someone did next from a situation like mine.

26
00:05:18,416 --> 00:05:22,317
[Hal Turing] That's a different query from ordinary RAG. How does the pile get built?

27
00:05:22,317 --> 00:05:51,156
[Dr. Ada Shannon] Two phases. First, pre-population with public trajectories. For ALFWorld that's the benchmark authors' seq2seq model plus Qwen3-32B and GPT-OSS 20B. For WebArena it's the official runs of GPT-4-Turbo, GPT-4-Turbo-Preview and Claude 3.5 Sonnet, plus those same two open models. Second, incremental growth. Agents work through training questions, and only successful trajectories above a quality threshold get added.

28
00:05:51,156 --> 00:05:52,364
[Hal Turing] And how big does it get?

29
00:05:52,364 --> 00:06:21,389
[Dr. Ada Shannon] 86,833 chunks for ALFWorld, 20,102 for WebArena. In ALFWorld, 85,615 of those chunks are pre-population, so the index is almost entirely the seed corpus. The split is 3,553 training and 274 test episodes for ALFWorld, and the custom WebArena split is 724 and 88. Test tasks are never added to memory, but the authors say outright that the test set may overlap with training in task type and layout.

30
00:06:21,389 --> 00:06:22,736
[Hal Turing] At inference time, then?

31
00:06:22,736 --> 00:06:40,151
[Dr. Ada Shannon] Dense retrieval returns the top 20 chunks, a reranker picks one, and the agent sees that single five-step chunk. A separate LLM, the retrieval planner, decides at each step whether to call memory at all. The authors added it so constant retrieval doesn't clutter the context.

32
00:06:40,151 --> 00:06:44,330
[Hal Turing] Training the reranker is where I'd expect the trouble. How does it learn what's helpful?

33
00:06:44,330 --> 00:07:11,405
[Dr. Ada Shannon] By measuring it directly. The agent first runs a training task with no retrieval, which gives a baseline score. Then they pick a random midpoint of that run, retrieve, and replay from there with chunks from ranks 1, 5, 10, 15 and 20 injected. A chunk's label is the replay score minus the baseline score. So the label is real task gain, not text similarity. They use two branch points per task for ALFWorld and one for WebArena.

34
00:07:11,405 --> 00:07:12,705
[Hal Turing] What does the reranker see?

35
00:07:12,705 --> 00:07:42,937
[Dr. Ada Shannon] 44 features in six groups: producer metadata, consumer metadata, retrieval scores, query features, trajectory features, and query-trajectory similarity. The producer metadata includes that producer's benchmark scores. The authors call this trust modelling. The top ALFWorld feature is one producer's SciCode score, with a mix of positive and negative producer weights. They train three families: a feed-forward net, LambdaMART, and SVMRank.

36
00:07:42,937 --> 00:07:45,956
[Hal Turing] Before the numbers, I need the metric that isn't success rate.

37
00:07:45,956 --> 00:08:11,962
[Dr. Ada Shannon] Return-paired preference, from Fernando Diaz's 2026 work. It asks how often a method's trajectory beats the baseline's on success and efficiency together. Above zero means it wins more often than it loses. The no-retrieval row is negative because it's scored against a fixed reference baseline. Everything is averaged across the consumer population, over five runs with randomized task-to-model allocation.

38
00:08:11,962 --> 00:08:13,123
[Hal Turing] So, ALFWorld.

39
00:08:13,123 --> 00:08:29,888
[Dr. Ada Shannon] Out of every hundred episodes, the no-retrieval population finishes about 47. Plain similarity search gets about 55. With the SVMRank reranker, about 64. Episodes also run about 1.4 steps shorter. This is the strongest result in the paper.

40
00:08:29,888 --> 00:08:30,910
[Hal Turing] And WebArena?

41
00:08:30,910 --> 00:08:55,384
[Dr. Ada Shannon] Much smaller. Of 88 test tasks, the baseline solves 16 and the best retrieval variants solve 18. That's two tasks. Average steps fall from about 22 to about 20, roughly 8% shorter. LambdaMART gives the success gain back entirely, and different rerankers win in different environments: SVMRank leads ALFWorld, while FFN gets the fewest steps on WebArena.

42
00:08:55,384 --> 00:08:57,891
[Hal Turing] Shorter could also mean the agent quit earlier.

43
00:08:57,891 --> 00:09:06,204
[Dr. Ada Shannon] Yes, and the paper doesn't split step counts by success versus failure. It also reports no variance or significance tests on any of this.

44
00:09:06,204 --> 00:09:10,569
[Hal Turing] What about the producer side? Does a strong producer have to be feeding a weak consumer?

45
00:09:10,569 --> 00:09:32,443
[Dr. Ada Shannon] The paper plots each producer-consumer pair, with capability gap on one axis and retrieval advantage on the other. Consumers benefit whether the producer is weaker, comparable or stronger. The correlation is small and not significant, r around 0.09, p around 0.09. So it's not a story of strong agents teaching weak ones.

46
00:09:32,443 --> 00:09:36,018
[Hal Turing] Though that's an absence of a pattern, not a demonstrated mechanism.

47
00:09:36,018 --> 00:09:40,244
[Dr. Ada Shannon] Right. It rules out one explanation but doesn't establish another.

48
00:09:40,244 --> 00:09:42,474
[Hal Turing] You mentioned WebArena scaling gets odd.

49
00:09:42,474 --> 00:09:55,755
[Dr. Ada Shannon] Success dips at a mid-sized index before recovering. The authors' hypothesis is that a mid-sized index surfaces plausible but unhelpful trajectories without yet having enough good ones. They don't test that, so it's a story fitted to the curve.

50
00:09:55,755 --> 00:10:00,028
[Hal Turing] Which raises the real question. How much of the WebArena claim can the evidence carry?

51
00:10:00,028 --> 00:10:08,898
[Dr. Ada Shannon] On ALFWorld, the gain from reranking is too big to be noise. On WebArena, the success gain is not distinguishable from noise on the evidence given.

52
00:10:08,898 --> 00:10:10,802
[Hal Turing] That's blunt. Show the arithmetic.

53
00:10:10,802 --> 00:10:37,226
[Dr. Ada Shannon] This is our estimate, not the paper's. The WebArena test set is 88 tasks, so one task is about 1.1 percentage points. At roughly 20% success, a binomial standard error over 88 tasks is around 4 points. The headline gain is two tasks, 16 to 18. The whole scaling curve spans about two tasks too. The paper reports no confidence intervals, no seed variance and no significance tests.

54
00:10:37,226 --> 00:10:42,567
[Hal Turing] But they average five runs across 34 consumers. Doesn't that wash out the noise?

55
00:10:42,567 --> 00:11:05,044
[Dr. Ada Shannon] It shrinks model and sampling randomness. The 88 tasks stay fixed, and that fixed set is where most of the uncertainty sits. A paired comparison on the same tasks would help, but none is reported. And look at the pattern. LambdaMART lands exactly back at baseline, and SVMRank matches the single-stage gain without beating it. That's what a flat effect looks like.

56
00:11:05,044 --> 00:11:07,505
[Hal Turing] So is there anything on WebArena beyond noise?

57
00:11:07,505 --> 00:11:25,059
[Dr. Ada Shannon] One candidate: step count. It falls under all four retrieval variants, and it declines steadily with index size. A continuous measure moving in a consistent direction is weak positive evidence. But shorter could mean faster success or earlier giving up, and there's no breakdown for successful episodes only.

58
00:11:25,059 --> 00:11:28,449
[Hal Turing] Then Table 2. Why does WebArena drop to 47 tasks?

59
00:11:28,449 --> 00:11:51,577
[Dr. Ada Shannon] The main text doesn't say. Appendix L does: tasks with no same-type candidate in the index were dropped, because WebArena splits into 241 fine-grained intents. So the 47 are the tasks that happen to have close neighbours, which is where lookup helps most. Full retrieval solves 14, cross-task solves 9. And there's no no-retrieval baseline for those 47.

60
00:11:51,577 --> 00:11:55,710
[Hal Turing] That feeds the leakage worry. If the test is a sibling of something in memory...

61
00:11:55,710 --> 00:12:20,462
[Dr. Ada Shannon] Then the experiment may be measuring lookup of seen templates. ALFWorld has six task types over a fixed set of rooms. WebArena is template-generated over a few sites, and the custom split is random within intents. The authors themselves say the test set may contain overlapping task types and configurations. Retrieving the path for 'put a cleaned apple in the fridge' when the index holds 'put a cleaned mug in the cabinet' is close to retrieving the answer.

62
00:12:20,462 --> 00:12:24,920
[Hal Turing] Though cross-task retrieval on ALFWorld still lands well above no retrieval.

63
00:12:24,920 --> 00:12:55,385
[Dr. Ada Shannon] It does, about 60% against 47%. That's a real result. But shared rooms, objects and the nine-action grammar could explain it without any transferable strategy. What would settle it is a held-out-room or held-out-template split, deduplication by layout, or results broken out by seen versus unseen scenario. I believe the official ALFWorld test set mixes in-distribution and out-of-distribution episodes, but that's background knowledge to verify. The paper doesn't separate them.

64
00:12:55,385 --> 00:12:58,264
[Hal Turing] And the sharing part? The title is about populations.

65
00:12:58,264 --> 00:13:44,333
[Dr. Ada Shannon] There's no private-memory comparison. ExpeL, which we mentioned earlier, is cited but not run, and it's the obvious control. Without it, the paper shows that retrieving a relevant demonstration helps, not that cross-agent sharing adds anything. Agent Workflow Memory, by Zora Wang and colleagues at Carnegie Mellon, 2025, induces reusable workflows from trajectories and evaluates on WebArena. That's the abstraction-versus-raw-chunks test the paper argues for but doesn't run. Agent KB, by Xiangru Tang and colleagues, 2025, is a shared cross-agent experience base. It's very close to this pitch and isn't in the paper's reference list. Memp, by Runnan Fang and colleagues, 2025, is related work on procedural memory.

66
00:13:44,333 --> 00:13:47,073
[Hal Turing] Also, almost all the ALFWorld index is seed data.

67
00:13:47,073 --> 00:14:20,602
[Dr. Ada Shannon] 85,615 of 86,833 chunks. So 'grows with population contribution' is mostly a story about a seed corpus. And the reranker's producer features are benchmark scores for about 35 producers, all seen in training. That can be trust modelling, or it can be memorizing which producers' trajectories worked. The feature importances flip sign across benchmarks, which fits the second reading. And the capability-gap null result is a null, not a demonstration that benefit is distributed across the population.

68
00:14:20,602 --> 00:14:23,203
[Hal Turing] What about cost? Efficiency is part of the pitch.

69
00:14:23,203 --> 00:14:41,500
[Dr. Ada Shannon] Steps are counted. The extra planner call per step, retrieval latency and added context tokens are not. So we can't say it's cheaper in tokens, dollars or wall-clock time. The authors also train and test each reranker on the same benchmark, and they admit that limits what we can say about transfer.

70
00:14:41,500 --> 00:14:42,800
[Hal Turing] And the security angle?

71
00:14:42,800 --> 00:15:07,181
[Dr. Ada Shannon] The authors name adversarial or malicious producers as future work. But retrieved chunks go straight into the consumer's context, so any open repository of executable-looking procedures is an injection surface. Prior work on poisoning agent memory and RAG corpora, such as AgentPoison and PoisonedRAG, points the same way. Producer-score features could also be gamed by claiming a strong identity.

72
00:15:07,181 --> 00:15:08,528
[Hal Turing] Is anything here solid?

73
00:15:08,528 --> 00:15:25,572
[Dr. Ada Shannon] Yes. The producer-consumer framing, the state-conditioned key, and labelling chunks by measured downstream gain are real contributions. The code and trajectories are released, so others can run these controls. Open questions the authors name are consumer personalization and producer incentives.

74
00:15:25,572 --> 00:15:53,714
[Hal Turing] So where does that leave us? On ALFWorld, retrieving small trajectory chunks and reranking by measured gain helps a population of agents, with a gain too large to be noise. The boundary is that WebArena's gain is within noise. And with no private-memory control and a same-template test split, the paper hasn't shown that sharing across a population, rather than lookup of seen templates, is what makes it work. Thanks for listening, everyone.