1
00:00:01,000 --> 00:00:09,300
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:09,300 --> 00:00:50,500
[Hal Turing] Today we're looking at "EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?", a 2026 preprint. The first author is Kunming Shao at the Hong Kong University of Science and Technology, with colleagues Jierun Chen and Haoli Bai at Huawei Technologies, and coauthors at the University of Hong Kong and Sun Yat-sen University. Here's the puzzle. Same coding-agent workload, three GPUs. Offloading the cache to host memory speeds up the RTX 3090, slows down the H800, and does nothing on the H20. On the H20, loading a token back is over six times cheaper than recomputing it.

3
00:00:50,500 --> 00:01:10,224
[Dr. Ada Shannon] So the obvious explanation, that loading is cheaper, can't be the whole story. The paper asks why offloading helps, hurts, or does nothing, and whether you can size the host tier by the pool's reuse working set to make it pay off reliably. The short answer is that hardware sets what a hit is worth, and the working set decides whether a hit exists at all.

4
00:01:10,224 --> 00:01:13,424
[Hal Turing] Start me at the bottom. What's being cached?

5
00:01:13,424 --> 00:01:33,775
[Dr. Ada Shannon] The KV cache. When a transformer processes a token, it stores that token's key and value vectors so later tokens can attend to them without redoing the work. Prefix caching keeps that state after a request finishes. If a new prompt starts with the same tokens, the server skips prefill for that shared part. vLLM and SGLang both do this.

6
00:01:33,775 --> 00:01:36,900
[Hal Turing] And agents are the perfect customer.

7
00:01:36,900 --> 00:01:57,725
[Dr. Ada Shannon] Yes. A coding agent loops: model call, run a tool like the test suite, read the result, call the model again. Every call resends the entire conversation. In SWE-bench Verified traces, 98.1% of prompt tokens had already been processed in the same task's previous call. Prefix caching alone makes those runs up to 11.1 times faster.

8
00:01:57,725 --> 00:02:01,425
[Hal Turing] But a cache hit needs an exact match, right?

9
00:02:01,425 --> 00:02:23,300
[Dr. Ada Shannon] Right, and it's stricter than it sounds. Each chunk's cache key hashes everything before it. Edit something early in the conversation and every later chunk misses. The paper's example is context folding, where the agent trims old context. On an RTX 3090 the prefix hit rate falls from 96.5% to somewhere between 51 and 61%. Removing tokens can cost more than it saves.

10
00:02:23,300 --> 00:02:27,900
[Hal Turing] Now offloading. GPU memory fills up, so what happens?

11
00:02:27,900 --> 00:02:49,425
[Dr. Ada Shannon] The server copies KV chunks over PCIe into CPU memory, the host tier, which is sized per GPU. When a later prompt needs a missing prefix, the system chooses per token: load it if the transfer is cheaper than recomputing. That's a roofline-style argument, bytes over bandwidth against FLOPs over compute. AttentionStore did this between chat turns, and LMCache and Mooncake do versions of it in production.

12
00:02:49,425 --> 00:02:53,850
[Hal Turing] So where does that logic break? If loading is cheaper, you load.

13
00:02:53,850 --> 00:03:24,975
[Dr. Ada Shannon] It assumes the prefix is still there. Agents run concurrently. The paper calls the number in flight the active pool, and it's a closed loop. Each agent's next call waits on its own tool work, and when the server is busy, everyone's turn gets slower. At the 5 GiB tier, the median gap between one agent's turns was 9.0 seconds. That's about 14 times the agent's own time between calls, so most of the gap is spent waiting behind other agents, while the server processes everyone else's context.

14
00:03:24,975 --> 00:03:29,350
[Hal Turing] So the prefix has to survive someone else's traffic.

15
00:03:29,350 --> 00:03:48,850
[Dr. Ada Shannon] Yes, and that's the problem. 97.3% of newly computed KV shows up again in the very next call, so nearly everything is worth keeping. But 79.2% of chunks written to the 5 GiB tier were evicted before that next call. The system pays PCIe bandwidth to write state nobody gets to read.

16
00:03:48,850 --> 00:03:54,825
[Hal Turing] That's a classic cache thrashing story. A loop bigger than the cache under LRU.

17
00:03:54,825 --> 00:04:17,250
[Dr. Ada Shannon] That's the right analogy, with one limit. Here the writes also compete with restores for link bandwidth, so thrashing costs more than a wasted fill. The paper names the footprint the reuse working set. It's the amount of other context the server processes between two uses of one agent's prefix. The host tier needs to hold roughly that much for the prefix to survive. For 16 agents on H20, they estimate 11.4 GiB per GPU rank.

18
00:04:17,250 --> 00:04:19,274
[Hal Turing] And the hardware side?

19
00:04:19,274 --> 00:04:50,524
[Dr. Ada Shannon] Two measures. Offload Benefit Ratio is the fraction of recompute time you save by restoring a missing prefix. It tells you what a hit is worth, not whether you get one. On H20 the lower bound for long prefixes is 0.84. The second is compute per host-link byte: peak dense BF16 FLOPs divided by host-to-GPU bandwidth. RTX 3090 is about 2.2 thousand, H20 about 2.3 thousand, and H800 about 15.5 thousand.

20
00:04:50,524 --> 00:04:58,799
[Hal Turing] So on H800 compute is so cheap relative to the link that recomputing nearly keeps up with restoring.

21
00:04:58,799 --> 00:05:12,774
[Dr. Ada Shannon] Exactly. A hit is worth little there. So a hit has to be valuable, which is hardware, and it has to exist, which is the working set. The H20 satisfies the first condition, so its flat result comes from the second.

22
00:05:12,774 --> 00:05:20,699
[Hal Turing] Is this the only way people attack the problem? I'd think you could just keep the agent's cache pinned while the tool runs.

23
00:05:20,699 --> 00:05:42,300
[Dr. Ada Shannon] That's one camp. Continuum pins an agent's KV during tool calls. Another camp, KVFlow, uses the agent's known step graph to decide what to evict and prefetch. Those are the keep-it-resident and use-the-known-future-order approaches. EfficientAgent does neither. It asks how big the tier must be for the whole pool, then manages writes so the tier isn't wasted when it falls short. Whether that beats pinning is something we should come back to.

24
00:05:42,300 --> 00:05:47,875
[Hal Turing] So how does it work out how big the tier has to be? Presumably it doesn't just guess.

25
00:05:47,875 --> 00:06:08,524
[Dr. Ada Shannon] It counts. The tool is reuse distance, from the stack-distance analysis Mattson and colleagues published at IBM in 1970. For one chunk of KV, say 1,024 tokens, you count the distinct other chunks touched between two uses of it. Under least-recently-used eviction, the chunk is still in a tier if that count is smaller than the number of chunks the tier holds.

26
00:06:08,524 --> 00:06:12,299
[Hal Turing] But a prefix isn't one chunk. It's a chain.

27
00:06:12,299 --> 00:06:31,749
[Dr. Ada Shannon] Right, and that matters. A restore only helps for the unbroken run of surviving chunks from the start of the prompt. If chunk three was evicted, chunks four onward are useless for that prefix. The model also subtracts whatever the GPU already holds, so it only credits the host tier for restores the GPU couldn't do itself.

28
00:06:31,749 --> 00:06:34,199
[Hal Turing] And the headline estimate?

29
00:06:34,199 --> 00:06:59,449
[Dr. Ada Shannon] For a pool of A agents, about A minus one times the mean prompt length times the KV bytes per token. On H20 with sixteen agents, mean prompts around 33K tokens, and 24 KiB per token per rank for the mixture-of-experts model, that comes to 11.4 GiB per rank. Divide by the host tier and you get a pressure ratio: about 2.3 at 5 GiB, 1.1 at 10, and 0.57 at 20. Above one, the tier can't hold what the pool reuses.

30
00:06:59,449 --> 00:07:07,724
[Hal Turing] How do they test any of this? Every serving policy changes when agents come back, so each run sees different work.

31
00:07:07,724 --> 00:07:28,474
[Dr. Ada Shannon] That's why they built dependency-preserving replay. They recorded 4,427 calls, about 147 million prompt tokens. Each replay sends the exact recorded prompt, forces the model to emit the recorded output tokens, and waits out the recorded tool time before the next call. Policies change queueing and interleaving, but not the token work.

32
00:07:28,474 --> 00:07:30,674
[Hal Turing] Now the first result.

33
00:07:30,674 --> 00:07:58,949
[Dr. Ada Shannon] At 5 GiB per rank, offload is no faster than no offloading: 209 minutes against 212. The tier writes about six and a half tokens for every token it reads back. At 20 GiB the same replay finishes in 128 minutes, roughly 60% of the time, and computed prefill drops by 93%. Past 20 GiB it goes flat. One caveat: repeated 40 GiB runs vary by about eight minutes, so small makespan gaps aren't safe to read.

34
00:07:58,949 --> 00:08:05,274
[Hal Turing] So the tier stops being a graveyard and becomes a cache once it clears the working set.

35
00:08:05,274 --> 00:08:24,649
[Dr. Ada Shannon] Yes. Now the admission rule. If the working-set estimate exceeds the tier and the tier is full and evicting, it skips writing any request with more than kappa new chunks. Large refills of context that was already evicted get declined. Small extensions of prefixes still resident get saved. Otherwise it writes everything.

36
00:08:24,649 --> 00:08:29,024
[Hal Turing] Why does declining a refill help? You're throwing away data.

37
00:08:29,024 --> 00:08:51,074
[Dr. Ada Shannon] The refill is data the tier already failed to keep once. Writing it evicts resident prefixes that might still come back. Skipping it leaves room for them. Denning made the same argument for operating-system memory in 1968, with the working set model. When programs' combined working sets exceed memory, you thrash, so you suspend some load. The analogy's limit is that the OS suspends programs, while this rule drops writes.

38
00:08:51,074 --> 00:08:55,324
[Hal Turing] Is there anything guaranteeing it doesn't make things worse?

39
00:08:55,324 --> 00:09:15,949
[Dr. Ada Shannon] Their Proposition 1, under an idealized LRU model. If you only decline a miss whose chunk would next be reused beyond the tier's capacity, or never, then no hit under full admission is lost. That chunk would have been evicted before anyone read it. At 5 GiB, 84 to 85% of declined chunks met that condition. At 40 GiB only 4.8% did.

40
00:09:15,949 --> 00:09:20,499
[Hal Turing] Which suggests the same filter is harmful with a big tier.

41
00:09:20,499 --> 00:09:44,174
[Dr. Ada Shannon] It is. A filter applied always cuts prefill by 36% at 5 GiB. At 40 GiB it makes prefill more than four times worse, and makespan goes from 124 to 162 minutes. Capacity-conditioned admission keeps the 5 GiB gain, 35% less prefill, and at 40 GiB the pressure signal never fires, so it matches plain offload. The 5 GiB makespan gain is only about 10%, though. A bigger tier buys far more.

42
00:09:44,174 --> 00:09:47,399
[Hal Turing] And the odd one out, H800?

43
00:09:47,399 --> 00:10:08,799
[Dr. Ada Shannon] RTX 3090 and H20 both have about 2.2 to 2.3 thousand FLOPs of compute per host-link byte. On both, offload ran 8 to 40% faster once hits survived. H800 has about 15.5 thousand, and offload was slower in every run measured, by 8 to 87%. Restoring is relatively dear there and recomputing relatively cheap.

44
00:10:08,799 --> 00:10:11,249
[Hal Turing] Did halving the pool matter?

45
00:10:11,249 --> 00:10:25,899
[Dr. Ada Shannon] Yes. With eight agents, the working set drops to 5.3 GiB and a 3 GiB tier already recovers prefixes. Across all 17 runs, makespan tracks computed prefill, whatever intervention produced it.

46
00:10:25,899 --> 00:10:31,824
[Hal Turing] The abstract says predictions were made before the experiments. What did the model say?

47
00:10:31,824 --> 00:11:01,399
[Dr. Ada Shannon] It predicted 5.19 to 5.78 million prefill tokens at 20 GiB, and the run gave 5.28 million. At 80 GiB it predicted 5.19 to 5.31, and the run gave 5.29. At 10 GiB, inside the transition, it missed on restores: 65.2 million actual against 55.3 to 58.9 predicted. The predictions were recorded before those runs, but computed from the same recorded trajectories the replay executes.

48
00:11:01,399 --> 00:11:03,799
[Hal Turing] So how much should that count?

49
00:11:03,799 --> 00:11:21,899
[Dr. Ada Shannon] Less than the abstract suggests. The stack-distance model reads the same trajectories that the replay then executes, so it forecasts cache behavior for a workload it has already seen. Nobody calibrated on one set of tasks and tested on another, or changed the pool size or the model.

50
00:11:21,899 --> 00:11:25,674
[Hal Turing] And the one point inside the transition missed.

51
00:11:25,674 --> 00:11:45,324
[Dr. Ada Shannon] Yes, 10 GiB. The model was low by roughly 11 to 18 percent on restores. That is the point where a planner needs the model most. Far from the threshold, almost any model gets it right. So 'locates the transition' really means somewhere between 5 and 20 GiB, a factor of four around an 11.4 GiB estimate.

52
00:11:45,324 --> 00:11:50,074
[Hal Turing] What about the estimate itself? It uses a mean prompt length.

53
00:11:50,074 --> 00:12:11,449
[Dr. Ada Shannon] That's the soft spot. It assumes a backlogged pool of agents with comparable context lengths. These contexts run from near zero to 238K tokens and grow within a task. Agents on the same repository also share system prompts, repo maps and early tool output, so counting every agent's context as distinct probably overstates the working set.

54
00:12:11,449 --> 00:12:15,274
[Hal Turing] Does the paper's trace-based model handle that?

55
00:12:15,274 --> 00:12:33,199
[Dr. Ada Shannon] The trace model keeps real ordering, sharing and lengths. But the runtime gate uses the cheap windowed approximation. The paper doesn't show how often the two disagree. Comparing the closed-form estimate, the runtime estimate and the trace number across pools and workloads would settle it.

56
00:12:33,199 --> 00:12:37,574
[Hal Turing] Then there's the makespan evidence. You mentioned repeats.

57
00:12:37,574 --> 00:12:59,800
[Dr. Ada Shannon] Offload at 40 GiB ran three times: 124.0, 115.9 and 123.7 minutes. That's about a 7 percent spread. Conditioned admission at 5 GiB improves makespan by 10.4 percent, which is only a modest multiple of that noise. At 10 GiB the gain is 161 to 157 minutes, which sits inside it. Most points are single runs, with no confidence intervals.

58
00:12:59,800 --> 00:13:01,850
[Hal Turing] So what survives?

59
00:13:01,850 --> 00:13:14,775
[Dr. Ada Shannon] Computed prefill. It's far more reproducible, and the 4.3-fold reversal at 40 GiB is too large to be noise. I'd trust the sign flip and hold the small makespan gains loosely.

60
00:13:14,775 --> 00:13:17,975
[Hal Turing] The hardware story seems shakier still.

61
00:13:17,975 --> 00:13:45,550
[Dr. Ada Shannon] It is. Compute per host-link byte uses peak BF16 FLOPs, not achieved prefill throughput. RTX 3090 and H20 have nearly the same ratio, yet offload gives 0.91 on one and 0.60 on the other. Across the three GPUs, tensor-parallel degree, memory fraction, GPU KV size and model also change. A controlled sweep would throttle PCIe or lock clocks on one machine. Without one, the ratio is a plausible explanation, not an isolated cause.

62
00:13:45,550 --> 00:13:50,800
[Hal Turing] And the H800 slowdown? Were those host tiers even big enough?

63
00:13:50,800 --> 00:14:11,800
[Dr. Ada Shannon] The paper doesn't show the H800 sweeps reaching a tier above the working set. If the tier was undersized, that's thrashing plus a low benefit ratio, not the ratio alone. Separately, the memory fractions were chosen to make serving memory-constrained. How much of the H20 gain survives with GPU KV provisioned the way operators would do it, nobody knows.

64
00:14:11,800 --> 00:14:15,000
[Hal Turing] What's missing on the comparison side?

65
00:14:15,000 --> 00:14:41,275
[Dr. Ada Shannon] Head-to-head baselines. Continuum, from Hanchen Li and colleagues in 2025, is the keep-it-resident approach. MORI, from Tian Xia and colleagues in 2026, does admission control and idleness-based placement across GPU and CPU memory, which overlaps directly with capacity-conditioned admission. KVFlow, by Zaifeng Pan and colleagues in 2025, knows future order from the step graph, which the working-set model ignores. The authors call all three composable. None is measured.

66
00:14:41,275 --> 00:14:44,825
[Hal Turing] Is the admission rule itself fine-grained?

67
00:14:44,825 --> 00:15:03,400
[Dr. Ada Shannon] Hardly. Any threshold from 2 to 16 declines 90-plus percent of writes at 5 GiB. That looks like 'decline almost everything'. It also never tests a plain write-rate limiter, so we can't tell how much of the gain comes from Proposition 1's mechanism and how much from less PCIe traffic.

68
00:15:03,400 --> 00:15:06,175
[Hal Turing] What should an operator do with this?

69
00:15:06,175 --> 00:15:36,100
[Dr. Ada Shannon] Use the sizing rule as a first-pass check: active agents minus one, times context length, times bytes per token. The paper's own example is 64 agents at 64K tokens, about 94.5 GiB per GPU at 24 KiB per token. Don't turn offload on by default for high-compute-per-byte GPUs like the H800. Judge context folding by cache-stable prompt length, not tokens removed. And pool size is a memory knob, since halving it cut recompute makespan 24.8 percent.

70
00:15:36,100 --> 00:15:41,450
[Hal Turing] Bytes per token matters there. KV quantization would shrink that number.

71
00:15:41,450 --> 00:15:55,950
[Dr. Ada Shannon] It would shrink the working set and change the restore-versus-recompute trade at the same time. The verdict from a BF16 mixture-of-experts model may not carry over. The same goes for latent caches and hybrid models.

72
00:15:55,950 --> 00:15:58,225
[Hal Turing] So where does this go next?

73
00:15:58,225 --> 00:16:15,100
[Dr. Ada Shannon] Held-out validation across tasks, pools and models. Live runs at larger pools, rather than replay. Combining working-set sizing with TTL or step-graph information. And contexts that grow and are shared across agents, which the model doesn't treat.

74
00:16:15,100 --> 00:16:39,600
[Hal Turing] Let me sum up. Offload pays when a hit is worth something and the prefix survives until the agent returns. The right write filter flips sign across the working set, and the host tier should be sized to the pool. The boundary is one replayed workload on a few deployments, a coarse transition rather than a sharp forecast, and no head-to-head against pinning or step-graph policies. Thanks for listening.