1
00:00:01,000 --> 00:00:08,198
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:08,198 --> 00:00:42,331
[Hal Turing] Today we're covering ECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native Sparse Attention LLMs. It's from Guangda Liu and colleagues at Shanghai Jiao Tong University, with Huawei and Guizhou University, and Jieru Zhao is the corresponding author. It appeared at OSDI 2026. The word that jumps out is "lossless" in the title, sitting next to "prefetching." Prefetching is a guess, and guesses are usually lossy.

3
00:00:42,331 --> 00:01:03,229
[Dr. Ada Shannon] The question is this. When a sparse-attention model like DeepSeek-V3.2 runs out of GPU memory before it runs out of compute, can you push the KV cache to host memory and hide the PCIe cost, without changing which tokens attention uses? And how much throughput does that buy? We'll spend part of this checking exactly what "lossless" covers.

4
00:01:03,229 --> 00:01:08,941
[Hal Turing] Start with why memory is the problem. Sparse attention is supposed to make long context cheap.

5
00:01:08,941 --> 00:01:28,771
[Dr. Ada Shannon] It cuts compute, not storage. Define the terms first. The KV cache holds the keys and values for every earlier token, so you don't recompute them. Offloading means the full copy lives in host DRAM. Recall means copying the entries a step needs back to the GPU over PCIe.

6
00:01:28,771 --> 00:01:56,403
[Dr. Ada Shannon] DeepSeek Sparse Attention adds a small scorer called the lightning indexer. For each new query it scores every earlier token, using 64 heads of dimension 128 in FP8. The top 2048 tokens are the only ones the main attention, MLA, actually reads. That saves compute. But every token still has a cache entry, so the cache grows linearly with context. The indexer also keeps its own key cache, which adds more.

7
00:01:56,403 --> 00:01:57,889
[Hal Turing] So how tight is it in numbers?

8
00:01:57,889 --> 00:02:22,827
[Dr. Ada Shannon] On eight H200s, a worker holds about 655,000 tokens. That's six requests of 100K tokens. On eight H20s with 4-bit AWQ weights, each data-parallel worker holds at most about 380,000 tokens, so three or four 100K requests. MLA's compressed KV also can't be split by tensor parallelism, so adding GPUs doesn't spread it out.

9
00:02:22,827 --> 00:02:24,638
[Hal Turing] And small batches matter because?

10
00:02:24,638 --> 00:02:39,081
[Dr. Ada Shannon] Sparse kernels run at low utilization, so they need big batches to saturate the hardware. The paper's example is eight requests of 64K tokens, which gives only 53 percent utilization. So you're capacity-bound and starving the kernels at the same moment.

11
00:02:39,081 --> 00:02:42,053
[Hal Turing] Why can't you just copy the needed tokens up when you need them?

12
00:02:42,053 --> 00:02:56,821
[Dr. Ada Shannon] Because of the top-k dependency. The selected set is only known after the indexer has scored all earlier tokens. So the naive order is score, select, fetch, attend. The fetch sits on the critical path while PCIe was idle during scoring.

13
00:02:56,821 --> 00:02:59,097
[Hal Turing] Okay, so prefetch. Guess early and overlap.

14
00:02:59,097 --> 00:03:25,103
[Dr. Ada Shannon] Right, and prior work did that by guessing the token set. InfiniGen, by Wonbeom Lee and colleagues in 2024, prefetches across layers. It approximates attention with partial channels of the query and key to predict what the next layer needs. FreeKV, by Liu and colleagues in 2025, prefetches across steps, betting that adjacent decode steps select similar tokens. Both are lossy, because a wrong guess means a missing token.

15
00:03:25,103 --> 00:03:26,914
[Hal Turing] And the offloaders that don't guess?

16
00:03:26,914 --> 00:03:42,704
[Dr. Ada Shannon] The paper's framing is that they fall short in different ways. Strata is static. ArkVale, InfiniGen, FreeKV and CLO are single-GPU research systems. SparseServe can't run under graph execution, and that last one matters a lot.

17
00:03:42,704 --> 00:03:46,233
[Hal Turing] Graph execution. Explain that one for people who haven't fought with it.

18
00:03:46,233 --> 00:04:10,382
[Dr. Ada Shannon] Decoding a token is a long chain of small kernels, and launching each from the CPU costs microseconds, which is comparable to the kernels themselves at small batch. A CUDA graph records the whole decode step once and replays it with one launch, no CPU involved. The catch is that a captured graph is frozen. Shapes, addresses and topology are fixed, and CPU decisions can't run in the middle.

19
00:04:10,382 --> 00:04:40,614
[Dr. Ada Shannon] A normal cache manager is the opposite. It uses variable-length lists and CPU logic about what to evict and what to copy. And with sparse attention, the needed tokens change every step, for every request and every layer. So a CPU-side manager either lands on the critical path or forces you out of graph mode. ECHO's answer is a graph-friendly manager. The bookkeeping lives on the GPU as fixed-size integer tensors, with parallel allocate, free and recall operations that can be captured in the graph.

20
00:04:40,614 --> 00:04:45,026
[Hal Turing] Where does the predictability come from, though? Why would anyone expect to guess early?

21
00:04:45,026 --> 00:05:04,159
[Dr. Ada Shannon] From the indexer itself, which comes from the DeepSeek-V3.2-Exp report, DeepSeek-AI, 2025. The paper's claims about score predictability depend on this specific trained indexer. That's not a law of attention. It's a property of one model family, and I'd keep that in mind throughout.

22
00:05:04,159 --> 00:05:07,642
[Hal Turing] So back to the title. What does lossless prefetching mean here?

23
00:05:07,642 --> 00:05:24,964
[Dr. Ada Shannon] As a design property, it means prefetching starts a guess early, and after the indexer finishes, a guaranteed recall fetches any selected token that was missed. A wrong guess costs time, never the token set. InfiniGen and FreeKV guess which tokens. ECHO guesses only when to fetch them.

24
00:05:24,964 --> 00:05:27,704
[Hal Turing] That's a clean argument. Is it also a measurement?

25
00:05:27,704 --> 00:05:31,048
[Dr. Ada Shannon] That's the question worth coming back to once we've seen the machinery.

26
00:05:31,048 --> 00:05:34,299
[Hal Turing] Then the machinery. What does the cache manager actually keep?

27
00:05:34,299 --> 00:05:54,129
[Dr. Ada Shannon] There are five fixed-size integer tensors on the GPU. A free-slot bitmap, a priority per slot, an index buffer for results, a map from GPU slot to host slot, and a map back from host slot to GPU slot. Nothing changes length, so the whole decode step can be captured as one graph.

28
00:05:54,129 --> 00:05:57,472
[Hal Turing] Why per layer? One cache for the whole model sounds simpler.

29
00:05:57,472 --> 00:06:15,491
[Dr. Ada Shannon] Because each layer's indexer picks different tokens. Layer 3 and layer 50 want different sets, so each layer gets its own GPU cache. The host copy stays per model, since every token is backed up there anyway. The metadata costs about 10 megabytes per layer, roughly 610 total.

30
00:06:15,491 --> 00:06:18,184
[Hal Turing] And allocation without a CPU deciding anything?

31
00:06:18,184 --> 00:06:41,822
[Dr. Ada Shannon] Parallel threads each check whether their slot is free. The free ones do an atomicAdd on a global counter, and the number they get back decides who takes a slot. Freeing bumps the priority of tokens the indexer just selected, so they're protected, then runs an argtopk to find the lowest-priority slots. That's roughly LRU. There's no write-back on eviction, because the host already has everything.

32
00:06:41,822 --> 00:06:43,912
[Hal Turing] So eviction is just metadata edits.

33
00:06:43,912 --> 00:07:07,597
[Dr. Ada Shannon] Yes. Recall scatters the selected indices against the host-to-GPU map to find misses, allocates slots, then GPU kernels read host memory directly over PCIe. The allocator microbenchmark is about 12 microseconds for SGLang's against 6 to 8 for theirs. That's a small win by itself. The real gain is that the decode path stays in one graph.

34
00:07:07,597 --> 00:07:10,847
[Hal Turing] Now the prefetching. How do they get around the top-k dependency?

35
00:07:10,847 --> 00:07:32,395
[Dr. Ada Shannon] They exploit a cost asymmetry. The indexer touches all L earlier tokens, so its work grows with context. Recall moves at most k tokens, and k is 2048. They also use different resources, GPU compute and the PCIe link. So you can fetch while scoring, if you can start before the final top-k is known.

36
00:07:32,395 --> 00:07:33,742
[Hal Turing] Which needs a threshold.

37
00:07:33,742 --> 00:07:55,337
[Dr. Ada Shannon] Right. They predict the 2048th-highest score for the next step with an exponential moving average, alpha 0.5. During the indexer pass, any token scoring above that predicted threshold gets fetched immediately. A counter caps the early fetches in proportion to context length, so prefetching can't slow the indexer. Afterward the guaranteed recall fetches whatever was missed.

38
00:07:55,337 --> 00:07:57,473
[Hal Turing] So what does a bad prediction cost?

39
00:07:57,473 --> 00:08:16,978
[Dr. Ada Shannon] Too high a threshold, and fewer tokens get fetched early, so more waits for the final recall. Too low, and you fetch extras, up to the cap. Either way, the final selection still comes from the full scores. The prediction changes how much moves early. Their figures show candidate counts hovering near 2048, sometimes above it.

40
00:08:16,978 --> 00:08:19,950
[Hal Turing] What about prefill? There the queries come in blocks.

41
00:08:19,950 --> 00:08:44,284
[Dr. Ada Shannon] They use that ordering. While the indexer scores query block i+1, the system fetches block i's tokens. Exact top-k there slows the indexer, so they use a cruder version: a single-round histogram with 256 bins over the top 8 bits of each score. Scores are shifted by the predicted threshold so the boundary bin stays small. That selects a subset for prefetching only.

42
00:08:44,284 --> 00:08:46,467
[Hal Turing] And all of this lives inside one kernel?

43
00:08:46,467 --> 00:09:02,675
[Dr. Ada Shannon] They extend DeepGEMM's indexer kernel. Specialized warp groups handle loading, score computation, and a new prefetch stage, overlapped with asynchronous barriers. In decoding the work is split over the indexer K cache. In prefill it's split over query blocks.

44
00:09:02,675 --> 00:09:06,111
[Hal Turing] One deployment choice stood out. Offloading is off on prefill.

45
00:09:06,111 --> 00:09:25,152
[Dr. Ada Shannon] They recommend disaggregating prefill and decode. A prefill chunk can touch up to queries-times-k distinct tokens and evict other requests' cache. Prefill writes to host memory also interfered with recall. So prefill instances don't offload, and decode instances hold the KV in host memory.

46
00:09:25,152 --> 00:09:26,916
[Hal Turing] Now the results. What's the setup?

47
00:09:26,916 --> 00:09:52,458
[Dr. Ada Shannon] Eight H20 GPUs with 96 gigabytes each, PCIe Gen5, a 4-bit AWQ DeepSeek-V3.2-Exp, and 318 InfiniteBench requests of 80K to 100K tokens. It's a decode-only instance with precomputed KV, and output fixed at 256 tokens. The host pool holds 1.8 million tokens, about a terabyte. Baselines are SGLang and vLLM.

48
00:09:52,458 --> 00:09:53,480
[Hal Turing] And the headline?

49
00:09:53,480 --> 00:10:16,793
[Dr. Ada Shannon] At unlimited request rate, ECHO gets 2.15 times SGLang's generation throughput and 4.1 times vLLM's. It's a throughput number, so more tokens per second. The cause is batch size. SGLang's pool gives roughly 30 effective concurrent requests across workers, vLLM about 1.5, and ECHO's host pool lets it run far more.

50
00:10:16,793 --> 00:10:18,186
[Hal Turing] So it's a capacity result.

51
00:10:18,186 --> 00:10:52,226
[Dr. Ada Shannon] Yes, and the capped runs show it. Limit the GPU pool to 200K tokens and the gap grows to 3.1 times SGLang. At 110K it reaches up to 4.12 times. The tighter the memory, the more offloading helps. Two caveats. The 256-token cap keeps every worker busy. With uncapped outputs the advantage narrows, though the text gives no exact figure. And per-task results are mixed, from a 27 percent gain on one code task to a 1.7 percent loss on another.

52
00:10:52,226 --> 00:10:54,038
[Hal Turing] What does prefetching add to that?

53
00:10:54,038 --> 00:11:19,997
[Dr. Ada Shannon] Much less. The GPU cache already hits about 97 to 99 percent on most layers. In the microbenchmark, prefetching cuts indexer-plus-recall latency up to 1.29 times at a 50 percent hit rate and 1.51 times at 90. End to end it's at most 4 percent of throughput. Prefill prefetching reaches at most 1.1 times, assuming randomly distributed selections.

54
00:11:19,997 --> 00:11:20,926
[Hal Turing] Why so small?

55
00:11:20,926 --> 00:11:40,292
[Dr. Ada Shannon] With hit rates that high, little remains to prefetch. The authors also point out that MoE layers take about 70 percent of decode time. So the cache manager and the larger host pool carry the headline. Prefetching is a refinement, and it hides recall well mainly where hit rates are lower.

56
00:11:40,292 --> 00:11:44,889
[Hal Turing] So back to the word lossless. Does the paper run an end-to-end accuracy check?

57
00:11:44,889 --> 00:12:09,642
[Dr. Ada Shannon] In the sections I read, no. Lossless here is a property of the design. Any selected token the prefetch missed gets recalled after the indexer finishes, so the final top-k set is exact whatever the predictor did. That argument is sound. But it is an argument, not a measurement. InfiniteBench appears only as a throughput workload. I found no task scores comparing ECHO with SGLang.

58
00:12:09,642 --> 00:12:11,406
[Hal Turing] Is that a red flag, or just a gap?

59
00:12:11,406 --> 00:12:30,168
[Dr. Ada Shannon] A gap. Some risks sit outside the design argument, though. Every run uses a 4-bit AWQ model, the indexer runs in FP8, and the fused kernel could break ties or round differently from SGLang's path. The paper doesn't test any of that. I'm not claiming they differ, only that nobody checked.

60
00:12:30,168 --> 00:12:41,825
[Hal Turing] The prefill path worries me more. It picks what to prefetch with an approximate top-k, a one-round histogram over eight bits of the score. That's only safe if the final selection stays exact.

61
00:12:41,825 --> 00:12:54,549
[Dr. Ada Shannon] Right, and the design says the guaranteed recall covers it. The test is cheap, though. Dump the selected indices per layer per step from both systems and compare them. The text I read reports no such check.

62
00:12:54,549 --> 00:12:59,379
[Hal Turing] What about the decode predictor? The moving average of the k-th score looked tidy.

63
00:12:59,379 --> 00:13:29,797
[Dr. Ada Shannon] The evidence is Figures 7 and 8: two layers, 3 and 50, and one 20K-token ShareGPT request. The benchmark runs 80K to 100K tokens from InfiniteBench across 61 layers. There's no sweep over the smoothing factor, context length, or task type. Because of the design, a bad predictor costs speed, not correctness, and the cap on early fetches limits the damage. So it's a performance-robustness gap. To settle it, show threshold error and wasted-prefetch volume across all layers and several tasks.

64
00:13:29,797 --> 00:13:31,747
[Hal Turing] The prefill side has the same problem?

65
00:13:31,747 --> 00:13:48,280
[Dr. Ada Shannon] Thinner still. The prefill microbenchmark assumes random top-k overlap between queries, and the evaluation uses precomputed KV, so there's no end-to-end ablation. I'd call inter-query prefetching plausible but unvalidated on real prefill traces.

66
00:13:48,280 --> 00:13:50,974
[Hal Turing] How far does this carry beyond DeepSeek-V3.2?

67
00:13:50,974 --> 00:14:23,667
[Dr. Ada Shannon] The cache manager looks portable to other sparse attention. The prefetching depends on how this one trained indexer scores tokens, per the DeepSeek-V3.2-Exp report. Against InfiniGen and FreeKV, the contrast is clean. They guess which tokens matter, so they can be lossy. ECHO guesses only timing and recalls the exact set. The cost is that lossy methods can cut traffic by never fetching low-value tokens, and ECHO always fetches all of them. The text I read has no head-to-head against either, so I wouldn't claim ECHO beats them.

68
00:14:23,667 --> 00:14:26,593
[Hal Turing] Then latency. Offloading has to cost something.

69
00:14:26,593 --> 00:14:56,872
[Dr. Ada Shannon] It does, at light load. On ShareGPT, per-token latency rises 2.7 to 27.8 percent, and a whole request takes about a fifth longer at the lowest rates. From request rate 0.5 up, the overhead falls below 5 percent. Time to first token rises by at most 7.9 percent. This is a throughput-first system. The caveat is that latency was measured on short ShareGPT prompts, not the long-context workload behind the throughput claim.

70
00:14:56,872 --> 00:14:59,798
[Hal Turing] And the headline throughput number, how far does it travel?

71
00:14:59,798 --> 00:15:23,807
[Dr. Ada Shannon] It's the best case: saturated arrivals, outputs capped at 256 tokens. The hardware is H20 only, the model is quantized, and no variance is reported. The 4.1x over vLLM mostly reflects that vLLM fit only about 1.5 requests at a time in this setup, so that gap says more about capacity than about the algorithm.

72
00:15:23,807 --> 00:15:24,736
[Hal Turing] Who should care?

73
00:15:24,736 --> 00:15:42,290
[Dr. Ada Shannon] Anyone running batch-heavy, long-context serving of DSA models where HBM is the limit. Interactive, latency-sensitive serving is a worse fit. The authors argue faster links like NVLink-C2C will keep offloading relevant. That's a projection. Nothing here measures it.

74
00:15:42,290 --> 00:16:08,529
[Hal Turing] So here's where I land. ECHO shows that moving the KV cache to host memory, with a graph-friendly manager, lifts long-context throughput roughly twofold on this setup. It also shows prefetch timing can be separated from selection correctness by design. The limit is that output equivalence is argued rather than measured, the predictor evidence is thin, and the gains rest on one model family and one kind of hardware.

75
00:16:08,529 --> 00:16:10,108
[Dr. Ada Shannon] Agreed. Thanks for listening.

76
00:16:10,108 --> 00:16:12,058
[Hal Turing] Thanks, everyone. See you next time.