1
00:00:01,000 --> 00:00:07,919
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,919 --> 00:00:48,600
[Hal Turing] Today's paper is FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving, from 2026. The first author is Xuan Wang, with colleagues at Huawei Technologies and Huawei Ascend Computing Technology Development, plus Shanghai Jiao Tong University. Cheng Li is the corresponding author. The claim is that you can keep the entire KV cache in host DRAM, during both prefill and decode, and still get up to 1.38 times the prefill throughput and 1.91 times the decode throughput under a 30 millisecond per-token budget. That's on Ascend 910C chips, serving GLM-5.2.

3
00:00:48,600 --> 00:01:03,043
[Dr. Ada Shannon] The stakes are that million-token agent contexts don't fit in accelerator memory at any useful concurrency. The authors say they can serve them anyway, without changing what attention computes. We'll also check what those headline multipliers are actually comparing.

4
00:01:03,043 --> 00:01:07,269
[Hal Turing] Start with the memory problem. What's the KV cache, and how big does it get?

5
00:01:07,269 --> 00:01:29,050
[Dr. Ada Shannon] For every past token, each layer stores a key and a value, so decode doesn't recompute them. For GLM-5.2 that's 95.23 kilobytes per token. At 64k tokens that's about 6 gigabytes for one request. At 512k it's about 49 gigabytes, and at a million tokens it's about 97.5.

6
00:01:29,050 --> 00:01:30,489
[Hal Turing] And the chip has how much room?

7
00:01:30,489 --> 00:01:47,022
[Dr. Ada Shannon] After weights and activations, each decode die has roughly 28 gigabytes left for KV. At 64k that's a batch of about four requests. At 512k, not even one request fits. So HBM, the high-bandwidth memory on the chip, caps concurrency before compute does.

8
00:01:47,022 --> 00:01:51,619
[Hal Turing] Why does concurrency matter so much? Isn't latency the thing users feel?

9
00:01:51,619 --> 00:02:13,307
[Dr. Ada Shannon] Both matter, and the paper's metrics keep them apart. TTFT is the wait for the first token. TPOT is the average time per output token as the user sees it. ITL is the time for one decode step. Throughput is batch size divided by TPOT. A bigger batch only helps if TPOT doesn't grow faster than the batch.

10
00:02:13,307 --> 00:02:15,072
[Hal Turing] Why would TPOT and ITL differ?

11
00:02:15,072 --> 00:02:34,576
[Dr. Ada Shannon] Speculative decoding. The model drafts several tokens and verifies them together, so one step can emit more than one token. The paper uses a variant called MTP3, multi-token prediction, with four candidate positions per step. That's a multiplier on tokens per step, so TPOT comes out lower than ITL.

12
00:02:34,576 --> 00:02:36,434
[Hal Turing] And prefill-decode disaggregation?

13
00:02:36,434 --> 00:03:01,093
[Dr. Ada Shannon] Separate nodes do the two phases. Prefill nodes process the prompt in chunks, which is compute-heavy. Decode nodes generate step by step, which is memory-heavy. The KV cache moves from the prefill node to the decode node, and here it lands in the decode node's host DRAM. Mooncake, from Moonshot AI in 2024, is the well-known production example of that split with tiered KV storage.

14
00:03:01,093 --> 00:03:05,830
[Hal Turing] So the obvious move is parking cold KV in host DRAM. Why isn't that old news?

15
00:03:05,830 --> 00:03:22,549
[Dr. Ada Shannon] Because host links are far slower than HBM. If every decode step read the whole cache, you'd be waiting on transfers constantly. FlexGen, by Ying Sheng and colleagues in 2023, did layerwise offload well, but for offline throughput, not a per-token latency budget.

16
00:03:22,549 --> 00:03:23,570
[Hal Turing] So what changed?

17
00:03:23,570 --> 00:03:49,716
[Dr. Ada Shannon] The model. DeepSeek-V3.2 introduced sparse attention with a lightweight learned indexer, and GLM-5.2 uses the same idea. Each step, the indexer picks the top 2048 tokens, and attention reads only those. Nothing is evicted from the model's view, so you must keep the full history. But each step touches a tiny subset, about one kilobyte per access.

18
00:03:49,716 --> 00:03:53,385
[Hal Turing] How is that different from Quest or H2O, which also pick a subset?

19
00:03:53,385 --> 00:04:19,577
[Dr. Ada Shannon] Those bolt selection onto a model trained for dense attention. Quest, from Jiaming Tang and colleagues in 2024, scores pages with min and max key metadata. That's approximate, and the paper says such methods lose at least one percent quality. DeepSeek's Native Sparse Attention trains the selection in, but with coarser blocks. Here selection is native and fine-grained, and the paper's argument is that attention stays exact over the selected tokens.

20
00:04:19,577 --> 00:04:23,524
[Hal Turing] So the working set per step is small. Is the offloading problem solved?

21
00:04:23,524 --> 00:04:50,785
[Dr. Ada Shannon] No. You keep a small hot cache of tokens in HBM. Each step, selected tokens that aren't there are misses, and you load them over the host-to-device link, the H2D path, then evict the same number. Earlier systems for this kind of model, ESS, HiSparse, ECHO and AVSG, work this way. The paper's complaint is that their cache management cost grows with cache size, so the cache can't get big enough to cut misses.

22
00:04:50,785 --> 00:04:53,153
[Hal Turing] And prefill and decode behave differently here?

23
00:04:53,153 --> 00:05:09,360
[Dr. Ada Shannon] Very. Prefill has big compute windows per layer and bulk, contiguous transfers, so copies hide behind compute easily. Decode transfers are tiny, they can't start until top-k finishes, and the compute per layer is small. That's hard to overlap.

24
00:05:09,360 --> 00:05:10,429
[Hal Turing] So what's the plan?

25
00:05:10,429 --> 00:05:34,856
[Dr. Ada Shannon] Three pieces. Rolling buffers for prefill, so only a few layers' KV sit in HBM at once. A reverse index with a fused kernel for decode cache management. And a kernel that fetches misses while attention runs on hits. We'll unpack each one, and keep one question open: when the paper says 1.91 times, what is that a ratio of? That depends on what the baseline was allowed to do.

26
00:05:34,856 --> 00:05:42,751
[Hal Turing] Let's start with prefill, since that's the half that sounds easier. What does the system actually keep in accelerator memory while it processes a long prompt?

27
00:05:42,751 --> 00:06:01,095
[Dr. Ada Shannon] Four layer-sized buffers, instead of all 78 layers. Only the layer currently computing needs its full keys and values. So the full history stays in host memory, and the device holds a small rotating window. Buffer zero stays bound to layer zero. The other three rotate across the later layers.

28
00:06:01,095 --> 00:06:02,256
[Hal Turing] Why pin layer zero?

29
00:06:02,256 --> 00:06:31,884
[Dr. Ada Shannon] Layer zero never waits for a load, and its compute time is the head start for the pipeline. While it runs, layer one loads into the next buffer. After that, three things overlap: loading a future layer, computing the current one, and writing the finished one back to host. A buffer is reused only when both its compute and its writeback are done. If none is free, the pipeline waits rather than overwrite live data. A map called onload_layer_map sets how many layers ahead to prefetch.

30
00:06:31,884 --> 00:06:33,138
[Hal Turing] So that's layer streaming.

31
00:06:33,138 --> 00:07:01,606
[Dr. Ada Shannon] Yes, and the paper doesn't hide it. ZeRO-Infinity, by Rajbhandari and colleagues at Microsoft in 2021, streamed weights layer by layer. LayerKV, from Xiong and colleagues in 2024, did layer-wise KV management. This applies that pattern to KV. The result is that prefill keeps about four seventy-eighths of the K/V data, roughly 5 percent. KV preallocation drops from 36.75 to 9.04 gigabytes.

32
00:07:01,606 --> 00:07:03,138
[Hal Turing] And the freed memory buys what?

33
00:07:03,138 --> 00:07:30,352
[Dr. Ada Shannon] Bigger chunks. The maximum batched tokens per forward pass goes from 16,384 to 131,072. That matters for reading the prefill gain. At the same chunk size, offloading actually raises time to first token, because the reused prefix gets reloaded on every forward pass. The win comes from using the saved memory to run eight times larger chunks. A baseline tuned differently would narrow it.

34
00:07:30,352 --> 00:07:35,042
[Hal Turing] Now decode. The hot cache is in HBM, everything else in DRAM. What happens each step?

35
00:07:35,042 --> 00:08:00,492
[Dr. Ada Shannon] Five steps per layer. Find which selected tokens miss the cache. Merge the misses across the four speculative positions. Pick tokens to evict, one for each miss. Update the index. Then copy the misses from host to device. The catch is that a bigger hot cache means fewer misses, but older indices mapped slot to token. Checking a hit meant scanning the whole cache, so management cost grew with cache size.

36
00:08:00,492 --> 00:08:02,953
[Hal Turing] So you can't win by just making the cache bigger.

37
00:08:02,953 --> 00:08:19,718
[Dr. Ada Shannon] Not with the old index. The fix is cache_slots, which flips the direction. It's one integer array per request per layer, indexed by token, holding the HBM slot or minus one. A hit check is one lookup. Cost now scales with the misses, not the cache.

38
00:08:19,718 --> 00:08:26,359
[Hal Turing] But you still need those lookups for every selected token after top-k. Doesn't that cost k extra memory reads?

39
00:08:26,359 --> 00:08:53,851
[Dr. Ada Shannon] That's the LIM kernel's job. Before top-k, it packs each candidate's token index and slot value into a single integer, and they ride through the selection together. After top-k, decode them, and a minus one slot means a miss. The lookup comes free with the selection. It adds about 106 to 232 microseconds per kernel. Only 21 of the 78 layers have one, so around 3.4 milliseconds out of roughly 100 milliseconds per step.

40
00:08:53,851 --> 00:08:56,870
[Hal Turing] Then the copy itself. Why can't it hide behind compute?

41
00:08:56,870 --> 00:09:37,551
[Dr. Ada Shannon] Because it can't start until top-k finishes, and attention can't start until the copy finishes. ESS, from Chen and colleagues in 2025, split attention into a hit kernel and a miss kernel. CopySFA fuses the copy into the attention kernel instead. Attention works on four tiles, and only one tile touches host memory. The others do their math meanwhile. It also saves a pass over memory, three instead of four. This is the same instinct as FlashAttention-3, from Shah and colleagues in 2024, which overlaps loads and compute on NVIDIA hardware. Here it's done inside the kernel on Ascend.

42
00:09:37,551 --> 00:09:38,898
[Hal Turing] How much does it hide?

43
00:09:38,898 --> 00:09:49,765
[Dr. Ada Shannon] Overlap reaches 90 to 98 percent at batch 16 or more. But end-to-end the kernel is only 3 to 27 percent faster than serial copy then attend. It's a small gain.

44
00:09:49,765 --> 00:09:53,619
[Hal Turing] And policy versus capacity? Did they test a smarter eviction rule?

45
00:09:53,619 --> 00:10:17,768
[Dr. Ada Shannon] In simulation, on 20 requests of 66,000 to 131,000 tokens. Doubling the cache from 8,192 to 16,384 tokens cuts misses 43 to 49 percent. The best policy beats their rule by only another 18 to 20 percent. So capacity dominates. The limit is that this is shorter than the 256k-plus cases in the main table.

46
00:10:17,768 --> 00:10:19,533
[Hal Turing] Fine. What did the full system do?

47
00:10:19,533 --> 00:10:54,595
[Dr. Ada Shannon] Setup first: Ascend 910C, GLM-5.2 with speculative decoding, 90 percent prefix hit, a burst of concurrent requests. At 128k, the offloaded decode node produced about 3,700 output tokens per second versus about 1,900 for prefill-only offloading. That's the 1.91 times. It ran 6 requests per rank instead of 2. At 64k, about two-thirds more output with 12 requests instead of 4. At 256k, only 40 percent more, with 1 request growing to 2.

48
00:10:54,595 --> 00:10:55,988
[Hal Turing] What did latency do?

49
00:10:55,988 --> 00:11:27,335
[Dr. Ada Shannon] It got worse. Token time at 64k went from about 17 to 30 milliseconds, and at 128k from 16.6 to 26. At 1M the offloaded run sits at 34 milliseconds. But 512k and 1M are only servable with both offloads on, at 661 and 469 output tokens per second, at batch 1. On the prefill side, the gains are 5 to 9 percent at 64k and 128k, and 38 percent at 256k. No repeat runs or variance are reported.

50
00:11:27,335 --> 00:11:29,378
[Hal Turing] So the bigger batch is carrying the headline.

51
00:11:29,378 --> 00:11:38,480
[Dr. Ada Shannon] That's the hypothesis, and we can test it. Back to the open question: when the paper says 1.91 times, what is that a ratio of?

52
00:11:38,480 --> 00:11:39,595
[Hal Turing] Right, so what is it?

53
00:11:39,595 --> 00:11:57,567
[Dr. Ada Shannon] Throughput at two different operating points, not at equal latency. The 1.91 is the 128k decode row. The conclusion uses the 64k row, which is 1.68. And the 1.38 prefill number is the best row, at 256k.

54
00:11:57,567 --> 00:11:59,146
[Hal Turing] And the 30 millisecond budget?

55
00:11:59,146 --> 00:12:22,691
[Dr. Ada Shannon] It isn't held uniformly. The prefill-only baseline runs around 16 to 17 milliseconds per token. The fully offloaded runs sit between about 23 and 30, with 64k at 30.2, just over, and 1M at 34.1. So users get slower tokens in exchange for more concurrent users, and there's no latency-throughput curve to show where each system would land at the same latency.

56
00:12:22,691 --> 00:12:24,828
[Hal Turing] Why is the baseline's batch so small?

57
00:12:24,828 --> 00:12:47,630
[Dr. Ada Shannon] HBM. About 28 gigabytes per die for KV gives a batch of 4 at 64k, 2 at 128k and 1 at 256k. The paper says baseline throughput only saturates near batch 32. So the baseline is starved, and the headline measures how much batch offloading unlocks in that starved regime. That's real, but it's a regime claim.

58
00:12:47,630 --> 00:12:50,369
[Hal Turing] The equal-batch control is the cleaner test, then.

59
00:12:50,369 --> 00:13:19,302
[Dr. Ada Shannon] Yes, and it's the most useful measurement in the paper. At batch 4, offloading makes each decode step about 24 percent slower, 15 milliseconds on top of roughly 64. That's the inherent cost, and it buys nothing at that batch. In production, tripling the batch to 12 makes steps about 1.8 times longer. Three times the requests at 1.8 times the step time is roughly 1.7 times the work per second. It's also the only equal-batch point shown.

60
00:13:19,302 --> 00:13:20,741
[Hal Turing] Where does the extra time go?

61
00:13:20,741 --> 00:13:53,899
[Dr. Ada Shannon] Mostly not data movement. The largest piece is synchronization wait, 42.5 milliseconds at batch 12. Data-parallel ranks see different miss counts, and the mixture-of-experts dispatch makes everyone wait for the slowest. The authors call that deployment imbalance. But it comes with offloading and grows with batch, so I'd count it as a cost. They offer no mitigation. One small thing: the step-time increase in the breakdown figure is 50.6 milliseconds, while the table implies about 47. The paper doesn't explain the gap.

62
00:13:53,899 --> 00:13:55,339
[Hal Turing] What would settle the headline?

63
00:13:55,339 --> 00:14:17,816
[Dr. Ada Shannon] Both systems on a latency-throughput curve. A baseline given more dies, a different parallelism split, or a larger prefill chunk. Offloaded runs at matched latency. Repeat runs, since none report variance. And an output-equality check. They claim offloading is exact because it only changes data placement, but they show no logit or accuracy comparison.

64
00:14:17,816 --> 00:14:23,064
[Hal Turing] Is the fused kernel at risk there? It packs the slot into the same integer as the token index.

65
00:14:23,064 --> 00:14:35,649
[Dr. Ada Shannon] That's the right worry. If scores tie or nearly tie, packing could change which token wins. The paper doesn't show the selected set matches the native indexer's. It may well, but it's asserted.

66
00:14:35,649 --> 00:14:36,903
[Hal Turing] So what's actually new?

67
00:14:36,903 --> 00:15:00,076
[Dr. Ada Shannon] The rolling buffers are a known pattern, as with ZeRO-Infinity and LayerKV. Fusing load and compute echoes FlashAttention-3, but CopySFA does it inside one kernel on Ascend, and the kernel speedup is modest. The reverse index plus folding the miss lookup into top-k is the most original piece. It makes management cost scale with misses instead of cache size.

68
00:15:00,076 --> 00:15:01,284
[Hal Turing] And the competitors?

69
00:15:01,284 --> 00:15:32,445
[Dr. Ada Shannon] ESS, HiSparse, ECHO and AVSG are the real comparators, and none is run end to end. The comparison is an offline simulation on 20 coding traces. In it, the paper's cache holds 14,336 tokens and the HiSparse-style LRU holds 8,192. Bigger cache alone cuts misses 43 to 49 percent, so I can't tell how much comes from the design and how much from spending more HBM. LRU with the same cache plus the reverse index would separate them.

70
00:15:32,445 --> 00:15:34,906
[Hal Turing] Is the miss behavior at a million tokens known?

71
00:15:34,906 --> 00:15:56,222
[Dr. Ada Shannon] No. The simulation covers 66k to 131k tokens. At 1M, a cache of roughly 16k tokens is under 2 percent of the sequence, and nothing shows misses stay near 300 per request. The 512k and 1M rows run at batch 1 with no baseline, so they show those lengths become servable, not that they're fast.

72
00:15:56,222 --> 00:15:57,151
[Hal Turing] Who should care?

73
00:15:57,151 --> 00:16:24,225
[Dr. Ada Shannon] Anyone serving long-context decode on a natively sparse model where HBM caps the batch. Prefix-heavy agent workloads fit well, and ultra-long requests that otherwise can't run at all. Short contexts, or hardware with HBM to spare, may see little. For dense-attention models only the prefill half applies. Also, it all depends on Ascend's host links, and the kernels are Ascend-specific. Nothing here shows GPUs.

74
00:16:24,225 --> 00:16:27,940
[Hal Turing] What about the discussion's claim that chips could ship with less HBM?

75
00:16:27,940 --> 00:16:42,105
[Dr. Ada Shannon] Argued, not measured. The paper never quantifies the host DRAM capacity or bandwidth needed to replace it. The same goes for the claim that native sparse attention keeps capability while compression and linear attention lose it.

76
00:16:42,105 --> 00:16:43,359
[Hal Turing] Where does it go next?

77
00:16:43,359 --> 00:16:56,315
[Dr. Ada Shannon] Miss-aware scheduling across ranks to cut the sync wait. Hot-cache sizing that adapts to the workload. A storage tier below DRAM. And a GPU port with head-to-head runs against ECHO and HiSparse.

78
00:16:56,315 --> 00:17:22,507
[Hal Turing] So the takeaway: exact, host-backed KV offloading plausibly makes 512k and 1M requests servable, and gives roughly 1.4 to 1.9 times more decode throughput at long context on this stack. The boundary is that those gains come from an HBM-starved baseline at higher per-token latency, so they aren't a free speedup. Thanks for listening, everyone.

79
00:17:22,507 --> 00:17:23,854
[Dr. Ada Shannon] Thanks, everyone. Goodbye.