1
00:00:01,000 --> 00:00:07,362
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,362 --> 00:00:39,870
[Hal Turing] Today we're looking at FlatCake: A Flat KVCache Storage Architecture for LLM Serving, from first author Xin Huang and colleagues at Fudan University, Peking University, MetaX Technology, the Institute of Computing Technology at the Chinese Academy of Sciences, and an independent researcher. It's an arXiv paper from October 2026. The claim is that the standard memory hierarchy for cached model state is the wrong shape, and that you should drop the hierarchy and place each block once.

3
00:00:39,870 --> 00:00:55,474
[Dr. Ada Shannon] The core question is whether a write-once, no-migration layout beats putting DRAM in front of an SSD. The second question matters just as much: how much of that claim do the experiments actually test? We'll come back to that. First, the background.

4
00:00:55,474 --> 00:00:57,749
[Hal Turing] Start with the cache itself. What's stored?

5
00:00:57,749 --> 00:01:22,455
[Dr. Ada Shannon] During prefill, the model computes key and value tensors for every input token. That's the KV cache. If a later request shares the same prefix, you can load those tensors and skip the compute. The result is exact, not approximate. The metric to watch is time to first token, TTFT: how long the user waits before output starts. Lower is better, and for long contexts prefill dominates it.

6
00:01:22,455 --> 00:01:24,498
[Hal Turing] So a hit saves time. What's the catch?

7
00:01:24,498 --> 00:01:51,991
[Dr. Ada Shannon] A stored block only helps if it reaches GPU memory before prefill starts. The paper cares about that delivered hit rate, where higher is better. A block that sits on disk but arrives late gets recomputed anyway. And the volume is large. Agentic sessions in one trace average about 219 thousand input tokens per request, which means multiple gigabytes per session. That overflows GPU memory, then host DRAM, so the SSD is now on the critical read path.

8
00:01:51,991 --> 00:01:53,802
[Hal Turing] And the standard answer is tiering.

9
00:01:53,802 --> 00:02:24,406
[Dr. Ada Shannon] Yes. GPU memory, then host DRAM, then NVMe SSD. New blocks land in DRAM, an LRU-style policy pushes cold ones down to SSD, and an SSD hit gets staged back up through DRAM buffers. Mooncake from Moonshot AI, LMCache, and SGLang's HiCache all work roughly this way. That design rests on two premises. DRAM filters out cold data, and DRAM staging reassembles scattered SSD reads into something efficient.

10
00:02:24,406 --> 00:02:27,007
[Hal Turing] Mooncake keeps coming up. Is it the same Mooncake?

11
00:02:27,007 --> 00:02:47,115
[Dr. Ada Shannon] Same one. The 2024 paper by Ruoyu Qin and colleagues at Moonshot AI and Tsinghua is both the main baseline and the source of several traces. Its scheduler already treats the KV pool as a distributed store. So when we hear flat versus tiered, we should read it against Mooncake's real admission policies, not a textbook LRU.

12
00:02:47,115 --> 00:02:48,694
[Hal Turing] Where do the two premises break?

13
00:02:48,694 --> 00:03:27,286
[Dr. Ada Shannon] The paper's Table 1 says reuse is bimodal. In seven conversational and API traces, from Mooncake and from Alibaba's Qwen Bailian platform, between 53 and 94 percent of blocks are used exactly once, and reuse distances are long. A DRAM tier that admits everything fills with one-shot blocks and flushes out the reusable ones before they come back. Then there's the Claude Code agentic trace. There, 83.4 percent of blocks are reused, with an ideal hit rate of 98.3 percent. But sessions are so big that a few concurrent ones exceed DRAM. The paper's point is that DRAM filtering fails in both cases.

14
00:03:27,286 --> 00:03:33,973
[Hal Turing] Opposite workloads, same verdict. That's a strong claim. I'd want to see the second premise fail too, the staging part.

15
00:03:33,973 --> 00:03:57,657
[Dr. Ada Shannon] That's the second observation, and it's a causal chain. Staging memory scales with context length times concurrency. Under load the host buffers run out, and the serving engine revokes prefetches and recomputes blocks that were sitting on SSD. Separately, blocks get written in eviction order, so one session's blocks end up scattered across the drive. Restoring a session then takes on the order of layers times blocks small reads.

16
00:03:57,657 --> 00:04:02,069
[Hal Turing] Doesn't GPUDirect Storage fix the small-read problem? It skips the host entirely.

17
00:04:02,069 --> 00:04:24,825
[Dr. Ada Shannon] It fixes the bounce buffer, not the scatter. GDS lets the NVMe drive DMA straight into GPU memory, so no host staging copy. But an NVMe read command needs a contiguous range on disk. Scattered blocks still mean many small commands, and GDS performs well on large sequential I/O and poorly on small ones. So transport and layout are separate problems.

18
00:04:24,825 --> 00:04:26,125
[Hal Turing] So what's the proposal?

19
00:04:26,125 --> 00:04:51,156
[Dr. Ada Shannon] Here's the thesis, and it's the paper's bet, not a result. KV blocks are write-once, immutable, and reused at session scope. So decide the medium when the block is created, DRAM or SSD, and never migrate it. DRAM and SSD become peers behind one address space. On the SSD, lay each session out contiguously, layer by layer, so a restore becomes a few large reads.

20
00:04:51,156 --> 00:04:58,354
[Hal Turing] The kernel-person version: it's allocating a contiguous extent up front instead of living with a fragmented filesystem.

21
00:04:58,354 --> 00:05:06,992
[Dr. Ada Shannon] That mapping works. The cost is also visible: placement has to be right at write time, because nothing moves it later. A wrong guess sticks.

22
00:05:06,992 --> 00:05:10,011
[Hal Turing] That sounds like the risk to watch. What happens next?

23
00:05:10,011 --> 00:05:19,299
[Dr. Ada Shannon] Next we'll cover the design, how it decides placement and issues the I/O, and then the measurements on a Claude Code trace. After that we'll ask what those measurements leave open.

24
00:05:19,299 --> 00:05:24,128
[Hal Turing] Start with placement. If nothing migrates, who decides where a block lives, and when?

25
00:05:24,128 --> 00:05:53,246
[Dr. Ada Shannon] A component called FC-GVA decides, at the moment the block is created. It uses signals the engine already has. A prefix shared by several requests, or a short chat turn after a short reply, goes to DRAM. A long agentic session, or a long reply, goes to SSD. A first turn, or any write that arrives when DRAM is full, also goes to SSD. The thresholds are hand-tuned. The paper calls them tau and L_agent, and it gives no concrete values for either.

26
00:05:53,246 --> 00:05:55,708
[Hal Turing] And once it's placed, finding it again is cheap?

27
00:05:55,708 --> 00:06:14,376
[Dr. Ada Shannon] Yes, because the location is baked into the address. Each block gets a 64-bit address: four bits for the medium, twelve for the node, forty-eight for the byte offset. One hash lookup gives you the address, and the medium bits at the top tell you whether to use a DRAM copy or an SSD read. There's no per-tier lookup.

28
00:06:14,376 --> 00:06:18,092
[Hal Turing] What about freeing space? Tiered systems write dirty data back.

29
00:06:18,092 --> 00:06:38,525
[Dr. Ada Shannon] Blocks are write-once, so there's nothing dirty to write back. When the SSD fills, FlatCake frees the oldest session extent that nobody is reading. The contrast is Mooncake, which recycles fixed-size buckets in FIFO order. One bucket can hold pieces of many sessions, so evicting it can invalidate several live prefixes at once.

30
00:06:38,525 --> 00:06:41,497
[Hal Turing] Now the read side. How does one session become one big read?

31
00:06:41,497 --> 00:07:14,888
[Dr. Ada Shannon] That's FC-CIO, the I/O engine. Each SSD-bound session gets a pre-allocated single-extent file, and each turn's blocks are written layer-major. All the blocks for layer zero sit together, then layer one, and so on. Prefill consumes the cache layer by layer, so a restore reads contiguous ranges. At read time it sorts the block requests by file and offset, merges neighbors, and bridges gaps under 64 KiB. That takes the read count from layers times blocks down to roughly one per layer.

32
00:07:14,888 --> 00:07:18,231
[Hal Turing] But those reads land in GPU memory, which is already crowded.

33
00:07:18,231 --> 00:07:38,061
[Dr. Ada Shannon] Right, that's the cost. The data streams into a small double-buffered window in HBM, sized at two layers per stream. One layer is being used while the next one loads, then a kernel scatters the pages into the paged KV pool. It also assumes strict prefix reuse, because the whole layout is organized around a session's prefix.

34
00:07:38,061 --> 00:07:39,222
[Hal Turing] And the scheduler?

35
00:07:39,222 --> 00:08:12,427
[Dr. Ada Shannon] FC-MAS keeps two queues. Requests whose blocks are all in DRAM go straight to the GPU. Requests that touch SSD wait in a preparing queue while prefetches run in the background. That queue uses a highest-response-ratio rule, so small restores go first and large ones can't wait forever. When a prefetch finishes, the request re-enters by its original arrival time. Removing FC-MAS costs about 10 to 11 percent in mean TTFT and 27 percent at P99, so it matters. It's also the only component ablated on the agentic trace.

36
00:08:12,427 --> 00:08:14,284
[Hal Turing] Give me the setup before the numbers.

37
00:08:14,284 --> 00:08:39,873
[Dr. Ada Shannon] Eight B200 GPUs, a 1 TiB DRAM budget and a 4 TiB SSD budget for every system, running inside SGLang. The model is DeepSeek-V4-Flash. The trace is the Claude Code one, about 99,000 requests across 393 sessions, replayed from 0.1 to 1.0 requests per second with no warm-up. The baselines are DRAM-only HiCache and Mooncake with host staging.

38
00:08:39,873 --> 00:08:41,266
[Hal Turing] So what happens as load rises?

39
00:08:41,266 --> 00:09:15,260
[Dr. Ada Shannon] The delivered hit rate is the story. Higher is better. FlatCake stays around 94 percent across the whole range, close to the trace's reuse ceiling of about 94.7. Mooncake delivers about 88 percent at the lightest load, so it works there. By 0.5 requests per second it's down to about 64, meaning only about 64 of every 100 reusable blocks reach the GPU. The paper blames staging buffers running out, which makes SGLang revoke prefetches and recompute blocks that were already on SSD.

40
00:09:15,260 --> 00:09:16,792
[Hal Turing] And that shows up as wait time?

41
00:09:16,792 --> 00:09:40,244
[Dr. Ada Shannon] At peak load, the mean wait for a first token is about 7.6 seconds, against 12.2 for Mooncake and 17.5 for DRAM-only. Throughput is 98 thousand tokens per second against 62 thousand for Mooncake. Both are about 1.6 times better. One caution: the headline pits Mooncake's worst point against FlatCake's best, and the paper reports no variance.

42
00:09:40,244 --> 00:09:43,310
[Hal Turing] Does the mechanism evidence support the staging explanation?

43
00:09:43,310 --> 00:10:04,254
[Dr. Ada Shannon] It does. Mooncake reads six and a half times more bytes from the SSD than it delivers, from revoked prefetches and background promotion. FlatCake's ratio is one. Mean prefetch latency is 1.6 seconds versus 9.5, and per-worker SSD bandwidth is about 5 GB/s versus 2.7.

44
00:10:04,254 --> 00:10:08,898
[Hal Turing] Still, that run changes layout and removes staging together. Can they be separated?

45
00:10:08,898 --> 00:10:39,780
[Dr. Ada Shannon] That's the Llama-3.1-70B sweep on four A100s, and it's the cleanest test in the paper. It adds Mooncake-GDS, which turns on direct GPU reads but keeps eviction-ordered layout. At 16K tokens it ties FlatCake, 2.31 seconds against 2.32. At 64K, FlatCake's mean TTFT is 3.74 seconds against 6.60, about 1.8 times better. So GDS alone doesn't fix it. The layout is doing the work. The caveat is that this is a context-length sweep, not a load test.

46
00:10:39,780 --> 00:10:44,517
[Hal Turing] One more thing. In the DeepSeek run, how much of the work did DRAM actually do?

47
00:10:44,517 --> 00:10:57,892
[Dr. Ada Shannon] Not much. SSD supplied about 53 to 69 percent of hits, depending on load, and DRAM only 8 to 12. So this run mostly exercises the SSD half of the placement table. The DRAM rules barely got tested.

48
00:10:57,892 --> 00:11:02,072
[Hal Turing] The paper motivates all this with eight traces. How many were actually replayed?

49
00:11:02,072 --> 00:11:26,871
[Dr. Ada Shannon] One. The seven low-reuse traces appear only in the characterization. The end-to-end evaluation is the Claude Code trace on one model. The paper doesn't say why, and there's no limitations section to ask. So what's untested is exactly where DRAM matters: shared prompts and short chat turns under load, mixed workloads, any case where a DRAM hit is worth more than an SSD hit.

50
00:11:26,871 --> 00:11:31,282
[Hal Turing] And the agentic trace is the case where nearly everything belongs on SSD anyway.

51
00:11:31,282 --> 00:11:44,982
[Dr. Ada Shannon] Right, which favors a design with no DRAM staging almost by construction. That's my main reservation. The evidence is strong for session-contiguous layout on long agentic sessions. It's much thinner for the general claim that flat beats tiered.

52
00:11:44,982 --> 00:11:48,140
[Hal Turing] What about the placement rule itself? You said it was hand-tuned.

53
00:11:48,140 --> 00:12:14,564
[Dr. Ada Shannon] Yes. The tau threshold is borrowed from Bidaw, a FAST 2026 paper by Shipeng Hu and colleagues on workload-aware tiered KV caching, and the agentic-length threshold has no stated value. There's no sensitivity sweep, no comparison against oracle placement, and no misprediction rate. Blocks never move, so a session that starts as a short chat and turns agentic could have its history stuck on the wrong medium. The paper doesn't say how that's handled.

54
00:12:14,564 --> 00:12:19,301
[Hal Turing] But they argue a mistake is cheap, since an SSD hit is just a coalesced read.

55
00:12:19,301 --> 00:12:33,512
[Dr. Ada Shannon] That's a plausible argument, and it's only an argument. The one supporting measurement is long-context. A short chat turn read from SSD still pays the bandwidth gap and the latency, roughly five times slower than DRAM by their own table, and nobody measured it.

56
00:12:33,512 --> 00:12:36,391
[Hal Turing] Then the baseline. Is Mooncake a fair opponent here?

57
00:12:36,391 --> 00:12:58,775
[Dr. Ada Shannon] The collapse is blamed on host staging buffers running out, so SGLang revokes prefetches and recomputes. That's a property of one engine's bounded budget, which is a configuration. The paper doesn't vary it. Would a bigger pool, queued instead of revoked prefetches, or tighter admission control close much of the gap? We don't know. And remember Mooncake held 88 percent at low load.

58
00:12:58,775 --> 00:13:03,094
[Hal Turing] And the cleanest layout test, Mooncake with GDS, only ran on Llama?

59
00:13:03,094 --> 00:13:25,339
[Dr. Ada Shannon] Yes, and that was a context-length sweep, not concurrent sessions. So it isolates layout from transport, but not under the contention where staging exhaustion bites. Add that no variance is reported, and that the 94.7 percent replay ceiling and the 98.3 percent ideal in Table 1 are never reconciled. Block size is a likely cause, but that's my guess.

60
00:13:25,339 --> 00:13:27,336
[Hal Turing] Is anything missing from the related work?

61
00:13:27,336 --> 00:14:03,373
[Dr. Ada Shannon] Bidaw is a workload-aware tiered alternative, and it isn't compared. That's the most direct competitor to the placement idea. Then there are complements. CacheGen, by Yuhan Liu and colleagues, SIGCOMM 2024, compresses stored KV lossily. In principle you compress the extents and still coalesce reads, but that needs testing. CacheBlend, by Jiayi Yao and colleagues, 2025, reuses KV that isn't a prefix. FlatCake's layout assumes strict prefix reuse, so the same document at a different position, or in another session, is outside what it captures.

62
00:14:03,373 --> 00:14:04,534
[Hal Turing] So who should care?

63
00:14:04,534 --> 00:14:25,571
[Dr. Ada Shannon] Operators with long agentic sessions, big SSD arrays, and a GDS-capable stack. Be cautious if your fleet is chat-heavy or mixed. The layout idea may also port to tiered systems, and the Mooncake-GDS result shows that GDS alone doesn't deliver it. I won't put cost or capacity numbers on it, because the paper doesn't.

64
00:14:25,571 --> 00:14:28,869
[Hal Turing] The testbed was a high-end box, too. Eight drives in RAID-0.

65
00:14:28,869 --> 00:14:33,234
[Dr. Ada Shannon] Right. Whether it holds with one or two commodity drives, the paper doesn't test.

66
00:14:33,234 --> 00:14:34,813
[Hal Turing] What experiments would settle it?

67
00:14:34,813 --> 00:14:53,296
[Dr. Ada Shannon] Replay the seven low-reuse traces and mixed traces. Report sensitivity to the two thresholds and the gap to oracle placement. Test sessions that change class. Add a tuned Mooncake and a Bidaw baseline. Try compression and non-prefix reuse on top of the layout. And report variance.

68
00:14:53,296 --> 00:14:54,689
[Hal Turing] So where does that leave us?

69
00:14:54,689 --> 00:15:12,383
[Dr. Ada Shannon] On one agentic trace and one model, fixing the SSD layout and removing DRAM staging keeps cached blocks delivered under load. That's solid for that regime. It doesn't show the full placement policy is right across workloads, and it doesn't prove flat storage beats a well-tuned tiered system in general.

70
00:15:12,383 --> 00:15:16,562
[Hal Turing] That's a fair boundary. Thanks for listening, everyone. See you next time.