1
00:00:01,000 --> 00:00:05,783
[Hal Turing] Thanks for tuning in! I'm your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:05,783 --> 00:00:32,114
[Hal Turing] Today's paper is "HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving." The first author is Zhuoran Li, from the School of Integrated Circuits at Peking University, with Fudan University. It was posted to arXiv in September 2026. The headline claim is a strange one: the faster device yields a slower system.

3
00:00:32,114 --> 00:00:47,207
[Dr. Ada Shannon] They put a faster flash tier under a KV-offloading stack that was built for SSDs, and serving gets slower. The research question is when a faster far tier can pay off at all. Their answer is three conditions, and they argue that transient KV cache violates all three.

4
00:00:47,207 --> 00:00:58,260
[Hal Turing] Let's start with the KV cache, since everything hangs on it. For every token a request keeps, each transformer layer stores a key vector and a value vector. That's the cache.

5
00:00:58,260 --> 00:01:21,759
[Dr. Ada Shannon] And it's what limits concurrency. Per request, the size is two, for key and value, times tokens, layers, KV heads, head dimension, and bytes per element. Weights are fixed, but the cache grows with every conversation you hold open. Prefill is the compute-heavy pass over the prompt that builds the cache. Decode then generates one token at a time and reads that cache constantly.

6
00:01:21,759 --> 00:01:27,378
[Hal Turing] So reuse matters. If a new request shares a prefix with an old one, you skip prefill for that part.

7
00:01:27,378 --> 00:01:47,626
[Dr. Ada Shannon] Right. Multi-turn chat, agent loops, shared system prompts, and RAG all do this. But holding every prefix in GPU memory doesn't scale. The usual answer is paging, from PagedAttention by Kwon and colleagues at UC Berkeley in 2023, and then pooling KV across memory tiers.

8
00:01:47,626 --> 00:01:49,901
[Hal Turing] And the stack they test is Mooncake-style?

9
00:01:49,901 --> 00:02:23,245
[Dr. Ada Shannon] Yes. Mooncake is the KV-centric serving system from Moonshot AI. New KV lands in host DRAM. Past a watermark, a background policy flushes it to SSD. A master node tracks where objects live and promotes hot blocks back. When a matching request arrives, the blocks are restored to the GPU and only the missing suffix gets computed. The SSD side is strong now, with io_uring, GPU Direct Storage, and Tutti cutting host overhead. The paper keeps all that policy unchanged and swaps only the backing tier.

10
00:02:23,245 --> 00:02:38,245
[Hal Turing] Two terms I'll use. The near tier is GPU-local memory, HBM or GDDR7, holding weights and active KV. The far tier is the bigger, slower store for offloaded KV. Now, what is High-Bandwidth Flash?

11
00:02:38,245 --> 00:03:06,991
[Dr. Ada Shannon] Stacked 3D NAND dies on a base die, wired to the accelerator through a wide package-local interface. The paper's base-die spec uses AXI over UCIe, not NVMe. You get flash capacity: a 512 GB device is about 21 times the density of a 24 GB HBM stack. You also get high aggregate bandwidth from massive parallelism. What you don't get is DRAM-like latency. Reads still take microseconds.

12
00:03:06,991 --> 00:03:08,570
[Hal Turing] So wide, not fast per access.

13
00:03:08,570 --> 00:03:26,543
[Dr. Ada Shannon] Exactly. And writes are where NAND bites. It can't overwrite in place. It programs whole pages, four kilobytes in the spec they use, and erases larger blocks later. Every program spends a finite program/erase budget, heat speeds up wear, and writes cost far more energy and endurance than reads.

14
00:03:26,543 --> 00:03:28,214
[Hal Turing] Where does the roadmap put this?

15
00:03:28,214 --> 00:03:52,874
[Dr. Ada Shannon] The KAIST HBF roadmap has two packagings. HBF-1, around 2028, fills the interposer with HBF and demotes the near tier to off-interposer GDDR7. HBF-2, around 2030, splits the interposer sites between HBM and HBF. Modeled on an H100, both leave 48 GB of near tier, against 96 GB of HBM for the SSD baseline.

16
00:03:52,874 --> 00:03:56,311
[Hal Turing] So the flash isn't free. You pay for it in GPU memory.

17
00:03:56,311 --> 00:04:05,227
[Dr. Ada Shannon] That's the thing to track through the rest of this conversation. The package trade rides along with the faster medium, so the comparison isn't just flash versus SSD.

18
00:04:05,227 --> 00:04:07,131
[Hal Turing] Fine. What are the three conditions?

19
00:04:07,131 --> 00:04:37,874
[Dr. Ada Shannon] C1: exposed read I/O must be the bottleneck. C2: reads must outweigh writes. C3: the delivered bandwidth must be sustainable, thermally and for endurance. Two quantities make this concrete. The exposed fraction, f, is the share of request time spent in far-tier I/O that isn't overlapped with other work. Say a request takes ten seconds and half a second is stalled on a restore. Then f is five percent, and even an infinitely fast device can only trim that half second.

20
00:04:37,874 --> 00:04:39,871
[Hal Turing] So f caps what any media upgrade can buy.

21
00:04:39,871 --> 00:04:58,354
[Dr. Ada Shannon] Yes. The second quantity is rho, the useful bytes read per byte written to flash. A write is paid for immediately, but the reads that repay it are speculative. If you write ten gigabytes of KV and only five ever come back, rho is a half. The paper argues the break-even is at least about one.

22
00:04:58,354 --> 00:05:00,862
[Hal Turing] Everything here is simulated, right? No hardware.

23
00:05:00,862 --> 00:05:09,918
[Dr. Ada Shannon] The serving side is a trace-driven simulator called TokenSim, extended with the tiers. Thermals come from a stacked-chip solver, 3D-ICE. It's all modeled.

24
00:05:09,918 --> 00:05:14,469
[Hal Turing] One contrast first. Other HBF proposals don't do a drop-in swap, do they?

25
00:05:14,469 --> 00:05:30,166
[Dr. Ada Shannon] Right. H3 and FlashAccel keep the HBM and add HBF alongside it, for read-mostly data like weights and shared prefixes. That's an additive tier. This paper tests the other design, a swap under an unmodified SSD-style runtime.

26
00:05:30,166 --> 00:05:32,766
[Hal Turing] Let's get into how they ran this. What's the setup?

27
00:05:32,766 --> 00:06:06,342
[Dr. Ada Shannon] The workload is four two-hour production traces from Alibaba's Qwen-Bailian service: traceA, traceB, coder, and thinking. Five models, dense and mixture-of-experts, on modeled H100 and B200 eight-GPU systems with tensor parallelism 8. The SSD baseline is KIOXIA CM7-V drives, and KV is hashed in 16-token blocks. Offered load runs from 0.05 to 32 requests per second. The baseline is the conventional Mooncake connector in the simulator, not a tuned SSD path.

28
00:06:06,342 --> 00:06:08,386
[Hal Turing] And the comparison is capacity-matched?

29
00:06:08,386 --> 00:06:24,872
[Dr. Ada Shannon] Yes. HBF-1 is paired with a 24-terabyte SSD pool, HBF-2 with 12 terabytes. They also run a media-only sweep, holding the GPU fixed and varying just HBF latency, and a complete-architecture comparison where the near tier shrinks too.

30
00:06:24,872 --> 00:06:25,940
[Hal Turing] So what happens?

31
00:06:25,940 --> 00:07:02,117
[Dr. Ada Shannon] HBF loses on every request metric. Average end-to-end latency is two to five and a half times worse. Delivered throughput drops 4 to 34 percent. Maximum SLO goodput, the highest request rate that still meets the latency targets, falls by 1.1 to 2.7 times. H100 is the severe end, B200 the mild end. One example: GLM on traceA, B200, 32 requests per second. HBF-1's total connector time is about a third of the SSD's. Yet the average request takes about two and a half times longer.

32
00:07:02,117 --> 00:07:05,182
[Hal Turing] That's the paradox from the title. Where did the time go?

33
00:07:05,182 --> 00:07:38,711
[Dr. Ada Shannon] Into the near tier, which decides how many requests get admitted. Shrink it and queues build, and queues dominate latency. One caveat: neither comparison isolates one variable. HBF versus SSD halves near-tier capacity and bandwidth together. HBF-1 versus HBF-2 holds capacity fixed but swaps GDDR7 for HBM3e. Still, HBF-1 is 16 to 64 percent slower than HBF-2 despite twice the flash capacity and bandwidth, so more flash doesn't repair it.

34
00:07:38,711 --> 00:07:42,055
[Hal Turing] Now the exposed fraction, f. How do they get a number for it?

35
00:07:42,055 --> 00:08:13,541
[Dr. Ada Shannon] From the cost-benefit relation, Eq. 6. Faster flash improves only one term, critical-path I/O. Everything else HBF changes is a cost. Eq. 7 says a medium k times faster changes latency by at most f times one minus one over k. Then they sweep HBF latency by 3.75 times, from 30 and 300 microseconds read and write down to 8 and 80. Mean latency moves under one percent, and inverting Eq. 7 gives f of about one percent.

36
00:08:13,541 --> 00:08:16,281
[Hal Turing] So a zero-latency device removes one percent at most.

37
00:08:16,281 --> 00:08:27,055
[Dr. Ada Shannon] By that estimate, yes. The same sweep still moves throughput about 5 percent and goodput 14 to 23 percent. So the media change registers, just not on average latency.

38
00:08:27,055 --> 00:08:33,417
[Hal Turing] One fix people propose is putting compute on the base die. If the data stays put, maybe the exposed part shrinks.

39
00:08:33,417 --> 00:09:13,124
[Dr. Ada Shannon] They test that in a favorable setup: a near-memory attention engine charged no area or power. On an HBM base die it cuts decode-attention time 42.84 percent. The same engine on HBF averages about zero, and is slower than GPU attention at 16 of 40 sweep points. Coverage is the reason. On average only 15.48 percent of KV sits in HBF, and each step pays a 12 microsecond access. Even the strong HBM engine moves decode p50 by only 0.39 percent end to end. A 43 percent operator win mostly vanishes.

40
00:09:13,124 --> 00:09:16,003
[Hal Turing] Moving to writes. What does HBF actually receive?

41
00:09:16,003 --> 00:09:49,347
[Dr. Ada Shannon] More writes than reads on every trace. The write-to-read ratio runs from 1.14 times on coder to 4.90 on traceB, about five bytes written per byte read. Rho is 0.2 to 0.88, so even the best trace is under break-even. Prefix hit rates are real, 38 to 53 percent. But reuse is concentrated. On traceB and thinking, the top ten percent of blocks supply nearly all reuse reads. Those hot blocks stay in the near tier, so flash gets the write-once cold tail.

42
00:09:49,347 --> 00:09:52,365
[Hal Turing] So the hierarchy works as designed, and that's the problem.

43
00:09:52,365 --> 00:10:17,396
[Dr. Ada Shannon] Right. And SSD write optimizations don't transfer. Coarser Mooncake batching cuts request p50 by 13.98 percent on the SSD, because it amortizes fixed per-I/O overhead. On HBF it helps only 2.69 percent. HBF has little fixed overhead to amortize, and at 16-token blocks it gets slightly worse, up 9.2 percent.

44
00:10:17,396 --> 00:10:18,557
[Hal Turing] What about heat and wear?

45
00:10:18,557 --> 00:10:42,242
[Dr. Ada Shannon] A 3D-ICE thermal model hits the 80 degree limit at 202 gigabytes per second and 53.7 watts per stack, well under interface peak. Trace writes work out to 48 to 140 terabytes per day, against budgets of 38.4 for the SSD pool and 21.7 for HBF. So HBF lasts 0.56 times as long as the SSD pool.

46
00:10:42,242 --> 00:10:45,260
[Hal Turing] Bigger blocks help the SSD. Do they help HBF's wear?

47
00:10:45,260 --> 00:10:58,542
[Dr. Ada Shannon] The reverse. Going from 16 to 512-token blocks raises HBF writes from 188 to 258 terabytes, 1.38 times more. Coarser blocks write more data that is never reused.

48
00:10:58,542 --> 00:11:07,923
[Hal Turing] That's a clean story: faster flash, slower system. Before anyone repeats the title as a law of nature, what does the evidence support?

49
00:11:07,923 --> 00:11:36,530
[Dr. Ada Shannon] A narrower claim than the title. The slowdown comes from packages that give up half the GPU's near tier, 96 gigabytes down to 48 on H100, with bandwidth dropping too. So what's established is that the roadmap's packaging trade backfires under an unmodified SSD-style runtime. What's missing is the control that keeps a full-size near tier and adds HBF on top. That is a result about halving the HBM tier, not about HBF media.

50
00:11:36,530 --> 00:11:38,480
[Hal Turing] Other papers did run that additive setup.

51
00:11:38,480 --> 00:12:07,180
[Dr. Ada Shannon] H3, by Minho Ha and colleagues in 2026, keeps HBM and puts read-only weights and shared prefixes in HBF. FlashAccel, by Xinyu Wang and colleagues the same year, also adds HBF to an HBM GPU. As I read it, both are aimed at a different question. The likely difference is additive versus substitutive packaging, but this paper never runs the additive case in its own simulator, so it doesn't reconcile the two.

52
00:12:07,180 --> 00:12:13,775
[Hal Turing] What about the one-percent exposed fraction? That number carried the 'media isn't the bottleneck' argument.

53
00:12:13,775 --> 00:12:40,756
[Dr. Ada Shannon] It's inverted from a first-order formula on a sweep covering under 4x in latency. The formula assumes f doesn't depend on load. But the paper itself reports queue instability near saturation, and the same sweep moved goodput 14 to 23 percent. If the near tier is the bottleneck, requests wait for admission, and device speed would barely show up anyway. I'd read one percent as a property of this capacity-starved system, not of the workload.

54
00:12:40,756 --> 00:12:42,335
[Hal Turing] So what experiment settles it?

55
00:12:42,335 --> 00:12:54,642
[Dr. Ada Shannon] Rerun the media sweep with a full-size near tier and across a range of loads. If f stays near one percent there, the paper's claim gets much stronger. If it climbs, the claim only describes the halved configuration.

56
00:12:54,642 --> 00:13:01,887
[Hal Turing] Then there's the baseline. The paper talks up Tutti and GPU Direct Storage, but the comparison uses the conventional Mooncake connector.

57
00:13:01,887 --> 00:13:38,017
[Dr. Ada Shannon] Right. Tutti, from Shi Qiu and colleagues in 2026, makes SSD-backed KV practical with GPU-initiated I/O, and it's cited but never run. Strata, from Zhiqiang Xie and colleagues in 2025, overlaps loading with compute. CachedAttention, from Bin Gao and colleagues in 2024, did layer-wise preloading and asynchronous saving earlier still. All of these exist to shrink exposed I/O, which is exactly f. How much the simulated path overlaps I/O determines how small f can be, and the source doesn't say.

58
00:13:38,017 --> 00:13:39,085
[Hal Turing] And LMCache?

59
00:13:39,085 --> 00:14:03,187
[Dr. Ada Shannon] LMCache, from Yuhan Liu and colleagues in 2025, has configurable connectors and admission policies, and it's where an HBF tier would plausibly be integrated. So it's the natural place to test the paper's own positive claim that reuse-aware placement helps. That claim is never run. Even a simple second-hit admission filter isn't tried, though on traceB only 6 percent of blocks are ever reused.

60
00:14:03,187 --> 00:14:07,042
[Hal Turing] The thermal and endurance numbers worry me more. They looked precise.

61
00:14:07,042 --> 00:14:34,441
[Dr. Ada Shannon] They're projections. The thermal result is a single-stack steady-state model with an author-chosen 80 degree limit. The 0.56x lifetime ratio is identical on every trace because it's just a ratio of two budgets, not a measurement. And the writes, 48 to 140 terabytes a day, exceed the SSD budget too, so the SSD pool would also wear out. That suggests the write policy is a problem for any flash, and the HBF-specific part may be overstated.

62
00:14:34,441 --> 00:14:39,828
[Hal Turing] Same question for the batching result. It depended on assuming random and sequential bandwidth are equal.

63
00:14:39,828 --> 00:15:07,507
[Dr. Ada Shannon] Partly circular, yes. If the model removes the penalty for small writes, batching can't show much benefit on HBF. The attention-offload result has a similar limit: the engine only covers the 15 percent of KV that the placement policy leaves in HBF. A design that deliberately put cold long-context KV there might change that, and the paper doesn't test it. So 'wrong lever' is a verdict on this policy, not on in-flash attention.

64
00:15:07,507 --> 00:15:09,225
[Hal Turing] Is there anything the paper does well?

65
00:15:09,225 --> 00:15:26,408
[Dr. Ada Shannon] Quite a lot. Four full two-hour production traces, five models, and the direction holds across GPUs. It labels what is sourced, swept, or modeled. What's missing is variance across runs, a load-by-load view of where HBF stops losing, and any hardware measurement.

66
00:15:26,408 --> 00:15:29,705
[Hal Turing] So what should someone building a tiered KV layer do with this?

67
00:15:29,705 --> 00:15:57,940
[Dr. Ada Shannon] Use the three conditions as a placement test before putting any package-local far tier under KV. Is exposed I/O actually on the critical path? Do reads repay writes? Can the bandwidth be sustained thermally? Then treat objects differently. Weights and widely shared prefixes look like HBF material. Transient private KV doesn't. Admission should be reuse-aware, writes budgeted, and thermals coordinated with the runtime.

68
00:15:57,940 --> 00:15:59,287
[Hal Turing] And the next experiments?

69
00:15:59,287 --> 00:16:09,132
[Dr. Ada Shannon] An additive-tier control, a reuse-aware policy in LMCache, a load-dependent f sweep, and hardware validation of the thermal and endurance budgets.

70
00:16:09,132 --> 00:16:28,358
[Hal Turing] So here's where I land. Under an unmodified SSD-style runtime, the roadmap's HBF packages that halve the GPU's near tier lose, and the three-condition test is a useful way to reason about far tiers. What the paper doesn't show is that HBF is a bad medium, or that selective use would win. Those are still open.

71
00:16:28,358 --> 00:16:30,309
[Dr. Ada Shannon] That's fair. Thanks for listening, everyone.

72
00:16:30,309 --> 00:16:31,423
[Hal Turing] See you next time.