HBF Sucks? Why Faster Flash Slows KV-Centric LLM Serving
Oct 8, 2026
This episode examines "HBF Sucks?", a simulation study of what happens when High-Bandwidth Flash (stacked 3D NAND with a wide package-local interface) replaces the SSD tier in a Mooncake-style KV-offloading stack, and why serving gets slower. It first covers the KV cache, prefix reuse, and the near-tier/far-tier split, then HBF's tradeoffs: flash-scale capacity and aggregate bandwidth, but microsecond read latency and costly page-program writes. The paper's central argument is that a faster far tier only pays off under three conditions. Exposed read I/O must be the bottleneck, reads must outweigh writes (useful bytes read per byte written of at least about one), and the delivered bandwidth must be sustainable thermally and for endurance. Transient KV cache is argued to violate all three. The episode also covers the HBF-1 and HBF-2 packaging roadmaps, which cost GPU near-tier memory, and the trace-driven evaluation on Alibaba Qwen-Bailian traces across five models, modeled H100 and B200 systems, and a capacity-matched SSD baseline. It is useful for anyone weighing flash-based memory tiers for LLM serving.
Interactive Visualization: HBF Sucks? Why Faster Flash Slows KV-Centric LLM Serving
Sources:
1. HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving — Zhuoran Li, Zhuohang Bian, Xin Huang, Yibo Zhao, Guangyu Sun, Youwei Zhuo, 2026
http://arxiv.org/abs/2608.11668v4
2. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yuhan Liu, Yihua Cheng, et al., 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
3. Strata: Hierarchical Context Caching for Long Context Language Model Serving — Zhiqiang Xie, Ziyi Xu, Mark Zhao, et al., 2025
https://scholar.google.com/scholar?q=Strata%3A+Hierarchical+Context+Caching+for+Long+Context+Language+Model+Serving
4. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving — Shi Qiu, Yifan Hu, et al., 2026
https://scholar.google.com/scholar?q=Tutti%3A+Making+SSD-Backed+KV+Cache+Practical+for+Long-Context+LLM+Serving
5. KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider — Jiahao Wang, Jinbo Han, Xingda Wei, et al., 2025
https://scholar.google.com/scholar?q=KVCache+Cache+in+the+Wild%3A+Characterizing+and+Optimizing+KVCache+Cache+at+a+Large+Cloud+Provider
6. H3: Hybrid Architecture Using High Bandwidth Memory and High Bandwidth Flash for Cost-Efficient LLM Inference — Minho Ha, Euiseok Kim, Hoshik Kim, 2026
https://scholar.google.com/scholar?q=H3%3A+Hybrid+Architecture+Using+High+Bandwidth+Memory+and+High+Bandwidth+Flash+for+Cost-Efficient+LLM+Inference
7. FlashAccel: Leveraging High-Bandwidth Flash for High-Throughput LLM Inference — Xinyu Wang, Yalong Xue, et al., 2026
https://scholar.google.com/scholar?q=FlashAccel%3A+Leveraging+High-Bandwidth+Flash+for+High-Throughput+LLM+Inference
8. HBFSim: Fast and Faithful Simulation of High-Bandwidth Flash Under Real GPU Execution — Yanpeng Hu, Yiwei Yang, et al., 2026
https://scholar.google.com/scholar?q=HBFSim%3A+Fast+and+Faithful+Simulation+of+High-Bandwidth+Flash+Under+Real+GPU+Execution
9. Exploring High-Bandwidth Flash for Modern LLM Inference: Opportunities and Challenges — Dowon Son, Yonggon Park, et al. (with Onur Mutlu), 2026
https://scholar.google.com/scholar?q=Exploring+High-Bandwidth+Flash+for+Modern+LLM+Inference%3A+Opportunities+and+Challenges
10. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention — Bin Gao, Zhuomin He, et al., 2024
https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention
11. Data Retention in MLC NAND Flash Memory: Characterization, Optimization, and Recovery — Yu Cai, Yixin Luo, Erich Haratsch, Ken Mai, Onur Mutlu, 2015
https://scholar.google.com/scholar?q=Data+Retention+in+MLC+NAND+Flash+Memory%3A+Characterization%2C+Optimization%2C+and+Recovery
Interactive Visualization: HBF Sucks? Why Faster Flash Slows KV-Centric LLM Serving
1. HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving — Zhuoran Li, Zhuohang Bian, Xin Huang, Yibo Zhao, Guangyu Sun, Youwei Zhuo, 2026
http://arxiv.org/abs/2608.11668v4
2. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yuhan Liu, Yihua Cheng, et al., 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
3. Strata: Hierarchical Context Caching for Long Context Language Model Serving — Zhiqiang Xie, Ziyi Xu, Mark Zhao, et al., 2025
https://scholar.google.com/scholar?q=Strata%3A+Hierarchical+Context+Caching+for+Long+Context+Language+Model+Serving
4. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving — Shi Qiu, Yifan Hu, et al., 2026
https://scholar.google.com/scholar?q=Tutti%3A+Making+SSD-Backed+KV+Cache+Practical+for+Long-Context+LLM+Serving
5. KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider — Jiahao Wang, Jinbo Han, Xingda Wei, et al., 2025
https://scholar.google.com/scholar?q=KVCache+Cache+in+the+Wild%3A+Characterizing+and+Optimizing+KVCache+Cache+at+a+Large+Cloud+Provider
6. H3: Hybrid Architecture Using High Bandwidth Memory and High Bandwidth Flash for Cost-Efficient LLM Inference — Minho Ha, Euiseok Kim, Hoshik Kim, 2026
https://scholar.google.com/scholar?q=H3%3A+Hybrid+Architecture+Using+High+Bandwidth+Memory+and+High+Bandwidth+Flash+for+Cost-Efficient+LLM+Inference
7. FlashAccel: Leveraging High-Bandwidth Flash for High-Throughput LLM Inference — Xinyu Wang, Yalong Xue, et al., 2026
https://scholar.google.com/scholar?q=FlashAccel%3A+Leveraging+High-Bandwidth+Flash+for+High-Throughput+LLM+Inference
8. HBFSim: Fast and Faithful Simulation of High-Bandwidth Flash Under Real GPU Execution — Yanpeng Hu, Yiwei Yang, et al., 2026
https://scholar.google.com/scholar?q=HBFSim%3A+Fast+and+Faithful+Simulation+of+High-Bandwidth+Flash+Under+Real+GPU+Execution
9. Exploring High-Bandwidth Flash for Modern LLM Inference: Opportunities and Challenges — Dowon Son, Yonggon Park, et al. (with Onur Mutlu), 2026
https://scholar.google.com/scholar?q=Exploring+High-Bandwidth+Flash+for+Modern+LLM+Inference%3A+Opportunities+and+Challenges
10. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention — Bin Gao, Zhuomin He, et al., 2024
https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention
11. Data Retention in MLC NAND Flash Memory: Characterization, Optimization, and Recovery — Yu Cai, Yixin Luo, Erich Haratsch, Ken Mai, Onur Mutlu, 2015
https://scholar.google.com/scholar?q=Data+Retention+in+MLC+NAND+Flash+Memory%3A+Characterization%2C+Optimization%2C+and+Recovery
Interactive Visualization: HBF Sucks? Why Faster Flash Slows KV-Centric LLM Serving