Tutti: Making SSD-Backed KV Cache Practical for Long-Context Serving
Oct 2, 2026
This episode examines Tutti, a system that makes SSD-backed KV cache practical for long-context LLM serving by moving NVMe I/O control from the CPU onto the GPU. It explains why reusing a cached prefix from SSD can currently be slower than recomputing it, with the GPU sitting idle 70 to 80 percent of the time. The cause is the layout: vLLM's small paged blocks, split by layer and by key versus value, turn a 128K-token prompt into roughly a quarter-million small random reads, and GPUDirect Storage fixes the data path but leaves the CPU issuing and tracking every request. The discussion covers GPU-centric storage lineage such as BaM and GeminiFS, and Tutti's design. The CPU keeps the prefix index off the critical path, since hash tables run far slower on the GPU, while a tensor-stripe layout on pre-allocated files matches the I/O unit to the tensor. Throughout, the hosts ask whether the gains come from the CPU-to-GPU shift itself or from other changes, such as smaller blocks, a new object layout, and a weaker LMCache baseline.
Sources:
1. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving — Shi Qiu, Yifan Hu, Xintao Wang, Wenhao Zhu, Jianqin Yan, Hao Chen, Kaiqiang Xu, Kai Chen, Yiming Zhang, 2026
http://arxiv.org/abs/2605.03375v1
2. GPUfs: Integrating a File System with GPUs — Mark Silberstein, Bryan Ford, Idit Keidar, Emmett Witchel, 2013
https://scholar.google.com/scholar?q=GPUfs%3A+Integrating+a+File+System+with+GPUs
3. GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture — Zaid Qureshi, Vikram Sharma Mailthody, Isaac Gelado, Seung Won Min, Amna Masood, Jeongmin Park, Jinjun Xiong, C. J. Newburn, Dmitri Vainbrand, I-Hsin Chung, Michael Garland, William Dally, Wen-mei Hwu, 2023
https://scholar.google.com/scholar?q=GPU-Initiated+On-Demand+High-Throughput+Storage+Access+in+the+BaM+System+Architecture
4. GeminiFS: A Companion File System for GPUs — Shi Qiu, Weinan Liu, Yifan Hu, Jianqin Yan, Zhirong Shen, Xin Yao, Renhai Chen, Gong Zhang, Yiming Zhang, 2025
https://scholar.google.com/scholar?q=GeminiFS%3A+A+Companion+File+System+for+GPUs
5. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, Junchen Jiang, and collaborators, 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
6. Efficient IO with io_uring — Jens Axboe, 2019
https://scholar.google.com/scholar?q=Efficient+IO+with+io_uring
7. An Optimal Algorithm for Scheduling Soft-Aperiodic Tasks in Fixed-Priority Preemptive Systems (Slack Stealing) — John P. Lehoczky, Sandra Ramos-Thuel, 1992
https://scholar.google.com/scholar?q=An+Optimal+Algorithm+for+Scheduling+Soft-Aperiodic+Tasks+in+Fixed-Priority+Preemptive+Systems+%28Slack+Stealing%29
8. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang, 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
9. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention (AttentionStore) — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo, 2024
https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention+%28AttentionStore%29
10. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024 (FAST 2025)
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
11. Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications — Foteini Strati, Xianzhe Ma, Ana Klimovic, 2024
https://scholar.google.com/scholar?q=Orion%3A+Interference-aware%2C+Fine-grained+GPU+Sharing+for+ML+Applications
12. Managing Scalable Direct Storage Accesses for GPUs with GoFS — Shaobo Li, Yirui Eric Zhou, Yuqi Xue, Yuan Xu, Jian Huang, 2025
https://scholar.google.com/scholar?q=Managing+Scalable+Direct+Storage+Accesses+for+GPUs+with+GoFS
13. Phoenix: A Refactored I/O Stack for GPU Direct Storage without Phony Buffers — Jianqin Yan, Shi Qiu, et al., 2025
https://scholar.google.com/scholar?q=Phoenix%3A+A+Refactored+I%2FO+Stack+for+GPU+Direct+Storage+without+Phony+Buffers
14. An I/O Characterizing Study of Offloading LLM Models and KV Caches to NVMe SSD — Zebin Ren, Krijn Doekemeijer, Tiziano De Matteis, Christian Pinto, Radu Stoica, Animesh Trivedi, 2025
https://scholar.google.com/scholar?q=An+I%2FO+Characterizing+Study+of+Offloading+LLM+Models+and+KV+Caches+to+NVMe+SSD
15. Strata: Hierarchical Context Caching for Long Context Language Model Serving — Zhiqiang Xie et al., 2025
https://scholar.google.com/scholar?q=Strata%3A+Hierarchical+Context+Caching+for+Long+Context+Language+Model+Serving
16. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving (LMCache lineage) — Yuhan Liu et al., 2024
https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving+%28LMCache+lineage%29
17. Fast State Restoration in LLM Serving with HCache — Shiwei Gao, Youmin Chen, Jiwu Shu, 2025
https://scholar.google.com/scholar?q=Fast+State+Restoration+in+LLM+Serving+with+HCache
18. Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration — Zejia Lin et al., 2025
https://scholar.google.com/scholar?q=Bullet%3A+Boosting+GPU+Utilization+for+LLM+Serving+via+Dynamic+Spatial-Temporal+Orchestration
19. Slack stealing for hard real-time scheduling (Lehoczky and Ramos-Thuel) — John P. Lehoczky, Sandra Ramos-Thuel, 1992
https://scholar.google.com/scholar?q=Slack+stealing+for+hard+real-time+scheduling+%28Lehoczky+and+Ramos-Thuel%29
1. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving — Shi Qiu, Yifan Hu, Xintao Wang, Wenhao Zhu, Jianqin Yan, Hao Chen, Kaiqiang Xu, Kai Chen, Yiming Zhang, 2026
http://arxiv.org/abs/2605.03375v1
2. GPUfs: Integrating a File System with GPUs — Mark Silberstein, Bryan Ford, Idit Keidar, Emmett Witchel, 2013
https://scholar.google.com/scholar?q=GPUfs%3A+Integrating+a+File+System+with+GPUs
3. GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture — Zaid Qureshi, Vikram Sharma Mailthody, Isaac Gelado, Seung Won Min, Amna Masood, Jeongmin Park, Jinjun Xiong, C. J. Newburn, Dmitri Vainbrand, I-Hsin Chung, Michael Garland, William Dally, Wen-mei Hwu, 2023
https://scholar.google.com/scholar?q=GPU-Initiated+On-Demand+High-Throughput+Storage+Access+in+the+BaM+System+Architecture
4. GeminiFS: A Companion File System for GPUs — Shi Qiu, Weinan Liu, Yifan Hu, Jianqin Yan, Zhirong Shen, Xin Yao, Renhai Chen, Gong Zhang, Yiming Zhang, 2025
https://scholar.google.com/scholar?q=GeminiFS%3A+A+Companion+File+System+for+GPUs
5. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, Junchen Jiang, and collaborators, 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
6. Efficient IO with io_uring — Jens Axboe, 2019
https://scholar.google.com/scholar?q=Efficient+IO+with+io_uring
7. An Optimal Algorithm for Scheduling Soft-Aperiodic Tasks in Fixed-Priority Preemptive Systems (Slack Stealing) — John P. Lehoczky, Sandra Ramos-Thuel, 1992
https://scholar.google.com/scholar?q=An+Optimal+Algorithm+for+Scheduling+Soft-Aperiodic+Tasks+in+Fixed-Priority+Preemptive+Systems+%28Slack+Stealing%29
8. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang, 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
9. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention (AttentionStore) — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo, 2024
https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention+%28AttentionStore%29
10. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024 (FAST 2025)
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
11. Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications — Foteini Strati, Xianzhe Ma, Ana Klimovic, 2024
https://scholar.google.com/scholar?q=Orion%3A+Interference-aware%2C+Fine-grained+GPU+Sharing+for+ML+Applications
12. Managing Scalable Direct Storage Accesses for GPUs with GoFS — Shaobo Li, Yirui Eric Zhou, Yuqi Xue, Yuan Xu, Jian Huang, 2025
https://scholar.google.com/scholar?q=Managing+Scalable+Direct+Storage+Accesses+for+GPUs+with+GoFS
13. Phoenix: A Refactored I/O Stack for GPU Direct Storage without Phony Buffers — Jianqin Yan, Shi Qiu, et al., 2025
https://scholar.google.com/scholar?q=Phoenix%3A+A+Refactored+I%2FO+Stack+for+GPU+Direct+Storage+without+Phony+Buffers
14. An I/O Characterizing Study of Offloading LLM Models and KV Caches to NVMe SSD — Zebin Ren, Krijn Doekemeijer, Tiziano De Matteis, Christian Pinto, Radu Stoica, Animesh Trivedi, 2025
https://scholar.google.com/scholar?q=An+I%2FO+Characterizing+Study+of+Offloading+LLM+Models+and+KV+Caches+to+NVMe+SSD
15. Strata: Hierarchical Context Caching for Long Context Language Model Serving — Zhiqiang Xie et al., 2025
https://scholar.google.com/scholar?q=Strata%3A+Hierarchical+Context+Caching+for+Long+Context+Language+Model+Serving
16. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving (LMCache lineage) — Yuhan Liu et al., 2024
https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving+%28LMCache+lineage%29
17. Fast State Restoration in LLM Serving with HCache — Shiwei Gao, Youmin Chen, Jiwu Shu, 2025
https://scholar.google.com/scholar?q=Fast+State+Restoration+in+LLM+Serving+with+HCache
18. Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration — Zejia Lin et al., 2025
https://scholar.google.com/scholar?q=Bullet%3A+Boosting+GPU+Utilization+for+LLM+Serving+via+Dynamic+Spatial-Temporal+Orchestration
19. Slack stealing for hard real-time scheduling (Lehoczky and Ramos-Thuel) — John P. Lehoczky, Sandra Ramos-Thuel, 1992
https://scholar.google.com/scholar?q=Slack+stealing+for+hard+real-time+scheduling+%28Lehoczky+and+Ramos-Thuel%29