SPLASH: Sparse Attention Co-Designed with High-Bandwidth Flash
Oct 6, 2026
This episode examines SPLASH, a design from the National University of Singapore that co-designs sparse attention with High-Bandwidth Flash (HBF), NAND dies stacked next to the GPU, to serve million-token contexts. The starting problem is KV cache capacity: 64 concurrent million-token requests to Qwen3-235B need about 12.9 TB, enough to fill the HBM of 68 B200 GPUs. The discussion shows why flash is hard to use for this. Reads happen at 4 KiB page granularity, scattered access causes read amplification of up to 5.8x, and program-erase endurance limits how often pages can be rewritten. The hosts contrast SPLASH with tiered-cache systems that spill to DRAM or SSD, and with token-level top-k methods like Quest and Native Sparse Attention. They walk through the algorithm: an HBM window for sinks and recent tokens, page membership fixed at eviction by grouping tokens that the same queries hit, and dense-plus-sparse outputs merged on the GPU. The claimed gains are 3.5 to 11.4x decode throughput per GPU with accuracy within 4% of dense attention. The hosts stress that these figures come from simulation because HBF hardware doesn't exist yet.
Sources:
1. SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inference — Aditya Anirudh Jonnalagadda, Agasthi Haputhanthri, Pranav Dangi, Rohan Juneja, Wenshuo Yue, Aritra Bagchi, Bin Gao, Tulika Mitra, 2026
http://arxiv.org/abs/2609.23816
2. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024
https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference
3. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024 (OSDI)
https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management
4. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan et al. (DeepSeek-AI and collaborators), 2025
https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention
5. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023 (SOSP)
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
6. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — DeepSeek-AI, 2025
https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp%3A+Boosting+Long-Context+Efficiency+with+DeepSeek+Sparse+Attention
7. HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving — Z. Li, Z. Bian, X. Huang, Y. Zhao, G. Sun, Y. Zhuo, 2026
https://scholar.google.com/scholar?q=HBF+Sucks%3F+A+Full-Stack+Characterization+of+High-Bandwidth+Flash+for+KV-Centric+LLM+Serving
8. HBFlex: A Flexible Memory System for Bridging Fine-Grained LLM States and Coarse-Grained HBF Parallel Execution — S. Zhong, W. Xu, Y. Zhou, et al., 2026
https://scholar.google.com/scholar?q=HBFlex%3A+A+Flexible+Memory+System+for+Bridging+Fine-Grained+LLM+States+and+Coarse-Grained+HBF+Parallel+Execution
9. FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference — X. Wang, Y. Xue, X. Sun, et al., 2026
https://scholar.google.com/scholar?q=FlashAccel%3A+Leveraging+High-Bandwidth+Flash+%28HBF%29+for+High-Throughput+LLM+Inference
10. H3: Hybrid Architecture Using High Bandwidth Memory and High Bandwidth Flash for Cost-Efficient LLM Inference — M. Ha, E. Kim, H. Kim, 2026
https://scholar.google.com/scholar?q=H3%3A+Hybrid+Architecture+Using+High+Bandwidth+Memory+and+High+Bandwidth+Flash+for+Cost-Efficient+LLM+Inference
11. LongSight: Compute-Enabled Memory to Accelerate Large-Context LLMs via Sparse Attention — D. Quinn, E. E. Yücel, J. Kim, J. F. Martínez, M. Alian, 2025
https://scholar.google.com/scholar?q=LongSight%3A+Compute-Enabled+Memory+to+Accelerate+Large-Context+LLMs+via+Sparse+Attention
12. InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference — X. Pan, E. Li, Q. Li, et al., 2025
https://scholar.google.com/scholar?q=InstAttention%3A+In-Storage+Attention+Offloading+for+Cost-Effective+Long-Context+LLM+Inference
13. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference — Hanshi Sun et al., 2024
https://scholar.google.com/scholar?q=ShadowKV%3A+KV+Cache+in+Shadows+for+High-Throughput+Long-Context+LLM+Inference
14. ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction — Renze Chen et al., 2024
https://scholar.google.com/scholar?q=ArkVale%3A+Efficient+Generative+LLM+Inference+with+Recallable+Key-Value+Eviction
15. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval — Di Liu et al., 2024
https://scholar.google.com/scholar?q=RetrievalAttention%3A+Accelerating+Long-Context+LLM+Inference+via+Vector+Retrieval
16. MagicPIG: LSH Sampling for Efficient LLM Generation — Zhuoming Chen et al., 2024
https://scholar.google.com/scholar?q=MagicPIG%3A+LSH+Sampling+for+Efficient+LLM+Generation
17. Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning — Chaofan Lin et al., 2025
https://scholar.google.com/scholar?q=Twilight%3A+Adaptive+Attention+Sparsity+with+Hierarchical+Top-p+Pruning
18. LLM in a Flash: Efficient Large Language Model Inference with Limited Memory — Keivan Alizadeh et al., 2023
https://scholar.google.com/scholar?q=LLM+in+a+Flash%3A+Efficient+Large+Language+Model+Inference+with+Limited+Memory
19. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot — Ruoyu Qin et al., 2025
https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation+%E2%80%94+A+KVCache-centric+Architecture+for+Serving+LLM+Chatbot
1. SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inference — Aditya Anirudh Jonnalagadda, Agasthi Haputhanthri, Pranav Dangi, Rohan Juneja, Wenshuo Yue, Aritra Bagchi, Bin Gao, Tulika Mitra, 2026
http://arxiv.org/abs/2609.23816
2. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024
https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference
3. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024 (OSDI)
https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management
4. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan et al. (DeepSeek-AI and collaborators), 2025
https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention
5. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023 (SOSP)
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
6. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — DeepSeek-AI, 2025
https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp%3A+Boosting+Long-Context+Efficiency+with+DeepSeek+Sparse+Attention
7. HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving — Z. Li, Z. Bian, X. Huang, Y. Zhao, G. Sun, Y. Zhuo, 2026
https://scholar.google.com/scholar?q=HBF+Sucks%3F+A+Full-Stack+Characterization+of+High-Bandwidth+Flash+for+KV-Centric+LLM+Serving
8. HBFlex: A Flexible Memory System for Bridging Fine-Grained LLM States and Coarse-Grained HBF Parallel Execution — S. Zhong, W. Xu, Y. Zhou, et al., 2026
https://scholar.google.com/scholar?q=HBFlex%3A+A+Flexible+Memory+System+for+Bridging+Fine-Grained+LLM+States+and+Coarse-Grained+HBF+Parallel+Execution
9. FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference — X. Wang, Y. Xue, X. Sun, et al., 2026
https://scholar.google.com/scholar?q=FlashAccel%3A+Leveraging+High-Bandwidth+Flash+%28HBF%29+for+High-Throughput+LLM+Inference
10. H3: Hybrid Architecture Using High Bandwidth Memory and High Bandwidth Flash for Cost-Efficient LLM Inference — M. Ha, E. Kim, H. Kim, 2026
https://scholar.google.com/scholar?q=H3%3A+Hybrid+Architecture+Using+High+Bandwidth+Memory+and+High+Bandwidth+Flash+for+Cost-Efficient+LLM+Inference
11. LongSight: Compute-Enabled Memory to Accelerate Large-Context LLMs via Sparse Attention — D. Quinn, E. E. Yücel, J. Kim, J. F. Martínez, M. Alian, 2025
https://scholar.google.com/scholar?q=LongSight%3A+Compute-Enabled+Memory+to+Accelerate+Large-Context+LLMs+via+Sparse+Attention
12. InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference — X. Pan, E. Li, Q. Li, et al., 2025
https://scholar.google.com/scholar?q=InstAttention%3A+In-Storage+Attention+Offloading+for+Cost-Effective+Long-Context+LLM+Inference
13. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference — Hanshi Sun et al., 2024
https://scholar.google.com/scholar?q=ShadowKV%3A+KV+Cache+in+Shadows+for+High-Throughput+Long-Context+LLM+Inference
14. ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction — Renze Chen et al., 2024
https://scholar.google.com/scholar?q=ArkVale%3A+Efficient+Generative+LLM+Inference+with+Recallable+Key-Value+Eviction
15. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval — Di Liu et al., 2024
https://scholar.google.com/scholar?q=RetrievalAttention%3A+Accelerating+Long-Context+LLM+Inference+via+Vector+Retrieval
16. MagicPIG: LSH Sampling for Efficient LLM Generation — Zhuoming Chen et al., 2024
https://scholar.google.com/scholar?q=MagicPIG%3A+LSH+Sampling+for+Efficient+LLM+Generation
17. Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning — Chaofan Lin et al., 2025
https://scholar.google.com/scholar?q=Twilight%3A+Adaptive+Attention+Sparsity+with+Hierarchical+Top-p+Pruning
18. LLM in a Flash: Efficient Large Language Model Inference with Limited Memory — Keivan Alizadeh et al., 2023
https://scholar.google.com/scholar?q=LLM+in+a+Flash%3A+Efficient+Large+Language+Model+Inference+with+Limited+Memory
19. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot — Ruoyu Qin et al., 2025
https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation+%E2%80%94+A+KVCache-centric+Architecture+for+Serving+LLM+Chatbot