KV Cache Offloading for Concurrent Agents: When Does It Help?
Oct 1, 2026
This episode examines why offloading the KV cache to host memory speeds up an RTX 3090, slows down an H800, and does nothing on an H20 when the same concurrent coding-agent workload runs on all three. The EfficientAgent paper argues that hardware sets what a cache hit is worth, measured by the Offload Benefit Ratio and by compute per host-link byte. The reuse working set decides whether a hit exists at all. In the agent workloads, 97.3% of newly computed KV is reused in the very next call. Even so, 79.2% of chunks written to a 5 GiB host tier were evicted first, because agents running concurrently wait behind each other's traffic. The discussion also covers prefix caching and why context folding breaks exact-match hits. It contrasts this approach with pinning (Continuum) and graph-guided eviction (KVFlow), and it introduces reuse distance from 1970s stack-distance analysis as the way to size the tier. Listeners get a clear framework for deciding when KV offloading will pay off, and for why it can thrash like an undersized cache.
Interactive Visualization: KV Cache Offloading for Concurrent Agents: When Does It Help?
Sources:
1. KV Cache Offloading for Concurrent Agents: When Does It Help?
https://arxiv.org/pdf/2609.33762
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, et al., 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
3. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention (AttentionStore) — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, et al., 2024
https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention+%28AttentionStore%29
4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, et al. (Moonshot AI / Tsinghua), 2024
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
5. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, Junchen Jiang, et al., 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
6. Evaluation Techniques for Storage Hierarchies — R. L. Mattson, J. Gecsei, D. R. Slutz, I. L. Traiger, 1970
https://scholar.google.com/scholar?q=Evaluation+Techniques+for+Storage+Hierarchies
7. The Working Set Model for Program Behavior — Peter J. Denning, 1968
https://scholar.google.com/scholar?q=The+Working+Set+Model+for+Program+Behavior
8. Efficient MRC Construction with SHARDS — Carl A. Waldspurger, Nohhyun Park, Alexander Garthwaite, Irfan Ahmad, 2015
https://scholar.google.com/scholar?q=Efficient+MRC+Construction+with+SHARDS
9. Marconi: Prefix Caching for the Era of Hybrid LLMs — Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, Tri Dao, Yida Wang, Ravi Netravali, 2024
https://scholar.google.com/scholar?q=Marconi%3A+Prefix+Caching+for+the+Era+of+Hybrid+LLMs
10. TinyLFU: A Highly Efficient Cache Admission Policy — Gil Einziger, Roy Friedman, Ben Manes, 2017
https://scholar.google.com/scholar?q=TinyLFU%3A+A+Highly+Efficient+Cache+Admission+Policy
11. Flashield: a Hybrid Key-value Cache that Controls Flash Write Amplification — Assaf Eisenman, Asaf Cidon, Evgenia Pergament, Or Haimovich, Ryan Stutsman, Mohammad Alizadeh, Sachin Katti, 2019
https://scholar.google.com/scholar?q=Flashield%3A+a+Hybrid+Key-value+Cache+that+Controls+Flash+Write+Amplification
12. The CacheLib Caching Engine: Design and Experiences at Scale — Benjamin Berg, Daniel S. Berger, Sara McAllister, Isaac Grosof, Sathya Gunasekar, Jimmy Lu, Michael Uhlar, Jim Carrig, Nathan Beckmann, Mor Harchol-Balter, Gregory R. Ganger, 2020
https://scholar.google.com/scholar?q=The+CacheLib+Caching+Engine%3A+Design+and+Experiences+at+Scale
13. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live — Hanchen Li et al., 2025–2026
https://scholar.google.com/scholar?q=Continuum%3A+Efficient+and+Robust+Multi-Turn+LLM+Agent+Scheduling+with+KV+Cache+Time-to-Live
14. Idleness is relative: Exploiting tool-call idle windows for offloading in agentic systems with MORI — Tian Xia et al., 2026
https://scholar.google.com/scholar?q=Idleness+is+relative%3A+Exploiting+tool-call+idle+windows+for+offloading+in+agentic+systems+with+MORI
15. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows — Zaifeng Pan et al., 2025
https://scholar.google.com/scholar?q=KVFlow%3A+Efficient+Prefix+Caching+for+Accelerating+LLM-Based+Multi-Agent+Workflows
16. InferCept: Efficient Intercept Support for Augmented Large Language Model Inference — Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, Yiying Zhang, 2024
https://scholar.google.com/scholar?q=InferCept%3A+Efficient+Intercept+Support+for+Augmented+Large+Language+Model+Inference
17. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot — Ruoyu Qin et al., 2025
https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation+%E2%80%94+A+KVCache-centric+Architecture+for+Serving+LLM+Chatbot
18. Cache Modeling and Optimization using Miniature Simulations — Carl A. Waldspurger et al., 2017
https://scholar.google.com/scholar?q=Cache+Modeling+and+Optimization+using+Miniature+Simulations
19. AdaptSize: Orchestrating the Hot Object Memory Cache in a Content Delivery Network — Daniel S. Berger, Ramesh K. Sitaraman, Mor Harchol-Balter, 2017
https://scholar.google.com/scholar?q=AdaptSize%3A+Orchestrating+the+Hot+Object+Memory+Cache+in+a+Content+Delivery+Network
20. Stateful Large Language Model Serving with Pensieve — Lingfan Yu, Jinkun Lin, Jinyang Li, 2025
https://scholar.google.com/scholar?q=Stateful+Large+Language+Model+Serving+with+Pensieve
21. Cost-Efficient Large Language Model Serving for Multi-Turn Conversations with CachedAttention — Bin Gao et al., 2024
https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-Turn+Conversations+with+CachedAttention
22. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao et al., 2025
https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion
23. A Study of Replacement Algorithms for a Virtual-Storage Computer (MIN/OPT) — L. A. Belady, 1966
https://scholar.google.com/scholar?q=A+Study+of+Replacement+Algorithms+for+a+Virtual-Storage+Computer+%28MIN%2FOPT%29
Interactive Visualization: KV Cache Offloading for Concurrent Agents: When Does It Help?
1. KV Cache Offloading for Concurrent Agents: When Does It Help?
https://arxiv.org/pdf/2609.33762
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, et al., 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
3. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention (AttentionStore) — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, et al., 2024
https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention+%28AttentionStore%29
4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, et al. (Moonshot AI / Tsinghua), 2024
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
5. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, Junchen Jiang, et al., 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
6. Evaluation Techniques for Storage Hierarchies — R. L. Mattson, J. Gecsei, D. R. Slutz, I. L. Traiger, 1970
https://scholar.google.com/scholar?q=Evaluation+Techniques+for+Storage+Hierarchies
7. The Working Set Model for Program Behavior — Peter J. Denning, 1968
https://scholar.google.com/scholar?q=The+Working+Set+Model+for+Program+Behavior
8. Efficient MRC Construction with SHARDS — Carl A. Waldspurger, Nohhyun Park, Alexander Garthwaite, Irfan Ahmad, 2015
https://scholar.google.com/scholar?q=Efficient+MRC+Construction+with+SHARDS
9. Marconi: Prefix Caching for the Era of Hybrid LLMs — Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, Tri Dao, Yida Wang, Ravi Netravali, 2024
https://scholar.google.com/scholar?q=Marconi%3A+Prefix+Caching+for+the+Era+of+Hybrid+LLMs
10. TinyLFU: A Highly Efficient Cache Admission Policy — Gil Einziger, Roy Friedman, Ben Manes, 2017
https://scholar.google.com/scholar?q=TinyLFU%3A+A+Highly+Efficient+Cache+Admission+Policy
11. Flashield: a Hybrid Key-value Cache that Controls Flash Write Amplification — Assaf Eisenman, Asaf Cidon, Evgenia Pergament, Or Haimovich, Ryan Stutsman, Mohammad Alizadeh, Sachin Katti, 2019
https://scholar.google.com/scholar?q=Flashield%3A+a+Hybrid+Key-value+Cache+that+Controls+Flash+Write+Amplification
12. The CacheLib Caching Engine: Design and Experiences at Scale — Benjamin Berg, Daniel S. Berger, Sara McAllister, Isaac Grosof, Sathya Gunasekar, Jimmy Lu, Michael Uhlar, Jim Carrig, Nathan Beckmann, Mor Harchol-Balter, Gregory R. Ganger, 2020
https://scholar.google.com/scholar?q=The+CacheLib+Caching+Engine%3A+Design+and+Experiences+at+Scale
13. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live — Hanchen Li et al., 2025–2026
https://scholar.google.com/scholar?q=Continuum%3A+Efficient+and+Robust+Multi-Turn+LLM+Agent+Scheduling+with+KV+Cache+Time-to-Live
14. Idleness is relative: Exploiting tool-call idle windows for offloading in agentic systems with MORI — Tian Xia et al., 2026
https://scholar.google.com/scholar?q=Idleness+is+relative%3A+Exploiting+tool-call+idle+windows+for+offloading+in+agentic+systems+with+MORI
15. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows — Zaifeng Pan et al., 2025
https://scholar.google.com/scholar?q=KVFlow%3A+Efficient+Prefix+Caching+for+Accelerating+LLM-Based+Multi-Agent+Workflows
16. InferCept: Efficient Intercept Support for Augmented Large Language Model Inference — Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, Yiying Zhang, 2024
https://scholar.google.com/scholar?q=InferCept%3A+Efficient+Intercept+Support+for+Augmented+Large+Language+Model+Inference
17. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot — Ruoyu Qin et al., 2025
https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation+%E2%80%94+A+KVCache-centric+Architecture+for+Serving+LLM+Chatbot
18. Cache Modeling and Optimization using Miniature Simulations — Carl A. Waldspurger et al., 2017
https://scholar.google.com/scholar?q=Cache+Modeling+and+Optimization+using+Miniature+Simulations
19. AdaptSize: Orchestrating the Hot Object Memory Cache in a Content Delivery Network — Daniel S. Berger, Ramesh K. Sitaraman, Mor Harchol-Balter, 2017
https://scholar.google.com/scholar?q=AdaptSize%3A+Orchestrating+the+Hot+Object+Memory+Cache+in+a+Content+Delivery+Network
20. Stateful Large Language Model Serving with Pensieve — Lingfan Yu, Jinkun Lin, Jinyang Li, 2025
https://scholar.google.com/scholar?q=Stateful+Large+Language+Model+Serving+with+Pensieve
21. Cost-Efficient Large Language Model Serving for Multi-Turn Conversations with CachedAttention — Bin Gao et al., 2024
https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-Turn+Conversations+with+CachedAttention
22. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao et al., 2025
https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion
23. A Study of Replacement Algorithms for a Virtual-Storage Computer (MIN/OPT) — L. A. Belady, 1966
https://scholar.google.com/scholar?q=A+Study+of+Replacement+Algorithms+for+a+Virtual-Storage+Computer+%28MIN%2FOPT%29
Interactive Visualization: KV Cache Offloading for Concurrent Agents: When Does It Help?