Workload–Router–Pool: Unifying LLM Routing, Models, and GPU Fleets
Oct 12, 2026
This episode explores a vision paper arguing that choosing a model, routing a request, and sizing a GPU fleet are a single coupled problem. The paper organizes LLM inference into workload, router, and pool dimensions, and it presents a 3-by-3 matrix of their interactions. The discussion defines the latency and serving terms (TTFT, TPOT, prefill versus decode, KV-cache limits, warm versus cold requests). It then examines how workload shape drives the design. The examples are short chat requests that waste long-context KV slots, and agent sessions whose caches get evicted during tool pauses. It also covers how routers can infer workload type from cheap signals, how feedback-driven adaptation differs for chat and agents, and a three-tier vision-language cascade with a projected 78 percent cost reduction. Throughout, the episode stresses that the paper reports no new experiments. Listeners get a way to separate measured results from simulated or projected ones.
Interactive Visualization: Workload–Router–Pool: Unifying LLM Routing, Models, and GPU Fleets
Sources:
1. The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project — Huamin Chen, Xunzhuo Liu, Bowei He, Fuyuan Lyu, Yankai Chen, Xue Liu, Yuhan Liu, Junchen Jiang, 2026
http://arxiv.org/abs/2603.21354
2. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — Lingjiao Chen, Matei Zaharia, James Zou, 2023
https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance
3. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, Ion Stoica, 2024
https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data
4. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing — Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, Ahmed Hassan Awadallah, 2024 (ICLR)
https://scholar.google.com/scholar?q=Hybrid+LLM%3A+Cost-Efficient+and+Quality-Aware+Query+Routing
5. RouterBench: A Benchmark for Multi-LLM Routing System — Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, Shriyash Kaustubh Upadhyay, 2024
https://scholar.google.com/scholar?q=RouterBench%3A+A+Benchmark+for+Multi-LLM+Routing+System
6. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini, 2024 (ISCA)
https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting
7. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 (OSDI)
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
8. Vidur: A Large-Scale Simulation Framework for LLM Inference — Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, Alexey Tumanov, 2024 (MLSys)
https://scholar.google.com/scholar?q=Vidur%3A+A+Large-Scale+Simulation+Framework+for+LLM+Inference
9. Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity — Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, Ion Stoica, 2024
https://scholar.google.com/scholar?q=M%C3%A9lange%3A+Cost+Efficient+Large+Language+Model+Serving+by+Exploiting+GPU+Heterogeneity
10. BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems — Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, Xiaowen Chu, 2024
https://scholar.google.com/scholar?q=BurstGPT%3A+A+Real-world+Workload+Dataset+to+Optimize+LLM+Serving+Systems
11. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024 (FAST 2025)
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
12. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset — Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, Hao Zhang, 2023
https://scholar.google.com/scholar?q=LMSYS-Chat-1M%3A+A+Large-Scale+Real-World+LLM+Conversation+Dataset
13. Orca: A Distributed Serving System for Transformer-Based Generative Models and Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Gyeong-In Yu et al. (Orca, OSDI 2022); Woosuk Kwon et al. (vLLM, SOSP 2023), 2022 / 2023
https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models+and+Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29
14. GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching — Sajal Regmi, Chetan Phakami Pun, 2024
https://scholar.google.com/scholar?q=GPT+Semantic+Cache%3A+Reducing+LLM+Costs+and+Latency+via+Semantic+Embedding+Caching
15. MeanCache: User-Centric Semantic Caching for LLM Web Services — Waris Gill, Mohamed Elidrisi, Pallavi Kalapatapu, Ammar Ahmed, Ali Anwar, Muhammad Ali Gulzar, 2024
https://scholar.google.com/scholar?q=MeanCache%3A+User-Centric+Semantic+Caching+for+LLM+Web+Services
16. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (completion cache component) — Lingjiao Chen, Matei Zaharia, James Zou, 2023
https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance+%28completion+cache+component%29
17. Efficient Memory Management for Large Language Model Serving with PagedAttention (prefix/KV reuse contrast) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023 (SOSP)
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28prefix%2FKV+reuse+contrast%29
18. AutoMix: Automatically Mixing Language Models — Pranjal Aggarwal et al., 2023
https://scholar.google.com/scholar?q=AutoMix%3A+Automatically+Mixing+Language+Models
19. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving — Yinmin Zhong et al., 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+LLM+Serving
20. ServeGen: Workload Characterization of Large Language Model Serving in Production — Yuxin Wang et al., 2025
https://scholar.google.com/scholar?q=ServeGen%3A+Workload+Characterization+of+Large+Language+Model+Serving+in+Production
21. GPTCache / MeanCache: user-centric semantic caching for LLM web services — Waris Gill et al. (MeanCache); Bang (GPTCache), 2023–2025
https://scholar.google.com/scholar?q=GPTCache+%2F+MeanCache%3A+user-centric+semantic+caching+for+LLM+web+services
22. Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning — Haozhen Zhang et al., 2025
https://scholar.google.com/scholar?q=Router-R1%3A+Teaching+LLMs+Multi-Round+Routing+and+Aggregation+via+Reinforcement+Learning
23. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Vikranth Srivatsa et al., 2024
https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving
Interactive Visualization: Workload–Router–Pool: Unifying LLM Routing, Models, and GPU Fleets
1. The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project — Huamin Chen, Xunzhuo Liu, Bowei He, Fuyuan Lyu, Yankai Chen, Xue Liu, Yuhan Liu, Junchen Jiang, 2026
http://arxiv.org/abs/2603.21354
2. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — Lingjiao Chen, Matei Zaharia, James Zou, 2023
https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance
3. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, Ion Stoica, 2024
https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data
4. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing — Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, Ahmed Hassan Awadallah, 2024 (ICLR)
https://scholar.google.com/scholar?q=Hybrid+LLM%3A+Cost-Efficient+and+Quality-Aware+Query+Routing
5. RouterBench: A Benchmark for Multi-LLM Routing System — Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, Shriyash Kaustubh Upadhyay, 2024
https://scholar.google.com/scholar?q=RouterBench%3A+A+Benchmark+for+Multi-LLM+Routing+System
6. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini, 2024 (ISCA)
https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting
7. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 (OSDI)
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
8. Vidur: A Large-Scale Simulation Framework for LLM Inference — Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, Alexey Tumanov, 2024 (MLSys)
https://scholar.google.com/scholar?q=Vidur%3A+A+Large-Scale+Simulation+Framework+for+LLM+Inference
9. Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity — Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, Ion Stoica, 2024
https://scholar.google.com/scholar?q=M%C3%A9lange%3A+Cost+Efficient+Large+Language+Model+Serving+by+Exploiting+GPU+Heterogeneity
10. BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems — Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, Xiaowen Chu, 2024
https://scholar.google.com/scholar?q=BurstGPT%3A+A+Real-world+Workload+Dataset+to+Optimize+LLM+Serving+Systems
11. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024 (FAST 2025)
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
12. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset — Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, Hao Zhang, 2023
https://scholar.google.com/scholar?q=LMSYS-Chat-1M%3A+A+Large-Scale+Real-World+LLM+Conversation+Dataset
13. Orca: A Distributed Serving System for Transformer-Based Generative Models and Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Gyeong-In Yu et al. (Orca, OSDI 2022); Woosuk Kwon et al. (vLLM, SOSP 2023), 2022 / 2023
https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models+and+Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29
14. GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching — Sajal Regmi, Chetan Phakami Pun, 2024
https://scholar.google.com/scholar?q=GPT+Semantic+Cache%3A+Reducing+LLM+Costs+and+Latency+via+Semantic+Embedding+Caching
15. MeanCache: User-Centric Semantic Caching for LLM Web Services — Waris Gill, Mohamed Elidrisi, Pallavi Kalapatapu, Ammar Ahmed, Ali Anwar, Muhammad Ali Gulzar, 2024
https://scholar.google.com/scholar?q=MeanCache%3A+User-Centric+Semantic+Caching+for+LLM+Web+Services
16. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (completion cache component) — Lingjiao Chen, Matei Zaharia, James Zou, 2023
https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance+%28completion+cache+component%29
17. Efficient Memory Management for Large Language Model Serving with PagedAttention (prefix/KV reuse contrast) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023 (SOSP)
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28prefix%2FKV+reuse+contrast%29
18. AutoMix: Automatically Mixing Language Models — Pranjal Aggarwal et al., 2023
https://scholar.google.com/scholar?q=AutoMix%3A+Automatically+Mixing+Language+Models
19. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving — Yinmin Zhong et al., 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+LLM+Serving
20. ServeGen: Workload Characterization of Large Language Model Serving in Production — Yuxin Wang et al., 2025
https://scholar.google.com/scholar?q=ServeGen%3A+Workload+Characterization+of+Large+Language+Model+Serving+in+Production
21. GPTCache / MeanCache: user-centric semantic caching for LLM web services — Waris Gill et al. (MeanCache); Bang (GPTCache), 2023–2025
https://scholar.google.com/scholar?q=GPTCache+%2F+MeanCache%3A+user-centric+semantic+caching+for+LLM+web+services
22. Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning — Haozhen Zhang et al., 2025
https://scholar.google.com/scholar?q=Router-R1%3A+Teaching+LLMs+Multi-Round+Routing+and+Aggregation+via+Reinforcement+Learning
23. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Vikranth Srivatsa et al., 2024
https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving
Interactive Visualization: Workload–Router–Pool: Unifying LLM Routing, Models, and GPU Fleets