Disaggregated Quantization: Specializing Numeric Formats for LLM Prefill and Decode

This episode examines "Disaggregated Quantization: Specializing LLM Prefill and Decode," which argues that prefill (compute-bound) and decode (memory-bound) should each get their own quantization format, weights, and storage placement instead of sharing one compromise. The discussion covers why weight-only formats like GPTQ suit decode while W4A4 formats like QuaRot and NVFP4 suit prefill, and why compressed weights can't simply be reused across phases without costly re-quantization. It contrasts this numeric split with earlier system-level prefill/decode separation in Splitwise, DistServe, and Mooncake. The episode also walks through the paper's reported results, including gains of 1.9 to 4.5 points on decode-heavy tasks, a 1.78x time-to-first-token improvement at 8K context, and a trained NVFP4 "prefiller" checkpoint that lifts a frozen 1-bit decoder by over 32 points on MMLU-Pro. The hosts also scrutinize how these numbers were built, for instance accuracy evaluated under disaggregated serving while latency was measured at batch one on a single device, and how quantization-aware distillation reuses the assistant-token mask so that response-only loss still trains the prefill weights through the keys and values they produce.

Sources:
1. Disaggregated Quantization: Specializing Numeric Formats for LLM Prefill and Decode
https://arxiv.org/pdf/2609.26333
2. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 (OSDI 2024; arXiv Jan 2024)
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
3. QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving — Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, Song Han, 2024 (MLSys 2025; arXiv May 2024)
https://scholar.google.com/scholar?q=QServe%3A+W4A8KV4+Quantization+and+System+Co-design+for+Efficient+LLM+Serving
4. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022 (ICLR 2023)
https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers
5. Microscaling Data Formats for Deep Learning — Bita Darvish Rouhani et al. (Microsoft, AMD, Arm, Intel, Meta, NVIDIA, Qualcomm; OCP MX alliance), 2023
https://scholar.google.com/scholar?q=Microscaling+Data+Formats+for+Deep+Learning
6. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, et al., 2024
https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting
7. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al., 2024/2025
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
8. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference with Chunked Prefills — Amey Agrawal, Nitin Kedia, Ashish Panwar, et al., 2024
https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Chunked+Prefills
9. Progressive Mixed-Precision Decoding for Efficient LLM Inference — Hao Mark Chen, Fuwen Tan, Alexandros Kouris, et al., 2025
https://scholar.google.com/scholar?q=Progressive+Mixed-Precision+Decoding+for+Efficient+LLM+Inference
10. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, et al., 2025
https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study
11. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021
https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation
12. Cache-to-Cache: Direct Semantic Communication Between Large Language Models — Tianyu Fu et al., 2025
https://scholar.google.com/scholar?q=Cache-to-Cache%3A+Direct+Semantic+Communication+Between+Large+Language+Models
13. DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — Yuhan Liu et al., 2024
https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+Cache+Sharing+for+Cross-LLM+Communication+and+Multi-LLM+Serving
14. You Only Cache Once: Decoder-Decoder Architectures for Language Models (YOCO) — Yutao Sun, Li Dong, Yi Zhu, et al., 2024
https://scholar.google.com/scholar?q=You+Only+Cache+Once%3A+Decoder-Decoder+Architectures+for+Language+Models+%28YOCO%29
15. T5Gemma: Encoder-Decoder Gemma Models (adapting decoder-only LLMs into encoder-decoder) — Google DeepMind (Zhang et al.), 2025
https://scholar.google.com/scholar?q=T5Gemma%3A+Encoder-Decoder+Gemma+Models+%28adapting+decoder-only+LLMs+into+encoder-decoder%29
16. LLM in a flash: Efficient Large Language Model Inference with Limited Memory — Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, et al., 2023/2024
https://scholar.google.com/scholar?q=LLM+in+a+flash%3A+Efficient+Large+Language+Model+Inference+with+Limited+Memory
17. QLoRA: Efficient Finetuning of Quantized LLMs — Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer, 2023
https://scholar.google.com/scholar?q=QLoRA%3A+Efficient+Finetuning+of+Quantized+LLMs
18. Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models — Ruikang Liu et al., 2025
https://scholar.google.com/scholar?q=Quantization+Hurts+Reasoning%3F+An+Empirical+Study+on+Quantized+Reasoning+Models
19. Pretraining Large Language Models with NVFP4 — NVIDIA, 2025
https://scholar.google.com/scholar?q=Pretraining+Large+Language+Models+with+NVFP4
20. QTIP: Quantization with Trellises and Incoherence Processing — Albert Tseng, Qingyao Sun, David Hou, Christopher De Sa, 2024
https://scholar.google.com/scholar?q=QTIP%3A+Quantization+with+Trellises+and+Incoherence+Processing