Puzzle Compression for Hybrid MoE LLMs
Jul 9, 2026
This episode explores NVIDIA’s Nemotron-Labs-3-Puzzle-75B-A9B, a July 2026 paper on compressing a hybrid Mamba-attention mixture-of-experts reasoning model after training instead of building a smaller model from scratch. It explains why hybrid MoE systems are harder to shrink than dense transformers, focusing on routing, active expert budgets, Mamba state, and long-context memory costs, and walks through the paper’s iterative Puzzle method of pruning, distilling, and recovery in staged rounds. The discussion highlights the headline result: a parent model reduced from 120.7B total parameters and 12.8B active per token to 75.3B total and 9.3B active, while reportedly delivering about 2x higher interactive throughput on an 8xB200 server and raising million-token concurrency on a single H100 from one request to eight. Listeners would find it interesting because it digs into whether those gains reflect a real quality-efficiency advance for long-context serving or depend heavily on extra tricks such as quantization, multi-token prediction, and substantial post-training recovery compute.
Sources:
1. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs — Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, Shay Aharon, Nir Ailon, Vladimir Anisimov, Omer Ullman Argov, Maor Ashkenazi, Tomer Asida, Nave Assaf, Tomer Bar Natan, Alexander Bukharin, Grzegorz Chlebus, Marcin Chochowski, Eric Chung, Mohammad Dabbah, Carlo del Mundo, Ewa Dobrowolska, Ido Galil, Yaniv Galron, Amnon Geifman, Yonatan Geifman, Izik Golan, Alex Gronskiy, Tomasz Grzegorzek, Netanel Haber, Lior Kadoch, Grzegorz Karch, Tomer Keren, Abhinav Khattar, Amir Klein, Tugrul Konuk, Roi Koren, Daniel Korzekwa, Shaun Kotek, Konstantinos Krommydas, Itay Levy, Ofri Masad, Yoav Miron, Pavlo Molchanov, Shahar Mor, Zach Moshe, Saurav Muralidharan, Najeeb Nabwani, Besmira Nushi, Mostofa Patwary, Omri Puny, Johannes Rausch, Tomer Ronen, Sepehr Sameni, Itamar Schen, Elad Segal, Daniel Serebrenik, Ido Shahaf, Soumye Singhal, Daniil Sorokin, Sharath Turuvekere Sreenivas, Marta Stepniewska-Dziubinska, Ali Taghibakhshi, Nima Tajbakhsh, Oren Tropp, Dor Tzur, Anna Warno, Yi-Fu Wu, Michal Zawalski, Jiaqi Zeng, Yian Zhang, Ran Zilberstein, Amit Zuker, Ran El-Yaniv, 2026
http://arxiv.org/abs/2607.04371
2. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2021
https://scholar.google.com/scholar?q=Switch+Transformers%3A+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity
3. Jamba: A Hybrid Transformer-Mamba Language Model — Opher Lieber, Barak Lenz, Hofit Bata, et al., 2024
https://scholar.google.com/scholar?q=Jamba%3A+A+Hybrid+Transformer-Mamba+Language+Model
4. Puzzle: Distillation-Based NAS for Inference-Optimized LLMs — Akhiad Bercovich, Tomer Ronen, Talor Abramovich, et al., 2024
https://scholar.google.com/scholar?q=Puzzle%3A+Distillation-Based+NAS+for+Inference-Optimized+LLMs
5. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs — Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, et al., 2026
https://scholar.google.com/scholar?q=Nemotron-Labs-3-Puzzle-75B-A9B%3A+Compressing+Hybrid+MoE+LLMs
6. Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning — NVIDIA et al., 2026
https://scholar.google.com/scholar?q=Nemotron+3+Super%3A+Open%2C+Efficient+Mixture-of-Experts+Hybrid+Mamba-Transformer+Model+for+Agentic+Reasoning
7. Extending Puzzle for Mixture-of-Experts Reasoning Models with Application to GPT-OSS Acceleration — Akhiad Bercovich et al., 2026
https://scholar.google.com/scholar?q=Extending+Puzzle+for+Mixture-of-Experts+Reasoning+Models+with+Application+to+GPT-OSS+Acceleration
8. Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control — Ali Taghibakhshi et al., 2026
https://scholar.google.com/scholar?q=Star+Elastic%3A+Many-in-One+Reasoning+LLMs+with+Efficient+Budget+Control
9. LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts — Venmugil Elango et al., 2026
https://scholar.google.com/scholar?q=LatentMoE%3A+Toward+Optimal+Accuracy+per+FLOP+and+Parameter+in+Mixture+of+Experts
10. SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding — Talor Abramovich et al., 2026
https://scholar.google.com/scholar?q=SPEED-Bench%3A+A+Unified+and+Diverse+Benchmark+for+Speculative+Decoding
11. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai et al., 2024
https://arxiv.org/abs/2401.06066
12. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling — Liliang Ren et al., 2024
https://arxiv.org/abs/2406.07522
13. Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning — Zhen Li et al., 2025
https://arxiv.org/abs/2501.03035
14. ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference — Yesheng Liang et al., 2025
https://arxiv.org/abs/2511.10645
15. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu et al., 2023
https://arxiv.org/abs/2310.07240
16. KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving — Zedong Liu et al., 2026
https://arxiv.org/abs/2605.13734
17. AI Post Transformers: Ministral 3: Cascade Distillation for Long-Context Multimodal Models — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-15-cascade-distillation-for-long-context-mu-0ebd1a.mp3
18. AI Post Transformers: Do Transformers Need Three Projections? — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-11-do-transformers-need-three-projections-c227d6.mp3
19. AI Post Transformers: Mooncake for KV Cache-Centric LLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-05-mooncake-for-kv-cache-centric-llm-servin-1086d0.mp3
20. AI Post Transformers: AIConfigurator for Cross-Framework LLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-07-02-aiconfigurator-for-cross-framework-llm-s-b39139.mp3
21. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp3
22. AI Post Transformers: EMO: Emergent Modularity in Sparse Language Models — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-06-emo-emergent-modularity-in-sparse-langua-9551c4.mp3
1. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs — Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, Shay Aharon, Nir Ailon, Vladimir Anisimov, Omer Ullman Argov, Maor Ashkenazi, Tomer Asida, Nave Assaf, Tomer Bar Natan, Alexander Bukharin, Grzegorz Chlebus, Marcin Chochowski, Eric Chung, Mohammad Dabbah, Carlo del Mundo, Ewa Dobrowolska, Ido Galil, Yaniv Galron, Amnon Geifman, Yonatan Geifman, Izik Golan, Alex Gronskiy, Tomasz Grzegorzek, Netanel Haber, Lior Kadoch, Grzegorz Karch, Tomer Keren, Abhinav Khattar, Amir Klein, Tugrul Konuk, Roi Koren, Daniel Korzekwa, Shaun Kotek, Konstantinos Krommydas, Itay Levy, Ofri Masad, Yoav Miron, Pavlo Molchanov, Shahar Mor, Zach Moshe, Saurav Muralidharan, Najeeb Nabwani, Besmira Nushi, Mostofa Patwary, Omri Puny, Johannes Rausch, Tomer Ronen, Sepehr Sameni, Itamar Schen, Elad Segal, Daniel Serebrenik, Ido Shahaf, Soumye Singhal, Daniil Sorokin, Sharath Turuvekere Sreenivas, Marta Stepniewska-Dziubinska, Ali Taghibakhshi, Nima Tajbakhsh, Oren Tropp, Dor Tzur, Anna Warno, Yi-Fu Wu, Michal Zawalski, Jiaqi Zeng, Yian Zhang, Ran Zilberstein, Amit Zuker, Ran El-Yaniv, 2026
http://arxiv.org/abs/2607.04371
2. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2021
https://scholar.google.com/scholar?q=Switch+Transformers%3A+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity
3. Jamba: A Hybrid Transformer-Mamba Language Model — Opher Lieber, Barak Lenz, Hofit Bata, et al., 2024
https://scholar.google.com/scholar?q=Jamba%3A+A+Hybrid+Transformer-Mamba+Language+Model
4. Puzzle: Distillation-Based NAS for Inference-Optimized LLMs — Akhiad Bercovich, Tomer Ronen, Talor Abramovich, et al., 2024
https://scholar.google.com/scholar?q=Puzzle%3A+Distillation-Based+NAS+for+Inference-Optimized+LLMs
5. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs — Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, et al., 2026
https://scholar.google.com/scholar?q=Nemotron-Labs-3-Puzzle-75B-A9B%3A+Compressing+Hybrid+MoE+LLMs
6. Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning — NVIDIA et al., 2026
https://scholar.google.com/scholar?q=Nemotron+3+Super%3A+Open%2C+Efficient+Mixture-of-Experts+Hybrid+Mamba-Transformer+Model+for+Agentic+Reasoning
7. Extending Puzzle for Mixture-of-Experts Reasoning Models with Application to GPT-OSS Acceleration — Akhiad Bercovich et al., 2026
https://scholar.google.com/scholar?q=Extending+Puzzle+for+Mixture-of-Experts+Reasoning+Models+with+Application+to+GPT-OSS+Acceleration
8. Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control — Ali Taghibakhshi et al., 2026
https://scholar.google.com/scholar?q=Star+Elastic%3A+Many-in-One+Reasoning+LLMs+with+Efficient+Budget+Control
9. LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts — Venmugil Elango et al., 2026
https://scholar.google.com/scholar?q=LatentMoE%3A+Toward+Optimal+Accuracy+per+FLOP+and+Parameter+in+Mixture+of+Experts
10. SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding — Talor Abramovich et al., 2026
https://scholar.google.com/scholar?q=SPEED-Bench%3A+A+Unified+and+Diverse+Benchmark+for+Speculative+Decoding
11. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai et al., 2024
https://arxiv.org/abs/2401.06066
12. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling — Liliang Ren et al., 2024
https://arxiv.org/abs/2406.07522
13. Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning — Zhen Li et al., 2025
https://arxiv.org/abs/2501.03035
14. ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference — Yesheng Liang et al., 2025
https://arxiv.org/abs/2511.10645
15. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu et al., 2023
https://arxiv.org/abs/2310.07240
16. KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving — Zedong Liu et al., 2026
https://arxiv.org/abs/2605.13734
17. AI Post Transformers: Ministral 3: Cascade Distillation for Long-Context Multimodal Models — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-15-cascade-distillation-for-long-context-mu-0ebd1a.mp3
18. AI Post Transformers: Do Transformers Need Three Projections? — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-11-do-transformers-need-three-projections-c227d6.mp3
19. AI Post Transformers: Mooncake for KV Cache-Centric LLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-05-mooncake-for-kv-cache-centric-llm-servin-1086d0.mp3
20. AI Post Transformers: AIConfigurator for Cross-Framework LLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-07-02-aiconfigurator-for-cross-framework-llm-s-b39139.mp3
21. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp3
22. AI Post Transformers: EMO: Emergent Modularity in Sparse Language Models — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-06-emo-emergent-modularity-in-sparse-langua-9551c4.mp3