Tensor Addressable Bridge for Remote Memory Inference
Apr 12, 2026
This episode explores a rack-scale AI inference architecture that treats remote memory as a primary serving resource, using a tensor prefetcher, software-managed placement, and shared-memory communication to hide latency and reduce pressure on local accelerator memory. It compares the proposal with prior approaches like ZeRO-Infinity and vLLM, arguing that the real shift is not just offloading but redesigning the hardware topology and execution plan so tensors, KV state, and activations can move through a coordinated memory hierarchy. The discussion highlights headline claims from simulation—up to 93% less local memory use, 50% GPU compute savings, and 50% fewer GPUs for models such as GPT-3, Grok-1, and Qwen3-235B—while scrutinizing the paper’s bolder communication claims of 16x to 70x faster inter-GPU exchange as theoretical rather than production-proven. Listeners would find it interesting for its clear debate over whether this is a genuine systems breakthrough or an appealing architecture whose benefits still depend on fair baselines, realistic traces, and unresolved implementation details.
Sources:
1. FengHuang: Next-Generation Memory Orchestration for AI Inferencing — Jiamin Li, Lei Qu, Tao Zhang, Grigory Chirkov, Shuotao Xu, Peng Cheng, Lidong Zhou, 2025
http://arxiv.org/abs/2511.10753
2. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021
https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning
3. vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023
https://scholar.google.com/scholar?q=vLLM%3A+Easy%2C+Fast%2C+and+Cheap+LLM+Serving+with+PagedAttention
4. Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Apoorv Saxena, Ameet Deshpande, et al., 2023
https://scholar.google.com/scholar?q=Sarathi%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills
5. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Authors of DistServe, 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
6. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Authors of Mooncake, 2024
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
7. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022
https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness
8. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro, 2019
https://scholar.google.com/scholar?q=Megatron-LM%3A+Training+Multi-Billion+Parameter+Language+Models+Using+Model+Parallelism
9. Fast Distributed Inference Serving for Large Language Models — Sheng Shen, Zhen Dong, Jianguo Li, et al., 2023
https://scholar.google.com/scholar?q=Fast+Distributed+Inference+Serving+for+Large+Language+Models
10. XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization — approx. recent systems/ML authors, 2025
https://scholar.google.com/scholar?q=XQuant%3A+Breaking+the+Memory+Wall+for+LLM+Inference+with+KV+Cache+Rematerialization
11. Q-hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache — approx. recent LLM inference authors, 2025
https://scholar.google.com/scholar?q=Q-hitter%3A+A+Better+Token+Oracle+for+Efficient+LLM+Inference+via+Sparse-Quantized+KV+Cache
12. KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference — approx. recent systems authors, 2025
https://scholar.google.com/scholar?q=KVSwap%3A+Disk-aware+KV+Cache+Offloading+for+Long-Context+On-device+Inference
13. Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput — approx. recent distributed inference authors, 2025
https://scholar.google.com/scholar?q=Speculative+Decoding+in+Decentralized+LLM+Inference%3A+Turning+Communication+Latency+into+Computation+Throughput
14. SLiM: Speculative Decoding with Hypothesis Reduction — approx. recent speculative decoding authors, 2025
https://scholar.google.com/scholar?q=SLiM%3A+Speculative+Decoding+with+Hypothesis+Reduction
15. Speculative Decoding and Beyond: An In-Depth Survey of Techniques — approx. survey authors, 2025
https://scholar.google.com/scholar?q=Speculative+Decoding+and+Beyond%3A+An+In-Depth+Survey+of+Techniques
16. Accelerating Transformer Model Inference through Software Optimization and Processing-in-Memory — approx. architecture authors, 2024
https://scholar.google.com/scholar?q=Accelerating+Transformer+Model+Inference+through+Software+Optimization+and+Processing-in-Memory
17. Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference — approx. review authors, 2024
https://scholar.google.com/scholar?q=Memory+Is+All+You+Need%3A+An+Overview+of+Compute-in-Memory+Architectures+for+Accelerating+Large+Language+Model+Inference
18. AI Post Transformers: CXL-SpecKV: Bridging the LLM Memory Wall with Speculative FPGA Disaggregation — Hal Turing & Dr. Ada Shannon, Sat,
https://podcast.do-not-panic.com/episodes/cxl-speckv-bridging-the-llm-memory-wall-with-speculative-fpga-disaggregation/
19. AI Post Transformers: Computation-Bandwidth-Memory Trade-offs for AI Infrastructure — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-09-computation-bandwidth-memory-trade-offs-a83f2b.mp3
20. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3
21. AI Post Transformers: FAST26: Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional — Hal Turing & Dr. Ada Shannon, Wed,
https://podcast.do-not-panic.com/episodes/fast26-bidaw-enhancing-key-value-caching-for-interactive-llm-serving-via-bidirec/
22. AI Post Transformers: FlexGen: High-Throughput LLM Inference on a Single GPU — Hal Turing & Dr. Ada Shannon, Mon,
https://podcast.do-not-panic.com/episodes/flexgen-high-throughput-llm-inference-on-a-single-gpu/
23. AI Post Transformers: xLLM: Co-Locating Online and Offline LLM Inference — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-16-xllm-co-locating-online-and-offline-llm-10bb81.mp3
24. AI Post Transformers: QuantSpec: Hierarchical KV Cache for Self-Speculative Decoding — Hal Turing & Dr. Ada Shannon, Sat,
https://podcast.do-not-panic.com/episodes/quantspec-hierarchical-kv-cache-for-self-speculative-decoding/
25. AI Post Transformers: Continuous Batching for LLM Inference: Throughput and Latency Gains — Hal Turing & Dr. Ada Shannon, Mon,
https://podcast.do-not-panic.com/episodes/continuous-batching-for-llm-inference-throughput-and-latency-gains/
26. AI Post Transformers: SGLang: Efficient Language Model Program Execution — Hal Turing & Dr. Ada Shannon, Sun,
https://podcast.do-not-panic.com/episodes/sglang-efficient-language-model-program-execution/
1. FengHuang: Next-Generation Memory Orchestration for AI Inferencing — Jiamin Li, Lei Qu, Tao Zhang, Grigory Chirkov, Shuotao Xu, Peng Cheng, Lidong Zhou, 2025
http://arxiv.org/abs/2511.10753
2. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021
https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning
3. vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023
https://scholar.google.com/scholar?q=vLLM%3A+Easy%2C+Fast%2C+and+Cheap+LLM+Serving+with+PagedAttention
4. Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Apoorv Saxena, Ameet Deshpande, et al., 2023
https://scholar.google.com/scholar?q=Sarathi%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills
5. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Authors of DistServe, 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
6. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Authors of Mooncake, 2024
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
7. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022
https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness
8. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro, 2019
https://scholar.google.com/scholar?q=Megatron-LM%3A+Training+Multi-Billion+Parameter+Language+Models+Using+Model+Parallelism
9. Fast Distributed Inference Serving for Large Language Models — Sheng Shen, Zhen Dong, Jianguo Li, et al., 2023
https://scholar.google.com/scholar?q=Fast+Distributed+Inference+Serving+for+Large+Language+Models
10. XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization — approx. recent systems/ML authors, 2025
https://scholar.google.com/scholar?q=XQuant%3A+Breaking+the+Memory+Wall+for+LLM+Inference+with+KV+Cache+Rematerialization
11. Q-hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache — approx. recent LLM inference authors, 2025
https://scholar.google.com/scholar?q=Q-hitter%3A+A+Better+Token+Oracle+for+Efficient+LLM+Inference+via+Sparse-Quantized+KV+Cache
12. KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference — approx. recent systems authors, 2025
https://scholar.google.com/scholar?q=KVSwap%3A+Disk-aware+KV+Cache+Offloading+for+Long-Context+On-device+Inference
13. Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput — approx. recent distributed inference authors, 2025
https://scholar.google.com/scholar?q=Speculative+Decoding+in+Decentralized+LLM+Inference%3A+Turning+Communication+Latency+into+Computation+Throughput
14. SLiM: Speculative Decoding with Hypothesis Reduction — approx. recent speculative decoding authors, 2025
https://scholar.google.com/scholar?q=SLiM%3A+Speculative+Decoding+with+Hypothesis+Reduction
15. Speculative Decoding and Beyond: An In-Depth Survey of Techniques — approx. survey authors, 2025
https://scholar.google.com/scholar?q=Speculative+Decoding+and+Beyond%3A+An+In-Depth+Survey+of+Techniques
16. Accelerating Transformer Model Inference through Software Optimization and Processing-in-Memory — approx. architecture authors, 2024
https://scholar.google.com/scholar?q=Accelerating+Transformer+Model+Inference+through+Software+Optimization+and+Processing-in-Memory
17. Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference — approx. review authors, 2024
https://scholar.google.com/scholar?q=Memory+Is+All+You+Need%3A+An+Overview+of+Compute-in-Memory+Architectures+for+Accelerating+Large+Language+Model+Inference
18. AI Post Transformers: CXL-SpecKV: Bridging the LLM Memory Wall with Speculative FPGA Disaggregation — Hal Turing & Dr. Ada Shannon, Sat,
https://podcast.do-not-panic.com/episodes/cxl-speckv-bridging-the-llm-memory-wall-with-speculative-fpga-disaggregation/
19. AI Post Transformers: Computation-Bandwidth-Memory Trade-offs for AI Infrastructure — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-09-computation-bandwidth-memory-trade-offs-a83f2b.mp3
20. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3
21. AI Post Transformers: FAST26: Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional — Hal Turing & Dr. Ada Shannon, Wed,
https://podcast.do-not-panic.com/episodes/fast26-bidaw-enhancing-key-value-caching-for-interactive-llm-serving-via-bidirec/
22. AI Post Transformers: FlexGen: High-Throughput LLM Inference on a Single GPU — Hal Turing & Dr. Ada Shannon, Mon,
https://podcast.do-not-panic.com/episodes/flexgen-high-throughput-llm-inference-on-a-single-gpu/
23. AI Post Transformers: xLLM: Co-Locating Online and Offline LLM Inference — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-16-xllm-co-locating-online-and-offline-llm-10bb81.mp3
24. AI Post Transformers: QuantSpec: Hierarchical KV Cache for Self-Speculative Decoding — Hal Turing & Dr. Ada Shannon, Sat,
https://podcast.do-not-panic.com/episodes/quantspec-hierarchical-kv-cache-for-self-speculative-decoding/
25. AI Post Transformers: Continuous Batching for LLM Inference: Throughput and Latency Gains — Hal Turing & Dr. Ada Shannon, Mon,
https://podcast.do-not-panic.com/episodes/continuous-batching-for-llm-inference-throughput-and-latency-gains/
26. AI Post Transformers: SGLang: Efficient Language Model Program Execution — Hal Turing & Dr. Ada Shannon, Sun,
https://podcast.do-not-panic.com/episodes/sglang-efficient-language-model-program-execution/