RedFuser: Automatic Fusion of Cascaded Reductions for AI Accelerators
Oct 12, 2026
This episode examines RedFuser, an Alibaba Cloud framework that automatically fuses cascaded reductions, such as softmax feeding a matrix multiply, into a single GPU kernel. It starts with why chained reductions resist standard compiler fusion: each stage needs the finished result of the one before, which forces repeated memory reads and prevents overlap. The discussion then works through the online-softmax rescaling trick with a small numeric example. Multiplying the running denominator by exp(old max − new max) gives the same answer as recomputing from scratch, and the same correction extends to the output accumulator and to merging partial results across blocks, as in FlashAttention and FlashDecoding. It also notes that the paper doesn't appear to cite the earlier online-softmax and chunked-attention work. The central question is whether a compiler can derive this kind of algebraic rewrite for patterns no one has hand-derived, rather than only tuning schedules for a fixed algorithm. That question matters to anyone who wants new attention variants, MoE routing, or FP8 quantization pipelines to get fast kernels without a new manual derivation each time.
Interactive Visualization: RedFuser: Automatic Fusion of Cascaded Reductions for AI Accelerators
Sources:
1. RedFuser: An Automatic Operator Fusion Framework for Cascaded Reductions on AI Accelerators — Xinsheng Tang, Yangcheng Li, Nan Wang, Zhiyi Shu, Xingyu Ling, Junna Xing, Peng Zhou, Qiang Liu, 2026
http://arxiv.org/abs/2603.10026v1
2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022
https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness
3. Welder: Scheduling Deep Learning Memory Access via Tile-graph — Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, Lidong Zhou, 2023
https://scholar.google.com/scholar?q=Welder%3A+Scheduling+Deep+Learning+Memory+Access+via+Tile-graph
4. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations — Philippe Tillet, H. T. Kung, David Cox, 2019
https://scholar.google.com/scholar?q=Triton%3A+An+Intermediate+Language+and+Compiler+for+Tiled+Neural+Network+Computations
5. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy, 2018
https://scholar.google.com/scholar?q=TVM%3A+An+Automated+End-to-End+Optimizing+Compiler+for+Deep+Learning
6. Ansor: Generating High-Performance Tensor Programs for Deep Learning — Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph E. Gonzalez, Ion Stoica, 2020
https://scholar.google.com/scholar?q=Ansor%3A+Generating+High-Performance+Tensor+Programs+for+Deep+Learning
7. Learning to Optimize Tensor Programs (AutoTVM) — Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy, 2018
https://scholar.google.com/scholar?q=Learning+to+Optimize+Tensor+Programs+%28AutoTVM%29
8. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024
https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision
9. Triton / TorchInductor: PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation — Jason Ansel and the PyTorch team, 2024
https://scholar.google.com/scholar?q=Triton+%2F+TorchInductor%3A+PyTorch+2%3A+Faster+Machine+Learning+Through+Dynamic+Python+Bytecode+Transformation+and+Graph+Compilation
10. Online normalizer calculation for softmax — Maxim Milakov, Natalia Gimelshein, 2018
https://scholar.google.com/scholar?q=Online+normalizer+calculation+for+softmax
11. Self-attention Does Not Need O(n^2) Memory — Markus N. Rabe, Charles Staats, 2021
https://scholar.google.com/scholar?q=Self-attention+Does+Not+Need+O%28n%5E2%29+Memory
12. Flash-Decoding for long-context inference (and FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning) — Tri Dao; Daniel Haziza, Francisco Massa, Grigory Sizov and collaborators, 2023
https://scholar.google.com/scholar?q=Flash-Decoding+for+long-context+inference+%28and+FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning%29
13. Ring Attention with Blockwise Transformers for Near-Infinite Context — Hao Liu, Matei Zaharia, Pieter Abbeel, 2023
https://scholar.google.com/scholar?q=Ring+Attention+with+Blockwise+Transformers+for+Near-Infinite+Context
14. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — Zihao Ye et al., 2025
https://scholar.google.com/scholar?q=FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving
15. FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention — Juechu Dong et al. (PyTorch team), 2024
https://scholar.google.com/scholar?q=FlexAttention%3A+The+Flexibility+of+PyTorch+with+the+Performance+of+FlashAttention
16. TileLang: A Composable Tiled Programming Model for AI Systems — Lei Wang et al., 2025
https://scholar.google.com/scholar?q=TileLang%3A+A+Composable+Tiled+Programming+Model+for+AI+Systems
17. Mirage: A Multi-Level Superoptimizer for Tensor Programs — Mengdi Wu, Xinhao Cheng, et al., 2025
https://scholar.google.com/scholar?q=Mirage%3A+A+Multi-Level+Superoptimizer+for+Tensor+Programs
18. Neptune: Advanced ML Operator Fusion for Locality and Parallelism on GPUs — Yifan Zhao et al., 2025
https://scholar.google.com/scholar?q=Neptune%3A+Advanced+ML+Operator+Fusion+for+Locality+and+Parallelism+on+GPUs
19. Souffle: Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions — Bojian Zheng et al., 2024
https://scholar.google.com/scholar?q=Souffle%3A+Optimizing+Deep+Learning+Inference+via+Global+Analysis+and+Tensor+Expressions
20. Lean Attention / Stream-K style decomposition for decode attention — Rya Sanovar et al., 2024
https://scholar.google.com/scholar?q=Lean+Attention+%2F+Stream-K+style+decomposition+for+decode+attention
21. DeepGEMM / DeepSeek FP8 fine-grained scaling — DeepSeek-AI, 2025
https://scholar.google.com/scholar?q=DeepGEMM+%2F+DeepSeek+FP8+fine-grained+scaling
Interactive Visualization: RedFuser: Automatic Fusion of Cascaded Reductions for AI Accelerators
1. RedFuser: An Automatic Operator Fusion Framework for Cascaded Reductions on AI Accelerators — Xinsheng Tang, Yangcheng Li, Nan Wang, Zhiyi Shu, Xingyu Ling, Junna Xing, Peng Zhou, Qiang Liu, 2026
http://arxiv.org/abs/2603.10026v1
2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022
https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness
3. Welder: Scheduling Deep Learning Memory Access via Tile-graph — Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, Lidong Zhou, 2023
https://scholar.google.com/scholar?q=Welder%3A+Scheduling+Deep+Learning+Memory+Access+via+Tile-graph
4. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations — Philippe Tillet, H. T. Kung, David Cox, 2019
https://scholar.google.com/scholar?q=Triton%3A+An+Intermediate+Language+and+Compiler+for+Tiled+Neural+Network+Computations
5. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy, 2018
https://scholar.google.com/scholar?q=TVM%3A+An+Automated+End-to-End+Optimizing+Compiler+for+Deep+Learning
6. Ansor: Generating High-Performance Tensor Programs for Deep Learning — Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph E. Gonzalez, Ion Stoica, 2020
https://scholar.google.com/scholar?q=Ansor%3A+Generating+High-Performance+Tensor+Programs+for+Deep+Learning
7. Learning to Optimize Tensor Programs (AutoTVM) — Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy, 2018
https://scholar.google.com/scholar?q=Learning+to+Optimize+Tensor+Programs+%28AutoTVM%29
8. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024
https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision
9. Triton / TorchInductor: PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation — Jason Ansel and the PyTorch team, 2024
https://scholar.google.com/scholar?q=Triton+%2F+TorchInductor%3A+PyTorch+2%3A+Faster+Machine+Learning+Through+Dynamic+Python+Bytecode+Transformation+and+Graph+Compilation
10. Online normalizer calculation for softmax — Maxim Milakov, Natalia Gimelshein, 2018
https://scholar.google.com/scholar?q=Online+normalizer+calculation+for+softmax
11. Self-attention Does Not Need O(n^2) Memory — Markus N. Rabe, Charles Staats, 2021
https://scholar.google.com/scholar?q=Self-attention+Does+Not+Need+O%28n%5E2%29+Memory
12. Flash-Decoding for long-context inference (and FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning) — Tri Dao; Daniel Haziza, Francisco Massa, Grigory Sizov and collaborators, 2023
https://scholar.google.com/scholar?q=Flash-Decoding+for+long-context+inference+%28and+FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning%29
13. Ring Attention with Blockwise Transformers for Near-Infinite Context — Hao Liu, Matei Zaharia, Pieter Abbeel, 2023
https://scholar.google.com/scholar?q=Ring+Attention+with+Blockwise+Transformers+for+Near-Infinite+Context
14. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — Zihao Ye et al., 2025
https://scholar.google.com/scholar?q=FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving
15. FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention — Juechu Dong et al. (PyTorch team), 2024
https://scholar.google.com/scholar?q=FlexAttention%3A+The+Flexibility+of+PyTorch+with+the+Performance+of+FlashAttention
16. TileLang: A Composable Tiled Programming Model for AI Systems — Lei Wang et al., 2025
https://scholar.google.com/scholar?q=TileLang%3A+A+Composable+Tiled+Programming+Model+for+AI+Systems
17. Mirage: A Multi-Level Superoptimizer for Tensor Programs — Mengdi Wu, Xinhao Cheng, et al., 2025
https://scholar.google.com/scholar?q=Mirage%3A+A+Multi-Level+Superoptimizer+for+Tensor+Programs
18. Neptune: Advanced ML Operator Fusion for Locality and Parallelism on GPUs — Yifan Zhao et al., 2025
https://scholar.google.com/scholar?q=Neptune%3A+Advanced+ML+Operator+Fusion+for+Locality+and+Parallelism+on+GPUs
19. Souffle: Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions — Bojian Zheng et al., 2024
https://scholar.google.com/scholar?q=Souffle%3A+Optimizing+Deep+Learning+Inference+via+Global+Analysis+and+Tensor+Expressions
20. Lean Attention / Stream-K style decomposition for decode attention — Rya Sanovar et al., 2024
https://scholar.google.com/scholar?q=Lean+Attention+%2F+Stream-K+style+decomposition+for+decode+attention
21. DeepGEMM / DeepSeek FP8 fine-grained scaling — DeepSeek-AI, 2025
https://scholar.google.com/scholar?q=DeepGEMM+%2F+DeepSeek+FP8+fine-grained+scaling
Interactive Visualization: RedFuser: Automatic Fusion of Cascaded Reductions for AI Accelerators