Why cascades hurt
Each stage needs the finished result of the previous one. Toggle to see the memory traffic change.
Same shape, different workloads
Cascaded reductions named in the episode.
Online softmax, step by step
Scores 1, 3 arrive in tile A; a third score arrives in tile B. Drag the slider to change it (episode example: 5).
Third score x₃ = 5
Rescale the output accumulator tooIllustrative v values
Value vectors v = [2, −1, 4] are made up to show the arithmetic; the correction factor is the paper's.
Merging partial states across blocks (FlashDecoding-style)
The algebraic test: split F into G ⊗ H
The per-element function splits into an input-only part and an earlier-result-only part.
Three conditions on each reduction
Lowering chain
Single-Segment vs Multi-SegmentIllustrative
Decode has one query row; splitting the key sequence recovers parallelism.
segments 4
Reported results
Only numbers stated in the episode are plotted; endpoints of reported ranges, not per-workload data.
How far does the evidence go?Editorial assessment
Qualitative reading of the episode's discussion, not a paper-reported score. Hover a cell. Gray = not applicable.
References
- RedFuser (Tang et al., 2026), arXiv:2603.10026
- FlashAttention (Dao et al., 2022)
- Welder (Shi et al., 2023)
- Triton (Tillet, Kung, Cox, 2019)
- TVM (Chen et al., 2018)
- Ansor (Zheng et al., 2020)
- AutoTVM (Chen et al., 2018)
- FlashAttention-3 (Shah et al., 2024)
- PyTorch 2 / TorchInductor (Ansel et al., 2024)
- Online normalizer calculation for softmax (Milakov, Gimelshein, 2018)
- Self-attention Does Not Need O(n²) Memory (Rabe, Staats, 2021)
- Flash-Decoding / FlashAttention-2
- Ring Attention (Liu, Zaharia, Abbeel, 2023)
- FlashInfer (Ye et al., 2025)
- FlexAttention (Dong et al., 2024)
- TileLang (Wang et al., 2025)
- Mirage (Wu, Cheng et al., 2025)
- Neptune (Zhao et al., 2025)
- Souffle (Zheng et al., 2024)
- Lean Attention (Sanovar et al., 2024)
- DeepGEMM / DeepSeek FP8 fine-grained scaling (2025)
Only the RedFuser paper has an arXiv ID in the source list; the other entries were provided as scholar searches.