Hierarchical Sparse Attention for Infinite Context

This episode explores the HiLS paper, which tackles the central long-context transformer problem: how to preserve normal-context quality while avoiding the exploding compute and KV-cache costs of dense attention at extreme sequence lengths. It explains the paper’s core mechanism of hierarchical sparse attention, where the model learns summary keys for context chunks, retrieves the most relevant chunks for each query, attends within them, and then keeps the retrieval scores in the forward pass so the chunk selector is trained directly by language-model loss. The discussion contrasts this with older sparse schemes, sliding-window attention, and positional stretching methods like YaRN, arguing that better retrieval inside attention matters as much as longer positional extrapolation or extra continued pretraining. Listeners would find it interesting because it connects the architecture details to concrete 7B OLMo 3 results on long-context benchmarks like RULER and LongBench, framing HiLS as a serious attempt to reach million-token-class context without paying full dense-attention cost.

Sources:
1. Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling — Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, Tian Liang, Huayang Li, Lei Zhu, Yan Wang, Sirui Han, Yushi Bai, Kewei Tu, Haitao Mi, Leo Liang, 2026
http://arxiv.org/abs/2607.02980
2. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — Ofir Press, Noah A. Smith, Mike Lewis, 2021
https://scholar.google.com/scholar?q=Train+Short%2C+Test+Long%3A+Attention+with+Linear+Biases+Enables+Input+Length+Extrapolation
3. Extending Context Window of Large Language Models via Positional Interpolation — Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian, 2023
https://scholar.google.com/scholar?q=Extending+Context+Window+of+Large+Language+Models+via+Positional+Interpolation
4. YaRN: Efficient Context Window Extension of Large Language Models — Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole, 2023
https://scholar.google.com/scholar?q=YaRN%3A+Efficient+Context+Window+Extension+of+Large+Language+Models
5. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, 2024
https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F
6. Random-Access Infinite Context Length for Transformers — Amirkeivan Mohtashami and Martin Jaggi, 2023
https://scholar.google.com/scholar?q=Random-Access+Infinite+Context+Length+for+Transformers
7. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan et al., 2025
https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention
8. Hardware-Aligned Hierarchical Sparse Attention for Efficient Long-Term Memory Access — Xiang Hu, Jiaqi Leng, Jun Zhao, Kewei Tu, and Wei Wu, 2026
https://scholar.google.com/scholar?q=Hardware-Aligned+Hierarchical+Sparse+Attention+for+Efficient+Long-Term+Memory+Access
9. DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention — Yuxiang Huang et al., 2026
https://scholar.google.com/scholar?q=DashAttention%3A+Differentiable+and+Adaptive+Sparse+Hierarchical+Attention
10. Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models — Jiaqi Leng, Xiang Hu, Junxiong Wang, Jianguo Li, Wei Wu, and Yucheng Lu, 2026
https://scholar.google.com/scholar?q=Understanding+and+Improving+Length+Generalization+in+Hierarchical+Sparse+Attention+Models
11. RingAttention with Blockwise Transformers for Near-Infinite Context — Hao Liu, Matei Zaharia, and Pieter Abbeel, 2024
https://scholar.google.com/scholar?q=RingAttention+with+Blockwise+Transformers+for+Near-Infinite+Context
12. MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling — Wenhao An et al. (MiniCPM Team), 2026
https://arxiv.org/abs/2602.11761
13. Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection — Dongwon Jo et al., 2026
https://arxiv.org/abs/2602.03216
14. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention — Huiqiang Jiang et al., 2024
https://arxiv.org/abs/2407.02490
15. Long-Context Generalization with Sparse Attention — Pavlo Vasylenko, Marcos Treviso, Andre F. T. Martins, 2025
https://arxiv.org/abs/2506.16640
16. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu et al., 2025
https://arxiv.org/abs/2502.00299
17. AI Post Transformers: Long Context Pre-Training with Lighthouse Attention — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-13-long-context-pre-training-with-lighthous-e85bbe.mp3
18. AI Post Transformers: MiniMax Sparse Attention at Million-Token Scale — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-13-minimax-sparse-attention-at-million-toke-300108.mp3
19. AI Post Transformers: MiA-Signature and Global Activation for Long Context — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-13-mia-signature-and-global-activation-for-5ad62f.mp3
20. AI Post Transformers: AllMem for Efficient Long-Context Modeling — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-12-allmem-for-efficient-long-context-modeli-7474db.mp3
21. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp3