Sliding-Window Attention Beats Linear Attention Without Any Training

References

1. Sliding-window beats linear attention — Jolicoeur-Martineau et al., 2026 (arXiv 2608.28444)
2. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Xiao, Tian, Chen, Han, Lewis, ICLR 2024
3. Longformer — Beltagy, Peters, Cohan, 2020
4. Mistral 7B — Jiang et al., 2023
5. Gemma 2 / Gemma 3 — Gemma Team, 2024/2025
6. Transformers are RNNs — Katharopoulos et al., 2020
7. Finetuning Pretrained Transformers into RNNs — Kasai et al., 2021
8. LoLCATs — Zhang et al., ICLR 2025
9. MOHAWK / Llamba — Bick et al., 2024/2025
10. The Mamba in the Llama — Wang et al., 2024
11. Why do LLMs attend to the first token? — Barbero et al., 2025
12. Short window attention enables long-term memorization — Cabannes et al., 2026
13. SWAA — Yu et al., 2025
14. STILL — Meng et al., 2026
15. Griffin — De et al., 2024
16. H2O; SnapKV — Zhang et al.; Li et al.
17. RULER; BABILong — Hsieh et al.; Kuratov et al., 2024
18. Gated Delta Networks; Delta Rule — Yang et al., 2024/2025