Attention Sinks Enable Streaming Language Models Over Endless Input

Xiao, Tian, Chen, Han, Lewis · MIT, Meta AI, CMU, NVIDIA · ICLR 2024 · "Efficient Streaming Language Models with Attention Sinks"
arXiv 2309.17453No fine-tuningStable ≠ remembers

References

  1. Xiao et al. — Efficient Streaming Language Models with Attention Sinks, 2023 (ICLR 2024). arXiv:2309.17453
  2. Sun et al. — Massive Activations in Large Language Models, 2024. arXiv:2402.17762
  3. Darcet et al. — Vision Transformers Need Registers, 2023. arXiv:2309.16588
  4. Bondarenko et al. — Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing, 2023. arXiv:2306.12929
  5. Zhang et al. — H2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs, 2023. arXiv:2306.14048
  6. Beltagy et al. — Longformer: The Long-Document Transformer, 2020. arXiv:2004.05150
  7. Pope et al. — Efficiently Scaling Transformer Inference, 2022. arXiv:2211.05102
  8. Gu et al. — When Attention Sink Emerges in Language Models: An Empirical View, 2024. arXiv:2410.10781
  9. Miller — Attention is Off by One (Softmax1), 2023. Scholar
  10. Han et al. — LM-Infinite, 2023. arXiv:2308.16137
  11. Li et al. — SnapKV, 2024. arXiv:2404.14469
  12. Liu et al. — Lost in the Middle, 2023. arXiv:2307.03172
  13. Kwon et al. — PagedAttention, 2023. arXiv:2309.06180
  14. Gu, Dao — Mamba, 2023. arXiv:2312.00752