YOCO: Caching Once for Decoder-Decoder Long-Context Language Models
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei · Microsoft Research & Tsinghua University · May 2024
arXiv 2405.05254one global KV cache≈L-fold cache cutprefill skips cross-decoder