LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
Oct 13, 2026
This episode examines LongAttnComp, a method that uses a small trained scorer to pick about 16,000 relevant tokens out of roughly 115,000. The goal is for other companies' models to find code bugs about as well as with the full input. It explains why long prompts are costly (prefill dominates time-to-first-token) and how extractive compression differs from summarization. It also ties the approach to Speculative Prefill, AttnComp, and SnapKV, which score chunks by query-aligned attention, and frames compression as the retrieval half of a retrieve-then-read pipeline. The method freezes the first 13 layers of Llama-3.1-8B-Instruct, adds one trainable cross-attention layer, and trains it on synthetic needle-in-a-haystack data. It keeps the top-scoring chunks under a token budget and sends them to the target model in their original order. Throughout, the discussion stays skeptical of the claim that compression "exceeds" full context. The baseline is middle truncation, which cuts the region where models already read poorly. The paper also never measures latency itself, and the training data is synthetic, so transfer to real inputs is open.
Interactive Visualization: LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
Sources:
1. LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning — Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu, 2026
http://arxiv.org/abs/2606.01336
2. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023
https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models
3. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (ACL 2024)
https://scholar.google.com/scholar?q=LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression
4. Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation — Jingyu Liu, Beidi Chen, Ce Zhang, 2025
https://scholar.google.com/scholar?q=Speculative+Prefill%3A+Turbocharging+TTFT+with+Lightweight+and+Training-Free+Token+Importance+Estimation
5. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation — Fangyuan Xu, Weijia Shi, Eunsol Choi, 2023
https://scholar.google.com/scholar?q=RECOMP%3A+Improving+Retrieval-Augmented+LMs+with+Compression+and+Selective+Augmentation
6. MuSiQue: Multihop Questions via Single-hop Question Composition — Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal, 2022
https://scholar.google.com/scholar?q=MuSiQue%3A+Multihop+Questions+via+Single-hop+Question+Composition
7. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2WikiMultiHopQA) — Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa, 2020
https://scholar.google.com/scholar?q=Constructing+A+Multi-hop+QA+Dataset+for+Comprehensive+Evaluation+of+Reasoning+Steps+%282WikiMultiHopQA%29
8. From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data — Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, Dimitris Papailiopoulos, 2024
https://scholar.google.com/scholar?q=From+Artificial+Needles+to+Real+Haystacks%3A+Improving+Retrieval+Capabilities+in+LLMs+by+Finetuning+on+Synthetic+Data
9. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks — Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li, 2024
https://scholar.google.com/scholar?q=LongBench+v2%3A+Towards+Deeper+Understanding+and+Reasoning+on+Realistic+Long-context+Multitasks
10. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, et al., 2024
https://scholar.google.com/scholar?q=LLMLingua-2%3A+Data+Distillation+for+Efficient+and+Faithful+Task-Agnostic+Prompt+Compression
11. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingqi Huang, Bowen Yang, et al., 2024
https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation
12. Make Your LLM Fully Utilize the Context (IN2 training) — Shengnan An, Zexiong Ma, Zeqi Lin, et al., 2024
https://scholar.google.com/scholar?q=Make+Your+LLM+Fully+Utilize+the+Context+%28IN2+training%29
13. Provence: Efficient and Robust Context Pruning for Retrieval-Augmented Generation — Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, Stéphane Clinchant, 2025
https://scholar.google.com/scholar?q=Provence%3A+Efficient+and+Robust+Context+Pruning+for+Retrieval-Augmented+Generation
14. Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models — Shubhangi Upasani, Ravi Shanker Raju, Bo Li, Mengmeng Ji, et al., 2026
https://scholar.google.com/scholar?q=Cross-Family+Speculative+Prefill%3A+Training-Free+Long-Context+Compression+with+Small+Draft+Models
15. Compressing Context to Enhance Inference Efficiency of Large Language Models (Selective Context) — Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin, 2023
https://scholar.google.com/scholar?q=Compressing+Context+to+Enhance+Inference+Efficiency+of+Large+Language+Models+%28Selective+Context%29
16. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2024
https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29
17. Retrieval Head Mechanistically Explains Long-Context Factuality — Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, Yao Fu, 2024
https://scholar.google.com/scholar?q=Retrieval+Head+Mechanistically+Explains+Long-Context+Factuality
Interactive Visualization: LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
1. LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning — Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu, 2026
http://arxiv.org/abs/2606.01336
2. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023
https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models
3. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (ACL 2024)
https://scholar.google.com/scholar?q=LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression
4. Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation — Jingyu Liu, Beidi Chen, Ce Zhang, 2025
https://scholar.google.com/scholar?q=Speculative+Prefill%3A+Turbocharging+TTFT+with+Lightweight+and+Training-Free+Token+Importance+Estimation
5. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation — Fangyuan Xu, Weijia Shi, Eunsol Choi, 2023
https://scholar.google.com/scholar?q=RECOMP%3A+Improving+Retrieval-Augmented+LMs+with+Compression+and+Selective+Augmentation
6. MuSiQue: Multihop Questions via Single-hop Question Composition — Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal, 2022
https://scholar.google.com/scholar?q=MuSiQue%3A+Multihop+Questions+via+Single-hop+Question+Composition
7. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2WikiMultiHopQA) — Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa, 2020
https://scholar.google.com/scholar?q=Constructing+A+Multi-hop+QA+Dataset+for+Comprehensive+Evaluation+of+Reasoning+Steps+%282WikiMultiHopQA%29
8. From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data — Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, Dimitris Papailiopoulos, 2024
https://scholar.google.com/scholar?q=From+Artificial+Needles+to+Real+Haystacks%3A+Improving+Retrieval+Capabilities+in+LLMs+by+Finetuning+on+Synthetic+Data
9. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks — Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li, 2024
https://scholar.google.com/scholar?q=LongBench+v2%3A+Towards+Deeper+Understanding+and+Reasoning+on+Realistic+Long-context+Multitasks
10. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, et al., 2024
https://scholar.google.com/scholar?q=LLMLingua-2%3A+Data+Distillation+for+Efficient+and+Faithful+Task-Agnostic+Prompt+Compression
11. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingqi Huang, Bowen Yang, et al., 2024
https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation
12. Make Your LLM Fully Utilize the Context (IN2 training) — Shengnan An, Zexiong Ma, Zeqi Lin, et al., 2024
https://scholar.google.com/scholar?q=Make+Your+LLM+Fully+Utilize+the+Context+%28IN2+training%29
13. Provence: Efficient and Robust Context Pruning for Retrieval-Augmented Generation — Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, Stéphane Clinchant, 2025
https://scholar.google.com/scholar?q=Provence%3A+Efficient+and+Robust+Context+Pruning+for+Retrieval-Augmented+Generation
14. Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models — Shubhangi Upasani, Ravi Shanker Raju, Bo Li, Mengmeng Ji, et al., 2026
https://scholar.google.com/scholar?q=Cross-Family+Speculative+Prefill%3A+Training-Free+Long-Context+Compression+with+Small+Draft+Models
15. Compressing Context to Enhance Inference Efficiency of Large Language Models (Selective Context) — Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin, 2023
https://scholar.google.com/scholar?q=Compressing+Context+to+Enhance+Inference+Efficiency+of+Large+Language+Models+%28Selective+Context%29
16. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2024
https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29
17. Retrieval Head Mechanistically Explains Long-Context Factuality — Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, Yao Fu, 2024
https://scholar.google.com/scholar?q=Retrieval+Head+Mechanistically+Explains+Long-Context+Factuality
Interactive Visualization: LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning