LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu · SambaNova Systems · 2026 · builds on Speculative Prefill (ICML 2025) and AttnComp (2025)
Extractive retrieve-then-read pipeline from episode
Step through the six stages. The compressor is the retrieval half; the target model is the reader.
Why prompts hurt: prefill cost vs length Illustrative
Relative, normalized to 120k tokens. Prefill grows at least linearly; attention itself quadratically. The paper measures no latency.
Chunk scoring and selection Illustrative
200 chunks of ~1k tokens (chunk size 1024 is the paper's Code-Debug setting). Scores below are synthetic, to show the mechanism. Query sits at the end.
Stopping rule: cumulative score ≥ 0.95 or budget Illustrative
On RULER single-needle, selection stopped near 2k tokens vs a 16k budget with 100% accuracy. Spread-out evidence stops later, or gets cut. On LongBench v2 the authors went budget-only.
Scorer = frozen Llama-3.1-8B-Instruct prefix + one cross-attention layer from episode
Hover a layer. First 13 of 32 layers frozen; one trainable cross-attention layer (~0.5% of parameters); later layers are not used.
Hover or tap a block.
Question tokens attend over context chunks Illustrative
Heads are averaged to give one score per chunk. Values here are synthetic; the bottom row is the true mean of the rows above.
Two-stage synthetic training from episode
Click a stage. Labels come from dataset annotations; no LLM labeling.
What changed from AttnComp from episode
Reported results
Does "exceeds full context" survive scrutiny? Illustrative
Same budget of 16 chunks. Middle truncation to a 120-chunk window cuts exactly where the evidence lives, so the baseline is handicapped on the longest repos.
Confidence interval calculator arithmetic
Wald 95% interval: 1.96·√(p(1−p)/n). The episode infers n≈394 from score granularity (not stated in the paper).
Chunk size swings accuracy ~20 points from episode
Code-Debug, R1 target. Speculative Prefill was run at chunk 128 and never rerun at 1024.