AttnComp: Attention-Guided Adaptive Context Compression for RAG

RAG pipeline with a compressor in the middle

A retriever fetches 100 chunks (~100 words each). AttnComp decides how many survive, including zero.

Fixed budget vs adaptive keep Illustrative

Two of 100 chunks matter. A fixed top-5 budget can miss the bridge document; an adaptive rule keeps what the attention needs.

Bridge document: independent vs joint scoring Illustrative

"Where was the eldest brother of this person born?" The bridge document names the brother but never mentions the subject.

Attention as a shared budget Illustrative

Each query token spreads 1.0 across earlier tokens (rows sum to 1). Column averages become scores, so documents compete.

Top-P selection, nucleus-sampling style

Start the running total at the instruction's score. Add documents high to low until p is reached, or a score falls below 0.01.

The compressor, step by step

Mostly borrowed: 13 frozen layers plus one cross-attention layer. About half a percent of parameters is trained.

Where the training labels come from

The untrained compressor labels its own data, then a reader check filters it.

Results Reported

E5-base-v2 retrieves top 100 chunks from the Dec 2018 Wikipedia dump; Llama-3.1-8B-Instruct reads for every method. Single run, no significance tests.

Signals behind the design Reported

The authors' observations; the episode cautions they are motivation, not verified mechanisms.

Qwen reader check: accuracy vs F1 (direction only)

Only HotpotQA improves in accuracy; F1 rises on all five. The paper does not explain the divergence.

Open questions the episode flags