1
00:00:01,000 --> 00:00:07,640
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,640 --> 00:00:44,235
[Hal Turing] Today's paper is AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation. The first author is Lvzhou Luo, with Yixuan Cao and Ping Luo, from the Institute of Computing Technology at the Chinese Academy of Sciences and the University of Chinese Academy of Sciences. It's an arXiv paper from September 2025. The question is whether a language model's own middle-layer attention, lightly fine-tuned, can decide which retrieved documents to keep, how many to keep, and how much to trust the answer.

3
00:00:44,235 --> 00:00:56,820
[Dr. Ada Shannon] The tension is that the usual fixes each give something up. Summarizer-style compressors are slow, and cheap scorers judge each piece alone. This paper wants speed, adaptivity, and awareness of the whole context at once.

4
00:00:56,820 --> 00:01:09,313
[Hal Turing] Let's set the scene. In retrieval-augmented generation, a retriever fetches documents and a reader LLM answers from them. Here that means 100 chunks of about 100 words each per question. What goes wrong?

5
00:01:09,313 --> 00:01:37,037
[Dr. Ada Shannon] Three things. Irrelevant text can mislead the reader. Yoran and colleagues showed in 2024 that noisy retrieval can push accuracy below having no retrieval at all. Second, key facts get buried; the Lost in the Middle paper by Nelson Liu at Stanford, 2023, found models use information in the middle of a long context worst. Third, every extra token costs compute. Picture a multi-hop question where maybe two of those hundred chunks matter.

6
00:01:37,037 --> 00:01:38,245
[Hal Turing] Multi-hop meaning?

7
00:01:38,245 --> 00:01:54,592
[Dr. Ada Shannon] Single-hop questions, like Natural Questions or PopQA, need one passage. Multi-hop ones, like HotpotQA, 2WikiMultiHopQA, and MuSiQue, need facts joined across passages. And zero-shot here means the compressor was never trained on that dataset.

8
00:01:54,592 --> 00:01:56,914
[Hal Turing] So how do people compress the context today?

9
00:01:56,914 --> 00:02:37,316
[Dr. Ada Shannon] Two families. Abstractive compressors rewrite or summarize, decoding token by token, which is slow. Extractive ones pick spans from the original text, which is faster and keeps verbatim evidence. RECOMP, from Fangyuan Xu at UT Austin in 2023, has both flavors, plus the idea of returning nothing when the documents don't help. LongLLMLingua from Microsoft drops tokens. CompAct is abstractive. EXIT and Provence are extractive, at sentence level. We'll express compression as a rate: tokens before divided by tokens after, so 17x means about a seventeenth survives.

10
00:02:37,316 --> 00:02:39,360
[Hal Turing] And the extractive ones fall short where?

11
00:02:39,360 --> 00:03:01,233
[Dr. Ada Shannon] Two places. Many use a fixed budget, so they keep five pieces whether zero or twelve matter. And they score each sentence alone. Take a question like "where was the eldest brother of this person born?" One document names the brother but never mentions the question's subject. Scored alone, it looks irrelevant. It's a bridge document, and it only matters next to its neighbor.

12
00:03:01,233 --> 00:03:05,134
[Hal Turing] Now the attention part. Why would attention say anything about relevance?

13
00:03:05,134 --> 00:03:42,100
[Dr. Ada Shannon] Think of attention as a budget. Each token distributes a total of 1.0 across earlier tokens. Since everything shares one budget, a document's score depends on its competitors. That's what makes it joint rather than independent. Two caveats. When nothing deserves attention, the surplus has to go somewhere. Xiao and colleagues at MIT, 2023, showed that initial tokens soak it up, an attention sink. And Sarthak Jain and Byron Wallace, in their 2019 paper Attention is not Explanation, warned that attention isn't reliably an explanation, so any use needs validating.

14
00:03:42,100 --> 00:03:44,979
[Hal Turing] Is there prior work reading relevance off attention?

15
00:03:44,979 --> 00:04:12,797
[Dr. Ada Shannon] Wenhao Wu and colleagues in 2024 found sparse retrieval heads that copy relevant content from long context during generation. Fu and colleagues, 2024, used head-level behavior for KV-cache compression. Izacard and Grave, from Facebook AI Research, 2021, used a reader's cross-attention as a passage-relevance signal to train retrievers, in Distilling Knowledge from Reader to Retriever. AttnComp doesn't cite that one. I'll park the comparison for later.

16
00:04:12,797 --> 00:04:16,094
[Hal Turing] What does this paper itself observe that motivates the design?

17
00:04:16,094 --> 00:05:00,537
[Dr. Ada Shannon] Three observations, and they're the authors' motivation, not verified mechanisms. First, in middle layers, heads concentrate on the evidence; they checked this on LooGLE. Second, attention is more concentrated for short-dependency questions. Reaching 80 percent of the attention takes the top 39 sentences for short-dependency questions versus the top 63 for long-dependency ones. Third, attention on the first token rises when the context is less relevant. With random documents it climbs well above the retrieved settings, from roughly 0.25 to somewhere around 0.4 to 0.57, depending on the dataset. I'd read those figure values cautiously.

18
00:05:00,537 --> 00:05:03,370
[Hal Turing] So the first token is a "nothing here" signal.

19
00:05:03,370 --> 00:05:27,751
[Dr. Ada Shannon] That's the seed of the idea. Now the selection rule, borrowed from nucleus sampling by Holtzman and colleagues, 2019. Sort documents by attention score. Start a running total at the score of the fixed instruction at the start of the prompt. Add documents from highest to lowest until the total reaches p, which is 0.95, or until a document's score falls below 0.01.

20
00:05:27,751 --> 00:05:28,865
[Hal Turing] Give me numbers.

21
00:05:28,865 --> 00:05:52,550
[Dr. Ada Shannon] Say the instruction holds 0.2. Documents score 0.5, 0.2, and 0.05. The running total goes 0.2, then 0.7, then 0.9, then 0.95. Three documents kept. If one document dominated, you'd stop early. If the instruction alone reached 0.95, you'd keep nothing, and the prompt simply has no documents.

22
00:05:52,550 --> 00:05:56,776
[Hal Turing] That's the adaptive part. You keep as many as the attention needs, including zero.

23
00:05:56,776 --> 00:06:03,324
[Dr. Ada Shannon] Right. And the instruction's share doubles as a trust signal, which we'll get to. First, how they actually build it.

24
00:06:03,324 --> 00:06:06,157
[Hal Turing] So mechanically, what is the compressor? Is it a new model?

25
00:06:06,157 --> 00:06:27,937
[Dr. Ada Shannon] Mostly borrowed. They take the first 13 layers of Llama-3.1-8B-Instruct and freeze them. On top goes one cross-attention layer. Its queries come from the question tokens, its keys from the whole retrieved context. Its projections are initialized from 16 heads of layer 14. Only about half a percent of the parameters get trained.

26
00:06:27,937 --> 00:06:29,423
[Hal Turing] And the score for a document?

27
00:06:29,423 --> 00:06:54,454
[Dr. Ada Shannon] Average the head attention over the query tokens, then sum over the tokens in each document. The key point is that all 100 documents go through the encoder together. That's what context-aware means mechanically. Provence, RECOMP and LongLLMLingua score each piece separately, so the eldest-brother document in their appendix case study, about Arellano Félix, looks irrelevant until the birthdate document sits next to it.

28
00:06:54,454 --> 00:06:58,912
[Hal Turing] Then Top-P runs on those scores. Why does the instruction count in the sum?

29
00:06:58,912 --> 00:07:18,185
[Dr. Ada Shannon] Because attention is a fixed budget. If the instruction already holds most of the mass, little is left for documents, and the loop can stop with nothing kept. That's what lets the number kept vary: HotpotQA keeps 7.5 documents on average and PopQA 3.7, with a range from zero to 23.

30
00:07:18,185 --> 00:07:21,761
[Hal Turing] If the base model already has sink behavior, why fine-tune at all?

31
00:07:21,761 --> 00:07:43,913
[Dr. Ada Shannon] Because untrained attention still gives high scores to irrelevant documents, especially when everything is irrelevant. So they add two binary cross-entropy terms. One is per document. The other is on the instruction, which is labeled 1 exactly when no document is relevant. The weight is 0.8. After training, the instruction behaves like a learned sink that says nothing here is relevant.

32
00:07:43,913 --> 00:07:45,306
[Hal Turing] Where do the labels come from?

33
00:07:45,306 --> 00:08:16,838
[Dr. Ada Shannon] From the untrained compressor itself. Shuffle the 100 documents three times, run iterative Top-P at 0.95 until nothing more drops, and keep the intersection as the relevant set. Then Llama-3.1-8B-Instruct checks it against the ground-truth answer. If the answer from that set is correct, it's a training example. If it fails but the full 100 documents succeed, the example is discarded. If both fail, they swap the set for random corpus documents and label everything irrelevant.

34
00:08:16,838 --> 00:08:20,089
[Hal Turing] Interesting. We'll come back to that. What was the training run?

35
00:08:20,089 --> 00:08:35,693
[Dr. Ada Shannon] 8,000 HotpotQA questions, 100 documents each, 2,000 of them all-irrelevant. Eight epochs, learning rate 2e-4, document order reshuffled each epoch. About four hours on four RTX 4090s.

36
00:08:35,693 --> 00:08:36,900
[Hal Turing] And the test setup?

37
00:08:36,900 --> 00:09:04,346
[Dr. Ada Shannon] E5-base-v2 retrieves the top 100 chunks from the December 2018 Wikipedia dump. Five datasets: NQ and PopQA single-hop, HotpotQA, 2Wiki and MuSiQue multi-hop. Llama-3.1-8B-Instruct reads for every method. One caveat on the baselines: RECOMP-ext and CompAct were also trained on in-distribution data, so the training exposure isn't unique to AttnComp.

38
00:09:04,346 --> 00:09:05,461
[Hal Turing] Headline result?

39
00:09:05,461 --> 00:09:26,684
[Dr. Ada Shannon] The main metric is accuracy against reference answers, higher is better. Keep only the top five documents, a similar 18x squeeze, and you get about 40 answers right out of 100. AttnComp at 17x gets about 44. Reading all 100 documents gets about 42. So it's the only compressor that beats reading everything.

40
00:09:26,684 --> 00:09:27,891
[Hal Turing] Is that gap solid?

41
00:09:27,891 --> 00:09:56,127
[Dr. Ada Shannon] It's about two points, one run, no significance tests, so I'd call it parity or slightly better. The paper says every other method loses at least three points, but CompAct loses 2.9. Single-hop NQ is slightly worse than uncompressed, 53.0 against 53.9. The gains concentrate in multi-hop. And Provence compresses at sentence level, so its compression rates aren't the same granularity as AttnComp's.

42
00:09:56,127 --> 00:09:56,916
[Hal Turing] Latency?

43
00:09:56,916 --> 00:10:21,762
[Dr. Ada Shannon] About one second end to end instead of about two. Compression takes 0.91 seconds and generation 0.16, against 2.18 for reading everything. CompAct compresses about 80x but takes roughly 41 seconds, because it calls an LLM repeatedly. RECOMP is faster, but it's less accurate. Retrieval time is excluded, and it's one hardware setup with an 8B reader.

44
00:10:21,762 --> 00:10:23,248
[Hal Turing] Why 13 layers? Did they check?

45
00:10:23,248 --> 00:10:47,304
[Dr. Ada Shannon] They did. Middle layers 13 to 15 work even without fine-tuning. Layers 7, 23 and 31 don't, and the deep layers over-compress by more than 200x. Fine-tuning rescues every configuration to roughly 42 to 44 accuracy. For layer 13, fine-tuning mostly buys compression: the same 44.2 accuracy at 17x instead of 6.2x.

46
00:10:47,304 --> 00:10:49,068
[Hal Turing] Is it fussy about the threshold?

47
00:10:49,068 --> 00:11:09,502
[Dr. Ada Shannon] Stable for p of 0.9 and above, going by the figure, with no exact values. Its advantage also grows with the number of retrieved documents. And scoring at sentence level, with no sentence-level training, gives 43.1 at 21.1x. That's more compression for a bit less accuracy.

48
00:11:09,502 --> 00:11:11,499
[Hal Turing] And the trust signal I was teased with?

49
00:11:11,499 --> 00:11:22,366
[Dr. Ada Shannon] Confidence is one minus the instruction's score. Since attention sums to one, that's roughly the mass sitting on documents. High instruction share means the model found little worth attending to.

50
00:11:22,366 --> 00:11:32,025
[Hal Turing] The part I keep returning to is that the compressor trains on HotpotQA, and HotpotQA is one of the five test sets. How much of the headline gain is just fitting that dataset?

51
00:11:32,025 --> 00:11:53,945
[Dr. Ada Shannon] First a correction to how I framed the gain. Over reading everything, it's about 2.5 points on HotpotQA and 3.4 on 2Wiki. The multi-hop benefit is real with the Llama reader. Whether it transfers beyond HotpotQA-like data isn't shown. The paper has no leave-HotpotQA-out run and no mixed-dataset training, and it gives no reason.

52
00:11:53,945 --> 00:11:55,803
[Hal Turing] But the zero-shot sets do improve.

53
00:11:55,803 --> 00:12:27,057
[Dr. Ada Shannon] Some do. MuSiQue gains about four points before any tuning, so that one comes from the architecture. 2Wiki is different. The untrained version sits below the uncompressed baseline there, and fine-tuning adds about six points over it. 2Wiki is Wikipedia-sourced two-hop data, close in structure to HotpotQA. Then the Qwen reader: only HotpotQA improves in accuracy, and the four unseen sets are flat to slightly down. F1 does go up on all five there, and the paper doesn't explain why the two metrics diverge.

54
00:12:27,057 --> 00:12:28,589
[Hal Turing] So what experiment settles it?

55
00:12:28,589 --> 00:12:37,413
[Dr. Ada Shannon] Train on NQ or a mix and test on HotpotQA. Train on 2Wiki. Vary p per dataset. All cheap, given a four-hour training run.

56
00:12:37,413 --> 00:12:48,326
[Hal Turing] Now the labels. The teacher is the untrained attention itself, and then a filter keeps only examples where the reader answers correctly from the selected documents. That sounds circular.

57
00:12:48,326 --> 00:13:10,478
[Dr. Ada Shannon] It has a survivorship problem. The filter throws away exactly the cases where the base selection missed something that the full context would have recovered. So fine-tuning can't learn to find what the teacher systematically misses. And the paper reports no agreement between these labels and gold supporting facts. Its stated reason is that gold annotations are incomplete once you chunk Wikipedia.

58
00:13:10,478 --> 00:13:12,336
[Hal Turing] Is answer-correct even a strong check?

59
00:13:12,336 --> 00:13:34,487
[Dr. Ada Shannon] Weak. With no retrieval at all, HotpotQA accuracy is 23.6, so some right answers come from the model's memory. And the verifier is the same Llama family, which the authors admit. My inference, not a finding: negatives built from reader failures may teach the student that 'the reader failed' means 'no relevant documents'.

60
00:13:34,487 --> 00:13:37,413
[Hal Turing] Is there a fair case that it's more than replaying the teacher?

61
00:13:37,413 --> 00:13:58,775
[Dr. Ada Shannon] Yes, a real one. The student collapses an ensemble of shuffled, iterated passes into a single pass. It learns the abstain behavior the base never had. And fine-tuning rescues layers whose untrained attention collapsed, from about 26 accuracy back to 42 to 44. A replay wouldn't do that.

62
00:13:58,775 --> 00:14:02,073
[Hal Turing] Then the head choice. Sixteen heads from layer 14, averaged.

63
00:14:02,073 --> 00:14:29,286
[Dr. Ada Shannon] The text doesn't spell out how those 16 were picked. Wu and colleagues' retrieval heads are defined by copy-paste behavior during generation, which is a different thing. Since the Q and K projections get fine-tuned anyway, initialization probably matters less. But there's no ablation: all heads, random 16, top retrieval heads. The effect is unknown. My guess is that retrieval heads would help the untrained variant most, and that is a hypothesis.

64
00:14:29,286 --> 00:14:31,469
[Hal Turing] What about the ancestors? Izacard and Grave.

65
00:14:31,469 --> 00:14:47,816
[Dr. Ada Shannon] The AttnComp paper doesn't cite them. The differences are real: they attend from the answer decoder after reading, to train a retriever. AttnComp attends from the question before generation, to compress at inference. The shared risk is that the model itself supplies the supervision.

66
00:14:47,816 --> 00:14:51,438
[Hal Turing] And attention as explanation? Doesn't that undercut the whole idea?

67
00:14:51,438 --> 00:15:19,163
[Dr. Ada Shannon] Jain and Wallace showed attention weights can disagree with gradient-based importance, and very different attention patterns can give the same prediction. That fits what the authors found, since raw attention needed fine-tuning. AttnComp sidesteps it by treating attention as a trained scorer, validated end to end. It doesn't claim to explain what the reader used. But the story that the model leans on internal knowledge when the instruction score is high? Untested.

68
00:15:19,163 --> 00:15:20,974
[Hal Turing] Which brings us to the confidence number.

69
00:15:20,974 --> 00:15:52,367
[Dr. Ada Shannon] Instances below 0.1 confidence average F1 of 0.13. Above 0.9, 0.91. That looks strong until you see the Pearson correlation: 0.35 for AttnComp, 0.32 for the uncompressed baseline, on HotpotQA only, the training distribution. So it's a decent ranking signal, not a calibrated probability. There's no calibration error, no AUROC, no unseen sets. And it measures retrieval quality only, not reader reasoning.

70
00:15:52,367 --> 00:15:53,528
[Hal Turing] So the practical picture?

71
00:15:53,528 --> 00:16:23,714
[Dr. Ada Shannon] A small compressor: 13 layers of an 8B model, about half a percent trained, about four hours on four 24-gigabyte GPUs. Roughly a second of overhead, and one knob, p, stable at 0.9 or higher. A confidence gate that triggers re-retrieval or abstention is plausible, but the paper only proposes it. Untested: leave-one-dataset-out, head ablations, non-Wikipedia or long-form corpora, readers beyond one Qwen check, seed variance. The authors themselves flag 8B dense models only.

72
00:16:23,714 --> 00:16:54,272
[Hal Turing] Then here's where I land. Attention from middle layers, tuned lightly, works as a fast, adaptive document selector. On the Llama reader it beats reading all 100 documents at about 17x compression and half the latency, mostly on HotpotQA-style multi-hop questions. Whether that holds for other readers and for data unlike HotpotQA isn't shown yet. And the confidence score ranks answers but isn't a calibrated probability.

73
00:16:54,272 --> 00:16:56,269
[Dr. Ada Shannon] That's the claim the evidence supports.

74
00:16:56,269 --> 00:16:58,591
[Hal Turing] Thanks for listening, everyone. See you next time.