1
00:00:01,000 --> 00:00:07,408
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,408 --> 00:00:48,182
[Hal Turing] Today's paper is LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning, by Mengmeng Ji and colleagues at SambaNova Systems, posted to arXiv in 2026. It builds on two predecessors. One is Speculative Prefill, from Jingyu Liu, Beidi Chen and Ce Zhang, at ICML 2025. The other is AttnComp, from Lvzhou Luo, Yixuan Cao and Ping Luo, 2025. The question is this: can a small trained scorer pick the right 16,000 tokens out of roughly 115,000, so that other companies' models answer about as well as with the whole input?

3
00:00:48,182 --> 00:01:00,396
[Dr. Ada Shannon] And the claim is yes, for code bug-finding. The abstract says it matches or exceeds full context. I'd read "exceeds" carefully, and we'll get there. First the setup, because the cost story matters.

4
00:01:00,396 --> 00:01:03,275
[Hal Turing] Right. Why is a long prompt expensive in the first place?

5
00:01:03,275 --> 00:01:36,387
[Dr. Ada Shannon] Before a model emits its first output token, it has to process the entire prompt. That's prefill. Its cost grows at least linearly with length, and attention itself grows quadratically. At 100k tokens, prefill dominates time-to-first-token, the wait before anything appears. Compression drops most of the prompt before the big model sees it. One caveat: this paper never measures time-to-first-token. The dramatic latency figures come from the Speculative Prefill line of work.

6
00:01:36,387 --> 00:01:39,220
[Hal Turing] So what kind of compression is this? Summarizing?

7
00:01:39,220 --> 00:02:14,979
[Dr. Ada Shannon] No, and that distinction matters. Abstractive methods write a condensed version. Extractive methods keep a subset of the original tokens or chunks and drop the rest. LongAttnComp is extractive, so the target model reads verbatim text with gaps. For code that's important, since a paraphrased function isn't the function. The best-known trained extractive baseline is LLMLingua-2, by Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang and colleagues, 2024. It's task-agnostic and distilled from GPT-4 labels. This paper doesn't compare against it.

8
00:02:14,979 --> 00:02:18,740
[Hal Turing] And Speculative Prefill? The name sounds like speculative decoding.

9
00:02:18,740 --> 00:02:44,793
[Dr. Ada Shannon] The only similarity is that a cheap model helps an expensive one. In speculative decoding the small model guesses output tokens. Here a small draft model reads the whole prompt, and its attention weights score which input tokens matter. The top-scoring ones go to the big model. There's no training, and because the output is plain text, any target model can consume it. The authors' complaint is that on code reasoning it leaves a large gap.

10
00:02:44,793 --> 00:02:47,208
[Hal Turing] Scoring by attention. What is it attending from?

11
00:02:47,208 --> 00:03:22,270
[Dr. Ada Shannon] The query. That's query-aligned importance. Take a prompt that ends with a question and four answer options. You ask how strongly those final tokens attend to each earlier chunk. Chunks the question looks at score high. SnapKV, by Yuhong Li, Yingqi Huang, Bowen Yang and colleagues, 2024, does something similar inside the model. It scores KV entries using attention from the last window of tokens. Both rest on one assumption, that the query sits somewhere findable, usually the end. That's why this paper needs a query parser.

12
00:03:22,270 --> 00:03:25,753
[Hal Turing] Why does that count as compression at all? It sounds like retrieval.

13
00:03:25,753 --> 00:03:44,004
[Dr. Ada Shannon] That's exactly how the authors frame it. Their Figure 1 splits the job into finding the evidence and then reasoning over it. They treat compression as the retrieval half. It's a retrieve-then-read pipeline, except the retriever works on chunks inside one prompt instead of a corpus index. The danger is that the target never sees what the compressor discards.

14
00:03:44,004 --> 00:03:47,208
[Hal Turing] What's the baseline they compare against when the prompt is too long?

15
00:03:47,208 --> 00:04:07,224
[Dr. Ada Shannon] Middle truncation. If the input exceeds the model's window, you cut out the middle and keep the head and tail. Hold onto that. Models also use the middle of long inputs poorly, the lost-in-the-middle effect. So the baseline is cutting exactly the region models already handle badly. I'll flag why that matters for the headline claim later.

16
00:04:07,224 --> 00:04:12,054
[Hal Turing] Okay. So how do you train the scorer? Labels for 'this chunk matters' seem expensive.

17
00:04:12,054 --> 00:04:43,911
[Dr. Ada Shannon] They use needle-in-a-haystack style data. Plant a few relevant passages among many irrelevant ones, and the relevance labels come free from dataset annotations. They add multi-hop data later, where answering needs several linked passages. The support is From Artificial Needles to Real Haystacks, by Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, Dimitris Papailiopoulos and colleagues, 2024. They fine-tuned language models on synthetic retrieval data and improved real long-context retrieval.

18
00:04:43,911 --> 00:04:48,045
[Hal Turing] But that fine-tuned the generator. This one trains a separate scorer.

19
00:04:48,045 --> 00:05:24,639
[Dr. Ada Shannon] Right, so it's supporting evidence, not proof. And I'd be plain about the data here. Both training stages are synthetic, NIAH-style. The open question is whether that transfers to real inputs, and the rest of the paper is about how far it does. Now the method, which is small. Take the first 13 layers of Llama-3.1-8B-Instruct and freeze them. Add one trainable cross-attention layer on top, about half a percent of the parameters. The question tokens attend over the context tokens, the attention is averaged over heads, and that gives each chunk a score.

20
00:05:24,639 --> 00:05:29,562
[Hal Turing] So the scorer is the draft model's attention, but with one layer taught what relevant means.

21
00:05:29,562 --> 00:05:44,237
[Dr. Ada Shannon] Yes. The chunks are fixed-size token blocks, taken in score order until the cumulative score hits 0.95 or the kept tokens hit a 16k budget. The kept chunks go back into original order, and that prompt goes to the target model over an API.

22
00:05:44,237 --> 00:05:47,673
[Hal Turing] That's several changes from AttnComp. What did each one fix?

23
00:05:47,673 --> 00:06:12,519
[Dr. Ada Shannon] Four of them. Token chunks, because a real repository has no neat document boundaries. A token budget instead of AttnComp's minimum-score cutoff, because the cutoff fired first on long inputs and over-compressed. Restoring original order, so the code reads coherently. And a query parser that just takes the last 128 tokens, which costs about one point on Code-Debug compared with parsing the query exactly.

24
00:06:12,519 --> 00:06:16,652
[Hal Turing] Why keep the cumulative threshold at all, if the budget is the reliable part?

25
00:06:16,652 --> 00:06:35,832
[Dr. Ada Shannon] It adapts. On a RULER single-needle task, selection stops around 2k tokens against a 16k budget and accuracy is 100%. But when evidence is spread out, stopping early drops it. On LongBench v2, stopping at six to nine thousand tokens lost evidence, so they switched to budget-only there.

26
00:06:35,832 --> 00:06:37,039
[Hal Turing] And the training data?

27
00:06:37,039 --> 00:07:05,739
[Dr. Ada Shannon] Stage 1 is 32,000 synthetic samples from SQuAD and HotpotQA, 8k to 48k tokens each. Stage 2 continues from that checkpoint on MuSiQue and 2WikiMultiHopQA, which need two to four supporting documents, plus replay of the Stage 1 data. Labels come from the datasets' own annotations, so no LLM labeling. The subq variant appends MuSiQue's sub-questions to the training query. The nosubq variant doesn't.

28
00:07:05,739 --> 00:07:07,829
[Hal Turing] Okay, so what does the headline test look like?

29
00:07:07,829 --> 00:07:44,052
[Dr. Ada Shannon] InfiniteBench Code-Debug. The model reads a code repository averaging about 115k tokens, some over 200k, and picks which of four options identifies the bug. Chance is 25%, higher is better. R1's window here is 120k, so the full-context baseline gets middle-truncated. With R1 as the target, the training-free Speculative Prefill gets about 62 right out of every hundred. LongAttnComp gets about 75. That's roughly 13 more per hundred. Full context sits at 74.4, so the trained scorer is level with it or slightly ahead.

30
00:07:44,052 --> 00:07:45,724
[Hal Turing] How many questions is that, though?

31
00:07:45,724 --> 00:08:05,136
[Dr. Ada Shannon] The paper never says. Every score is a multiple of one over 394, so I infer about 394 items. That's my inference from the arithmetic, not a stated number. At that size, Stage 1 is about four questions ahead of full context, and the best variant, subq, about ten.

32
00:08:05,136 --> 00:08:06,947
[Hal Turing] Does it hold on other target models?

33
00:08:06,947 --> 00:08:29,145
[Dr. Ada Shannon] Not as a win. On the other three, full context comes out ahead by about two to three points: V3.1 by under two, MiniMax by two and a half, GPT-OSS by three. Those windows are 128k, 152k and 120k. So the R1 edge shows up only where truncation is heaviest, and it's one point.

34
00:08:29,145 --> 00:08:32,210
[Hal Turing] But GPT-OSS has R1's window, and it still loses.

35
00:08:32,210 --> 00:08:47,860
[Dr. Ada Shannon] Right, so window size can't be the whole story. What the table does show is that a scorer trained around Llama transfers to three unrelated families, and every compressed run is well above Speculative Prefill. That baseline used chunk size 128, though, which matters later.

36
00:08:47,860 --> 00:08:49,904
[Hal Turing] Does it transfer to harder, messier tasks?

37
00:08:49,904 --> 00:09:09,269
[Dr. Ada Shannon] LongBench v2 is multi-document reasoning, and that's the counterexample. The untruncated model, from the leaderboard, gets about 57. Capped at 100k it gets 51. Speculative Prefill gets 46. Stage 1 drops to 41.7, worse than the training-free method.

38
00:09:09,269 --> 00:09:10,616
[Hal Turing] So Stage 2 is the repair.

39
00:09:10,616 --> 00:09:28,960
[Dr. Ada Shannon] It recovers most of it, to about 49 or 50, level with the 100k cap and still about seven below untruncated. One caution: the nosubq row has an overall score above both its Easy and Hard scores, which a weighted average can't produce. That's probably a table typo, so I wouldn't lean on it.

40
00:09:28,960 --> 00:09:30,817
[Hal Turing] Does RULER say anything different?

41
00:09:30,817 --> 00:09:50,043
[Dr. Ada Shannon] It shows both sides. On one single-needle variant, the truncated baseline scores 57 and the compressor 99, which is lost-in-the-middle damage to the baseline. But on multi-value and multi-query retrieval, the compressor is worse than full context. When the answer is spread across many places, the scorer loses some of it.

42
00:09:50,043 --> 00:09:53,294
[Hal Turing] What about subq versus nosubq? Did the sub-questions help?

43
00:09:53,294 --> 00:10:04,811
[Dr. Ada Shannon] Mixed. Each wins on two of four Code-Debug targets, by under three points. Only V3.1 on LongBench v2 clearly favors subq, and the authors call it exploratory.

44
00:10:04,811 --> 00:10:06,158
[Hal Turing] And the efficiency story?

45
00:10:06,158 --> 00:10:21,065
[Dr. Ada Shannon] It's an estimate. Thirteen of 32 layers means about a third of Speculative Prefill's compressor overhead. The 46 seconds down to 2.5 comes from earlier work. This paper measures no latency, throughput or memory at all.

46
00:10:21,065 --> 00:10:29,471
[Hal Turing] Let me start with the sentence in the abstract: matches or exceeds full context. Is that a statement about compression quality, or about a handicapped baseline?

47
00:10:29,471 --> 00:10:47,861
[Dr. Ada Shannon] I'd say mostly the baseline. The advantage shows up only on the R1 target, which has the tightest window, and it reverses on the longer-window models. Middle truncation throws away the center of the longest repositories. A 16k prompt that kept the buggy function can beat a 120k prompt that lost it.

48
00:10:47,861 --> 00:10:51,344
[Hal Turing] So the compressor could be winning by not being truncated blindly.

49
00:10:51,344 --> 00:11:10,942
[Dr. Ada Shannon] Right. And the paper can't rule that out. It doesn't report what fraction of samples were truncated. It also doesn't report full-context accuracy on only the samples that fit. The experiment is simple: bin by token length, then report full-context and compressed accuracy per bin. If the gap lives in the longest bin, that's your answer.

50
00:11:10,942 --> 00:11:17,165
[Hal Turing] Then the statistics. I was going to say the test set is around 200 questions, so a couple of points is five questions.

51
00:11:17,165 --> 00:11:34,208
[Dr. Ada Shannon] The percentages imply about 394 items, so the best variant's edge is about ten questions, not five. A confidence interval is the range of true accuracies compatible with the score you measured. At roughly 75 percent on 394 questions, that range is about four points either side.

52
00:11:34,208 --> 00:11:36,948
[Hal Turing] So a gap of one to two and a half points sits inside it.

53
00:11:36,948 --> 00:11:52,273
[Dr. Ada Shannon] Yes, and so does the ten-question edge. A paired test on per-question outcomes would be tighter, but those aren't reported. Neither is run-to-run variance, and R1 samples stochastically. The honest claim is comparable accuracy at about one-sixth the prompt. Not better.

54
00:11:52,273 --> 00:11:54,409
[Hal Turing] Doesn't tuning make it even more optimistic?

55
00:11:54,409 --> 00:12:21,066
[Dr. Ada Shannon] Yes. Chunk size alone moves Code-Debug accuracy by about twenty points. The final settings were picked from sweeps on Code-Debug. The appendix mentions a held-out subset for tuning, but the tables report the full set, and the subset's size and overlap aren't stated. The top-p ablation uses ten samples, where one question is ten points. Tune-on-test means selecting settings on the same data you report, so the winner is partly selected for noise.

56
00:12:21,066 --> 00:12:25,756
[Hal Turing] But chunk size 1024 isn't arbitrary, is it? A function spans many tokens.

57
00:12:25,756 --> 00:12:52,320
[Dr. Ada Shannon] Agreed, that effect is plausibly real. I'd also credit that the LongBench v2 settings were tuned on V3.1 and reported on R1. But that's still the same benchmark, and a related model. And the baseline didn't get the same sweep. Speculative Prefill ran at chunk 128 and was never rerun at 1024. The paper's own scorer falls to about 57 at chunk 128, below Speculative Prefill, so part of that 13-point gap may be chunk size rather than training.

58
00:12:52,320 --> 00:12:54,224
[Hal Turing] What else is missing from the comparison?

59
00:12:54,224 --> 00:13:14,472
[Dr. Ada Shannon] A plain retrieval baseline, BM25 or embedding top-k filled to 16k. Simple head-and-tail truncation to 16k. And trained compressors like LLMLingua-2. Without them, we can't tell how much the learned cross-attention layer adds beyond any query-aware selection. Code-Debug may just need function-level retrieval.

60
00:13:14,472 --> 00:13:16,190
[Hal Turing] And the query-at-the-end assumption?

61
00:13:16,190 --> 00:13:37,785
[Dr. Ada Shannon] It's the same bet SnapKV makes, scoring from the last tokens. Here the last 128 tokens hold the question and four code-flavored options, which may help the scorer. Whether it works when the instruction comes first, or is interleaved, is untested. And the best query length varied by task, so it isn't format-agnostic in practice.

62
00:13:37,785 --> 00:13:39,642
[Hal Turing] That sounds like a lot of knobs.

63
00:13:39,642 --> 00:13:55,200
[Dr. Ada Shannon] Chunk size, query length and selection mode all differ by task. A deployer facing an unknown workload has no single recipe, and a fixed setting would leave accuracy on the table. The authors propose an adaptive selector, which seems like the right fix.

64
00:13:55,200 --> 00:13:59,333
[Hal Turing] What does the Stage 2 result tell us, then? Training data, not architecture?

65
00:13:59,333 --> 00:14:13,172
[Dr. Ada Shannon] It's informative, not proof. Adding multi-hop data helped LongBench v2, which fits that hypothesis. But it was tested on one gap, and the data is still synthetic NIAH-style, as the paper's limitations section says.

66
00:14:13,172 --> 00:14:14,983
[Hal Turing] What about things the paper never says?

67
00:14:14,983 --> 00:14:31,980
[Dr. Ada Shannon] How inputs above the backbone's own 128k window get handled, when some samples exceed 200k. Real latency and memory, since the efficiency story is a layer-count estimate. Other backbones. And whether results hold outside the authors' own API setup.

68
00:14:31,980 --> 00:14:33,327
[Hal Turing] So who should care today?

69
00:14:33,327 --> 00:14:51,810
[Dr. Ada Shannon] Anyone who must cut a long prompt anyway. A query-aware 16k selection is a credible alternative to blind middle truncation. What it doesn't establish is that it beats an untruncated pass, or that it saves a particular amount of time or money. And I'd be wary on tasks where the evidence is spread out, where it looked weakest.

70
00:14:51,810 --> 00:15:18,699
[Hal Turing] So the work shows that a 13-layer frozen backbone plus one trained cross-attention layer can keep about 16k tokens and get close to full-context accuracy on code bug-finding, across four target models. The boundary is that the full-context comparison is confounded by truncation, the settings were tuned on the test benchmark, the test set is small, and LongBench v2 still shows a residual gap.

71
00:15:18,699 --> 00:15:22,182
[Dr. Ada Shannon] That's fair. A promising, narrowly validated result.

72
00:15:22,182 --> 00:15:24,318
[Hal Turing] Thanks for listening, everyone. Until next time.