1
00:00:01,000 --> 00:00:08,151
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:08,151 --> 00:00:35,411
[Hal Turing] Today's paper is "You don't need long context," by Peter Belcak and colleagues at NVIDIA, posted to arXiv on November 3rd, 2026. Its claim: later tokens cost more to generate than early ones, up to seven times more on some open-weight models. Frequent compaction can keep the live context short, and the authors say frontier models lose little accuracy when it does.

3
00:00:35,411 --> 00:00:47,672
[Dr. Ada Shannon] The stake is that serving stacks are built around ever-larger windows, and this paper says much of that depth is unnecessary. I'd treat the accuracy results and the cost results separately, because they rest on very different evidence.

4
00:00:47,672 --> 00:00:51,201
[Hal Turing] Start with the premise. Why is a later token more expensive?

5
00:00:51,201 --> 00:01:21,944
[Dr. Ada Shannon] Each generated token attends over the stored keys and values of everything before it. That's the KV cache. A deeper context means more memory traffic per token. So you get higher latency, a bigger memory footprint, and fewer concurrent requests per GPU. The paper's Figure 1 converts measured peak decode throughput on H100s into dollars per token, and finds serving a 256K window costs between 3.6 and 7 times more than serving 16K, depending on the model.

6
00:01:21,944 --> 00:01:24,963
[Hal Turing] But I pay a flat price per million tokens through an API.

7
00:01:24,963 --> 00:01:36,898
[Dr. Ada Shannon] Right, and the authors point that out: retail pricing hides the depth effect. So that figure is a measured serving number on open-weight models, on rented GPUs. It isn't anyone's bill.

8
00:01:36,898 --> 00:01:38,895
[Hal Turing] What's the gap they're pointing at?

9
00:01:38,895 --> 00:01:57,099
[Dr. Ada Shannon] Table 1. The Great Gatsby is about 64K tokens. Great Expectations is around 245K. Context windows run from 200K for Claude Opus 4.5 up to around a million for the largest models. Models have been built to accept far more than most tasks plausibly need.

10
00:01:57,099 --> 00:02:01,000
[Hal Turing] So the root question is whether long contexts are necessary at all.

11
00:02:01,000 --> 00:02:19,901
[Dr. Ada Shannon] Yes, that's RQ0. The paper splits it by where long contexts come from. The first source is long-horizon tasks: many sequential actions, with history piling up as the agent works. The second is long-input tasks: a huge document set sits behind a short question.

12
00:02:19,901 --> 00:02:21,759
[Hal Turing] Those sound like different problems.

13
00:02:21,759 --> 00:02:52,223
[Dr. Ada Shannon] The paper collapses one into the other. Store the long input as files, give the agent tools like sed and head, and let it read selectively. Now a long-input task is a long-horizon task with a short context. That gives three questions. RQ1: can long-horizon tasks finish without long contexts? RQ2: can long-input tasks? RQ3: can a long-horizon task finish when the agent takes only one action per context?

14
00:02:52,223 --> 00:02:54,871
[Hal Turing] Walk me through the regimes with a coding agent.

15
00:02:54,871 --> 00:03:07,549
[Dr. Ada Shannon] Take an agent fixing a bug in a repository. The default loop is Full-History Accumulation, FHA. Every command and every output gets appended to one ever-growing conversation. That's the baseline.

16
00:03:07,549 --> 00:03:33,276
[Dr. Ada Shannon] The second regime is End-of-Conversation Compaction, EoCC. The agent runs normally until a token threshold. Then the model writes a brief of the work so far, and a fresh conversation starts with the original task plus that brief. Settings are written as total limit over threshold, so 24K slash 18K means a 24K window, compacting at 18K. The brief is lossy and model-written.

17
00:03:33,276 --> 00:03:44,933
[Dr. Ada Shannon] The extreme case is Single-Action Compaction, SAC. Each fresh conversation gets the task and the current state, takes exactly one action, updates the state, and is discarded.

18
00:03:44,933 --> 00:03:47,394
[Hal Turing] That last one reminds me of a state space model.

19
00:03:47,394 --> 00:04:18,880
[Dr. Ada Shannon] That's the paper's proposal, the Semantic State Space Model, or SSSM. Compare Mamba, by Albert Gu and Tri Dao, 2023, out of Carnegie Mellon and Princeton. Mamba keeps a fixed-size hidden vector inside the network, trained end to end. History survives only through that state, and the cost per token stays constant. An SSSM keeps the idea but moves the recurrence outside the model. The state is readable natural-language text, and a short-context LLM call is the transition function.

20
00:04:18,880 --> 00:04:20,552
[Hal Turing] Is there an SSSM they trained?

21
00:04:20,552 --> 00:04:54,128
[Dr. Ada Shannon] No. The paper only outlines it. SAC is prompt-only on off-the-shelf models, and the SSSM is a proposal for post-training models to do this on purpose. Keep that in mind, along with a second term: the "estimated optimized retail cost." It's a synthetic price. The authors fit a cost-versus-window curve from the open-weight measurements and apply it to Opus 4.5 and GPT 5.6 Sol retail prices. It's not a measured bill, and it's not what those providers charge. When savings numbers come up later, that's the term to watch for.

22
00:04:54,128 --> 00:04:58,029
[Hal Turing] The idea of a small live context plus external memory isn't new, though.

23
00:04:58,029 --> 00:05:24,082
[Dr. Ada Shannon] No. MemGPT, by Charles Packer and colleagues at UC Berkeley, 2023, treats the context window like main memory in an operating system. The model pages information in and out of archival storage with function calls. Then there's the Recurrent Memory Transformer, by Aydar Bulatov, Yuri Kuratov and Mikhail Burtsev, 2022. It passes learned memory tokens between segments. So the lineage includes both text memory and learned memory.

24
00:05:24,082 --> 00:05:25,614
[Hal Turing] Does this paper build on those?

25
00:05:25,614 --> 00:05:46,002
[Dr. Ada Shannon] It cites Mamba and BABILong, a long-context benchmark. I don't see MemGPT or the agent-memory line cited, and MemGPT is the closest conceptual ancestor. The difference is that MemGPT is about extending effective memory. This paper is about cost, and about how little context an agent can get away with.

26
00:05:46,002 --> 00:05:50,599
[Hal Turing] Then the question for the rest of the episode is which parts are measured and which are modeled.

27
00:05:50,599 --> 00:06:32,349
[Dr. Ada Shannon] Exactly. The throughput gap is measured on H100s. The accuracy numbers are benchmark results. The dollar savings come from a fitted curve. We'll take them in that order. Start with how they measured throughput, because everything downstream leans on it. For each open model they used the smallest H100 node that could hold the weights plus a KV cache at the largest window tested. That's eight GPUs for Qwen 3 Coder Next and sixteen for GLM and Kimi. Then they sent requests of 1,024 generated tokens each, with prefix caching on, and raised concurrency until the server started evicting KV cache. They report the highest throughput before that point. Repeat runs varied by at most 2%.

28
00:06:32,349 --> 00:06:38,154
[Hal Turing] So the server is configured for 256K or 16K, and the hardware stays the same.

29
00:06:38,154 --> 00:06:50,042
[Dr. Ada Shannon] Yes. At 16K the same node emits 3.6 times as many tokens per second for GLM, 5.6 for Kimi, and about 7 for Qwen. Those gaps are measured. The next step is where it becomes modeled.

30
00:06:50,042 --> 00:06:54,361
[Hal Turing] Walk me through it. How do you get from three open models to a price for Opus?

31
00:06:54,361 --> 00:07:39,269
[Dr. Ada Shannon] Appendix B does it in a few steps. Invert each throughput gap into a cost ratio: 0.28, 0.18, 0.14. Average them and round to 0.20. So a token served at 16K costs one fifth of a token served at the full window. Then they take the retail output price for Opus 4.5, $25 per million, or GPT 5.6 Sol, $20. That price is pinned to the full window, 200K for Opus and 272K for GPT. They draw a straight line from there down to 0.20 times the price at 16K. Anything below 16K gets clamped at that floor. Input and cached input are derived from output with fixed ratios, 5 to 1 and 10 to 1.

32
00:07:39,269 --> 00:07:42,473
[Hal Turing] Did anyone measure Opus or GPT at different window sizes?

33
00:07:42,473 --> 00:07:56,683
[Dr. Ada Shannon] No. Neither closed model's curve was measured. The shape is borrowed from three open models and assumed linear. So "estimated optimized retail cost" is a what-if price, and it isn't what Anthropic or OpenAI charge.

34
00:07:56,683 --> 00:07:59,516
[Hal Turing] Then I want the results with both prices next to each other.

35
00:07:59,516 --> 00:08:37,969
[Dr. Ada Shannon] The accuracy results first, since those are real measurements. Across six long-horizon benchmarks, compaction at tighter and tighter limits mostly holds accuracy. Opus ranges from 3.7 points worse to 5.6 better. GPT ranges from 3.3 worse to 4.4 better. At most 5 of 22 configurations lose more than 2 points. FRAMES is the exception. It's multi-hop Wikipedia question answering, and it loses a little at every setting, between 1.3 and 3.7 points for Opus. And these are single runs with no confidence intervals, so a one or two point wobble is not established.

36
00:08:37,969 --> 00:08:38,804
[Hal Turing] Now the money.

37
00:08:38,804 --> 00:09:11,638
[Dr. Ada Shannon] SWE-bench Verified is a set of 500 real GitHub issues, scored by whether the patch passes hidden tests. Opus with full history solves 70 out of 100. At a 24K limit it solves about 75. The bill at actual flat retail prices goes from $420 to about $296, roughly 1.4 times cheaper. The paper's depth-priced model says $70, which is 6 times cheaper. The distance between $296 and $70 is the entire cost debate.

38
00:09:11,638 --> 00:09:16,792
[Hal Turing] Why is the real saving so small? Fewer tokens in context should mean fewer tokens billed.

39
00:09:16,792 --> 00:09:40,802
[Dr. Ada Shannon] Not quite. Every compaction forces new input that can't come from cache, and the model writes the briefs, so output tokens go up. On Terminal-Bench at 48K, Opus's nominal cost is $114.75 against $87.80 for the baseline. It rises. The depth-priced estimate for that same run is $39.33. Same tokens, two price lists.

40
00:09:40,802 --> 00:09:44,981
[Hal Turing] So the depth-priced number matters only if the party serving the model pays by depth.

41
00:09:44,981 --> 00:09:53,387
[Dr. Ada Shannon] Right. That holds for a self-hoster or a provider with an H100 cost curve like Figure 4's. It doesn't show up on a retail invoice.

42
00:09:53,387 --> 00:09:59,517
[Hal Turing] Long inputs. How does an agent answer a question about a 500K-token document without reading it?

43
00:09:59,517 --> 00:10:44,192
[Dr. Ada Shannon] The document sits in the sandbox as a file, say input01.txt. The agent gets the question and uses sed, head, awk, and grep to pull out just the lines it needs. Reading becomes an ordinary action. Across seven benchmarks, Opus changes by between minus 0.4 and plus 40 points, and GPT by plus 1 to plus 31. The biggest jumps are on needle-style tests, where a fact is buried in filler and you must find it, or find a handful scattered through the text. A text-search tool solves that nearly for free, so I'd read those gains as the full-context baselines struggling at length, as much as the method being strong. The comparison isn't same-harness either: baselines get everything in context, while the treatment gets tools.

44
00:10:44,192 --> 00:10:46,468
[Hal Turing] What about the tests that need real synthesis?

45
00:10:46,468 --> 00:11:10,245
[Dr. Ada Shannon] LongBench v2 is the realistic one. It's multiple-choice questions over messy natural documents. There the change is minus 0.4 for Opus and plus 1.8 for GPT, which is basically flat. No loss, no miracle. Also, conversation length stays within a fairly narrow band as documents grow, which supports the claim that these tasks need little information live at once.

46
00:11:10,245 --> 00:11:13,031
[Hal Turing] The compaction itself is lossy, though. What makes one good?

47
00:11:13,031 --> 00:11:40,292
[Dr. Ada Shannon] The authors found a plain summary is the wrong target. Summaries weight everything evenly and over-emphasize repeated material. The compactions that preserved accuracy are briefs for future work. They contain a near-verbatim copy of the task, what the agent is doing now and its next steps, a decomposed plan, the criteria for being done, and pointers to artifacts like files. It's a handoff note, not a diary.

48
00:11:40,292 --> 00:11:43,032
[Hal Turing] And if the lost detail is exactly what you needed later?

49
00:11:43,032 --> 00:12:11,964
[Dr. Ada Shannon] They concede it. Their example is a tool that emits coin tosses. With a fixed-size brief, you eventually forget some of them, and if you're asked for toss number 400, you fail. The escape hatch is to keep the full transcript on disk and give the agent a selective-read tool. Then looking up history becomes a long-input task, which we just saw it can do. The claim narrows to this: you don't need all history in context, though you may need it somewhere you can reach.

50
00:12:11,964 --> 00:12:20,044
[Hal Turing] Let me put the money question directly. If I'm a customer paying Anthropic or OpenAI, how much of the headline savings shows up on my bill?

51
00:12:20,044 --> 00:12:52,970
[Dr. Ada Shannon] Much less than the headline. On SWE-bench Verified with Opus, the best case is about 1.4 times cheaper at flat prices, and on Terminal-Bench the nominal cost goes the wrong way. Compaction adds output tokens and new input tokens. Nobody measured Opus or GPT cost by depth, so we don't know what their true curves look like. The depth-priced savings are real only for someone whose cost scales with depth, like a self-hoster or a provider that prices that way. The H100 measurement supports that case.

52
00:12:52,970 --> 00:13:04,301
[Hal Turing] Then the throughput number carries a lot. I worry about how it was taken. The node is sized for the full window, and the number comes at the highest concurrency without KV preemption.

53
00:13:04,301 --> 00:13:44,704
[Dr. Ada Shannon] Right. A server configured for 16K can batch many more concurrent requests in the same memory. So part of the 7x for Qwen is batch headroom from the configuration, not purely attention cost per token. Qwen is also the most favorable of the three. GLM's 3.6x is the more conservative anchor. And the mean of 0.20 mixes INT4 experts, an FP8 KV cache, and BF16. Also, the paper's two descriptions of how request lengths were drawn differ slightly, so I'd want to see the setup spelled out. And only open-weight models were measured, with no uncertainty beyond that 2% repeat variance.

54
00:13:44,704 --> 00:13:46,376
[Hal Turing] What about the 233x?

55
00:13:46,376 --> 00:14:10,432
[Dr. Ada Shannon] That's the extreme of the synthetic estimate. It comes from a needle-in-a-haystack-style test with GPT 5.6 Sol at a 48K window, priced by the fitted curve. It's not a billing outcome. The big accuracy gains also sit on tests where grep solves the task, so they partly show the full-input baseline failing at length. The one realistic benchmark, LongBench v2, is flat.

56
00:14:10,432 --> 00:14:15,447
[Hal Turing] And single-action compaction, the SAC regime? That motivates the whole text-state proposal.

57
00:14:15,447 --> 00:15:05,835
[Dr. Ada Shannon] The evidence is thinner than the abstract's tone. It covers two benchmarks and two closed models. Opus holds or gains: on SWE-bench Verified the largest single prompt drops from about 200K tokens to about 16K with the score unchanged. GPT 5.6 Sol loses 2.0 and 3.3 points, and a tuned EoCC beats SAC for it. Terminal-Bench has only 89 tasks, so one task is about a point, and I see no repeat runs or confidence intervals. So I can't call the +4.6 for Opus real. The open-weight models are uneven too. Qwen 3 Coder Next falls from 53.0% to 3.6% at the tightest setting. And I see no cost or latency figure for SAC, where every step starts a fresh conversation.

58
00:15:05,835 --> 00:15:11,872
[Hal Turing] Also, no semantic state space model was trained. The data-efficiency claim has nothing behind it.

59
00:15:11,872 --> 00:15:24,132
[Dr. Ada Shannon] Correct. SAC shows existing models can run on a text state alone, which is an argument for training for it on purpose. But the claim that a trained SSSM would be more data-efficient is a conjecture.

60
00:15:24,132 --> 00:15:26,082
[Hal Turing] Where does this sit against earlier work?

61
00:15:26,082 --> 00:15:56,129
[Dr. Ada Shannon] MemGPT already managed a bounded main context with archival memory and explicit retrieval. That's what the coin-toss escape hatch needs, and this paper's retrieval is just Bash on files. The Recurrent Memory Transformer carries learned memory tokens between segments. Mamba carries a learned hidden state. The text state is readable, but it's lossy and bounded by the brief. The RMT group built BABILong, which this paper uses, yet learned state versus text state is never compared.

62
00:15:56,129 --> 00:15:57,476
[Hal Turing] What else is left open?

63
00:15:57,476 --> 00:16:27,662
[Dr. Ada Shannon] Single runs, one compaction prompt tuned on SWE work, and a coding-heavy harness. Software engineering is close to a best case, since the file system holds state for free. Compaction also forces a restated task and plan, so some gain may be prompting, not context length. The paper's own appendix concedes that tasks with a human steering mid-run need more preserved history. And real APIs already cut cost with prefix caching, which the flat-price comparison has to contend with.

64
00:16:27,662 --> 00:16:28,823
[Hal Turing] So who should care?

65
00:16:28,823 --> 00:16:39,365
[Dr. Ada Shannon] Self-hosters, providers who price by depth, and on-device inference, where the paper's DGX Spark measurements apply. If you consume a closed API, check your own bill first.

66
00:16:39,365 --> 00:16:40,526
[Hal Turing] What would settle it?

67
00:16:40,526 --> 00:16:49,814
[Dr. Ada Shannon] Measure real closed-model latency and cost by depth. Compare against baselines with the same batching. Train an actual semantic state space model.

68
00:16:49,814 --> 00:17:11,873
[Hal Turing] So what's established? Prompted compaction mostly keeps accuracy within a few points for two frontier models in a bash-agent harness, with plausible serving gains on open-weight models. The boundary is that the dollar savings are modeled, SAC is mixed, and the case against long-context models goes past the evidence. Thanks for listening, everyone.