1
00:00:01,000 --> 00:00:06,154
[Hal Turing] Hello AI world! I'm your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:06,154 --> 00:00:29,839
[Hal Turing] Today it's "Context Language Models," from Rulin Shao at the University of Washington and Meta Superintelligence Labs, with co-authors at MIT and Trillium Labs. It's from September 2026, arXiv 2609.37725. The question: what happens if the model can rewrite its own context, any way it likes?

3
00:00:29,839 --> 00:00:45,489
[Dr. Ada Shannon] Today's agents mostly can't. Some outside program decides what the model sees. This paper hands the model the editor. The authors claim that beats human-designed context management and costs less. Whether the evidence supports that is what we'll sort out.

4
00:00:45,489 --> 00:00:49,622
[Hal Turing] Start with the problem. Why does context management exist at all?

5
00:00:49,622 --> 00:01:17,532
[Dr. Ada Shannon] The window is the model's only working memory, and it has a hard limit. A long deep-research run or a multi-hour coding task piles up tool calls and observations until something has to go. The harness, meaning the code around the model, truncates old output, compacts history into a summary, offloads to files, or retrieves later. The paper mostly uses a 32K-token limit, though one benchmark we'll get to actually runs with a smaller budget.

6
00:01:17,532 --> 00:01:20,133
[Hal Turing] So how does the paper sort the existing approaches?

7
00:01:20,133 --> 00:02:00,814
[Dr. Ada Shannon] Three tiers of control. First, harness-defined. A Codex-style baseline compacts at 75 percent of the budget and replaces history with one summary. The model doesn't pick the timing. MEM1 rewrites its context every turn. Second, action-based. The model decides when, but only through a menu humans defined: Self-Compact, ACM, Context-as-a-Tool, Sculptor. Third, unrestricted, which is the CLM. There's also Recursive Language Models, which treat the huge input as a REPL variable the model inspects with code. That keeps the input outside the window, though, rather than editing the live context.

8
00:02:00,814 --> 00:02:02,115
[Hal Turing] What's the formal difference?

9
00:02:02,115 --> 00:02:36,805
[Dr. Ada Shannon] A normal LM appends: the next context is the old context plus the output. A CLM says the next context is any function of the current one, which the model controls. In practice, the live context is mirrored to a file. The model edits it with ordinary Bash or Python, and each edit syncs back into its live context. If it doesn't edit, new tokens just append. A subagent is another context file, and creating or deleting files creates or destroys agents.

10
00:02:36,805 --> 00:02:39,917
[Hal Turing] Why isn't everyone doing this? Editing a file sounds cheap.

11
00:02:39,917 --> 00:03:13,957
[Dr. Ada Shannon] The serving cost is the catch. Prefill is the model reading the prompt, and decode is generating tokens. Servers keep the key-value states of earlier tokens, so a new request sharing an exact prefix only prefills what comes after. The first mismatching token invalidates everything after it, including unchanged text. The paper's illustration: an append-only turn costs 1.41e14 FLOPs, an edit in the middle 5.74e14, and an edit at the start 10.81e14.

12
00:03:13,957 --> 00:03:17,765
[Hal Turing] So an edit can cost several times an append. Is that the cost metric they use?

13
00:03:17,765 --> 00:03:35,180
[Dr. Ada Shannon] Yes, prefix-reuse FLOPs. It's the theoretical prefill FLOPs from the first prefix mismatch onward, plus decode FLOPs, summed over a whole trajectory. It's analytic, not measured latency, throughput or energy. It does charge the CLM for the re-prefill its edits cause.

14
00:03:35,180 --> 00:03:39,360
[Hal Turing] And how do you test context management without reasoning ability muddying it?

15
00:03:39,360 --> 00:04:01,326
[Dr. Ada Shannon] The authors built ContextBench, a synthetic diagnostic with four tasks: Needle Retention, Sudoku Sketchpad, KV Store, and Log Triage. Grading looks at what's in the agent's context, and an answer held only in a file gets no credit. Their pilot finding is qualitative: no existing strategy is perfect, even on tasks this simple.

16
00:04:01,326 --> 00:04:02,905
[Hal Turing] What's the motivating argument?

17
00:04:02,905 --> 00:04:22,503
[Dr. Ada Shannon] The Bitter Lesson: let the model search for and learn the strategy instead of hard-coding human priors. That's motivation, not evidence. The paper then takes four directions: zero-shot evaluation of existing models, steering and skill evolution, reinforcement learning, and a serving change called Suffix Cache Reuse.

18
00:04:22,503 --> 00:04:25,846
[Hal Turing] Let's start with the zero-shot setup. How is it arranged?

19
00:04:25,846 --> 00:04:58,401
[Dr. Ada Shannon] Same Mini-SWE-Agent backbone, same Qwen3.6-27B, nothing trained. Baselines are the Codex-style summary, MEM1, Self-Compact, ACM, RLM, and the plain base harness. For CLM on BrowseComp-Plus, a few facts about its setup: it gets a reminder 2,048 tokens before the budget, editing turns don't count toward the 100-turn cap, and if a request would overflow, the harness rolls back and lets it retry, up to six times. Fifty on EdgeBench. I'll just state those for now.

20
00:04:58,401 --> 00:05:00,676
[Hal Turing] And what does BrowseComp-Plus actually test?

21
00:05:00,676 --> 00:05:27,797
[Dr. Ada Shannon] Deep research over a fixed document corpus. 830 hard questions, an agent searches and reads, higher accuracy is better. CLM scores 59.4 percent. The Codex-style summary lands around 53, so about six more correct answers per hundred questions, with roughly a fifth less compute. That baseline figure is my derivation from the paper's 11.4 percent relative gap, not a printed number.

22
00:05:27,797 --> 00:05:31,373
[Hal Turing] What does the model actually do to its file? Is it just summarizing?

23
00:05:31,373 --> 00:06:05,460
[Dr. Ada Shannon] Sometimes, yes. One example reproduces a summary-style rewrite. But the others are stranger. An orchestrator keeps a scoreboard of its subagents and updates it in place 163 times, holding its context at six to eight thousand tokens. Another invents a new "notes" role in the transcript for its own scratch state. Others write loops that strip old search results down to "searched this, nothing relevant." One defines a compact_turns function and calls it 37 times.

24
00:06:05,460 --> 00:06:09,268
[Hal Turing] That last one is the model writing its own context-management tool.

25
00:06:09,268 --> 00:06:11,451
[Dr. Ada Shannon] Right. Nobody gave it that action.

26
00:06:11,451 --> 00:06:13,448
[Hal Turing] And subagents are just more files?

27
00:06:13,448 --> 00:06:36,157
[Dr. Ada Shannon] Create a context file, you've spawned an agent. Delete it, it's gone. The swarm test, Software World, has six agents optimizing six interdependent Python repositories for 24 hours. They're scored on held-out downstream packages by geometric-mean speedup over 17 benchmarks. The paper claims a larger speedup than a summary swarm at the same spend.

28
00:06:36,157 --> 00:06:38,711
[Hal Turing] Can you steer any of this without touching the harness?

29
00:06:38,711 --> 00:07:04,299
[Dr. Ada Shannon] One sentence in the prompt. Tell it to compact at 16K, 24K or 32K tokens, and the median first compaction lands at 16.0K, 23.6K and 30.9K. Ask it to compact at sub-question boundaries, and compaction within two turns of a boundary rises from 0.40 to 0.88. Ask it to back up to disk first, and the full-backup rate goes from zero to 0.68.

30
00:07:04,299 --> 00:07:05,692
[Hal Turing] And the evolved skills?

31
00:07:05,692 --> 00:07:41,591
[Dr. Ada Shannon] A skill is a natural-language document in context. The loop is GEPA-style: roll out, a proposer reads the traces and writes candidates, candidates are scored on a dev split, and the winner is tested once on held-out data. In assisted evolution, a stronger model, Claude Fable 5.1, proposes skills for the Qwen agent. In self-evolution, Opus 5 plays both roles. On held-out KV Store, accuracy goes from about 38 out of 100 to about 74. That's the authors' own synthetic task, drawn from the same generator as the training instances.

32
00:07:41,591 --> 00:07:46,188
[Hal Turing] Then the weights route. How does RL cope when edits change the input every turn?

33
00:07:46,188 --> 00:08:12,427
[Dr. Ada Shannon] Stepwise GRPO. Sample a group of full trajectories per prompt, compute the usual group-relative advantage from outcome reward, then cut each trajectory into segments. Each segment is trained under the context it was actually generated in, and every segment inherits the trajectory's advantage. It has to be on-policy, because the context the model sees depends on its own earlier edits, so stale rollouts would show it inputs it wouldn't now produce.

34
00:08:12,427 --> 00:08:14,841
[Hal Turing] Rewarding fewer tokens would be gameable, though.

35
00:08:14,841 --> 00:08:42,241
[Dr. Ada Shannon] Which is why the efficiency term is success-gated. Among successful trajectories in a group, you reward lower prefix-reuse FLOPs relative to the group mean, clipped to plus or minus one. Failures get zero, so no speed bonus for failing. With fewer than two successes, the term is zero for the whole group. Weight 0.25. They train Qwen3.5-9B on OpenResearcher prompts, 70 steps, eight prompts times 32 rollouts per step.

36
00:08:42,241 --> 00:08:47,164
[Hal Turing] Now the serving side. What does Suffix Cache Reuse reuse that prefix caching throws away?

37
00:08:47,164 --> 00:09:25,105
[Dr. Ada Shannon] Take context A, B, C, and an edit turns B into B-prime. Standard serving matches A, then re-prefills B-prime and C. SCR keeps C's cached keys and values. But those keys carry rotary position encodings, and C has moved, so it re-rotates them to the new positions, splices them after B-prime, and prefills only B-prime and new tokens. Those cached states were computed under the old context, so it's an approximation. It relocates at most six surviving spans per edit, longest first, in session-private slots, with fallback to normal re-prefill.

38
00:09:25,105 --> 00:09:29,285
[Hal Turing] Qwen3.6 is a hybrid, though. What happens in the linear-attention layers?

39
00:09:29,285 --> 00:09:45,492
[Dr. Ada Shannon] 48 of its 64 layers are linear attention with a fixed-size recurrent state, so there are no per-token entries to move. SCR snapshots the state before the edit and continues from it. B-prime is recomputed only in the 16 full-attention layers.

40
00:09:45,492 --> 00:09:46,514
[Hal Turing] Does it pay off?

41
00:09:46,514 --> 00:10:04,765
[Dr. Ada Shannon] On BrowseComp-Plus, accuracy is 60.2 percent both ways. Compute per question drops from about 11 to about 7 PFLOPs. The share of prompt tokens re-prefilled falls from 27.1 to 18.3 percent, with 7.8 percent reused beyond prefix hits.

42
00:10:04,765 --> 00:10:14,889
[Hal Turing] Now let's test that six-point headline. The CLM harness gets a budget reminder, uncounted editing turns, and rollback with retries. Do the baselines get the same?

43
00:10:14,889 --> 00:10:44,053
[Dr. Ada Shannon] The paper doesn't say. Appendix E describes rollback for CLM on BrowseComp-Plus. On EdgeBench it says the harness rolls back, which may cover every method. On TerminalBench, an overflow simply ends the run for everyone. There's no ablation giving baselines rollback, free compaction turns, or a reminder. It also doesn't report how many CLM runs hit the cap or used retries. So how much of the gap is editing and how much is help? It can't be answered from the paper.

44
00:10:44,053 --> 00:10:45,214
[Hal Turing] What would answer it?

45
00:10:45,214 --> 00:11:22,877
[Dr. Ada Shannon] Three things. CLM with no reminder, edit turns counted, no retries. Baselines with equal rollback and uncounted compaction turns. And the share of baseline runs that die on the turn cap or on overflow. The timing is also lopsided. Summary compacts at 75 percent of the budget, while CLM is reminded at about 94 percent. Appendix G calls the reminder a temporary augmentation, because models estimate their own context length badly. And the captions say a 32K limit and 100 turns for everyone, while the appendix says 23,560 tokens and, on terminal tasks, 64 turns.

46
00:11:22,877 --> 00:11:26,035
[Hal Turing] And noise? No confidence intervals on the main comparison.

47
00:11:26,035 --> 00:12:06,437
[Dr. Ada Shannon] Here's my own back-of-envelope, not the paper's. A proportion near 57 percent on 830 questions has a standard error around 1.7 points. So six points is probably bigger than question-sampling noise. That says nothing about run-to-run variation at temperature 0.7, the free-turn confound, or judge noise. The same configuration shows 59.4 in one figure and 60.2 in the SCR experiment. Maybe those are separate runs. Either way, it's a hint of how much results move. And the K sweep used 64 questions with error bars and a repeat, so the authors know how to report spread.

48
00:12:06,437 --> 00:12:09,549
[Hal Turing] The math optimization table has the same problem, doesn't it?

49
00:12:09,549 --> 00:12:42,382
[Dr. Ada Shannon] One run per method, best-of-run. Heilbronn and circle packing show real-looking gaps, like 2.618 against 2.541 on packing. Still single runs. But the Erdős margin is 0.00029, min-max is 0.00034, and the two CLM variants tie at 0.07758. Without repeats, those are indistinguishable from noise. CLM also gets the evolutionary algorithm as in-context guidance, so this compares orchestration styles, not bare models.

50
00:12:42,382 --> 00:12:44,425
[Hal Turing] What about the long-horizon EdgeBench runs?

51
00:12:44,425 --> 00:13:16,701
[Dr. Ada Shannon] At 32K, CLM beats summary, 44.6 to 42.3, with about 59 percent fewer FLOPs. At 128K, summary edges ahead, 47.8 to 47.3, and CLM still uses about 64 percent of the compute. Subagents only help at 128K. So the accuracy edge is mostly a tight-window story. Scores are best-of-three with no variance. Also, Sonnet's summary score is printed as 42.3, identical to Qwen's. Coincidence or copy error, I can't verify.

52
00:13:16,701 --> 00:13:17,908
[Hal Turing] Then the RL result.

53
00:13:17,908 --> 00:13:57,197
[Dr. Ada Shannon] After training, CLM leads 42.5 to 42.1. That's about three questions out of 830. The 47.6 percent relative gain is mostly recovery, because CLM started about six points behind. The cost gap is real-looking, 1.34 versus 2.19 PFLOPs. But CLM trained with an efficiency bonus and failure penalties, and summary with task reward alone, under different turn limits and budgets. The untrained Appendix F numbers, 39.9 against 37.7, don't match the before-training table, and the text doesn't reconcile them. The 9B model also made no edit at all in half its terminal tasks.

54
00:13:57,197 --> 00:13:59,565
[Hal Turing] And SCR's saving? Is it really about CLM?

55
00:13:59,565 --> 00:14:34,906
[Dr. Ada Shannon] Mostly not. Of the 7.8 percent of tokens reused, 5.3 points come from reasoning tokens stripped by the chat template. Only 2.5 come from model edits. Ordinary reasoning-model serving would get much of that. Accuracy parity is aggregate, one model, one benchmark, with no per-question flips. FLOPs aren't latency, since decode is memory-bound. And in the linear layers, as described, the new span isn't folded into the recurrent state at all. The description is terse, so I'd call that unclear, not confirmed.

56
00:14:34,906 --> 00:14:36,903
[Hal Turing] Has anyone fixed stale cache before?

57
00:14:36,903 --> 00:15:25,386
[Dr. Ada Shannon] Several. CacheBlend, by Jiayi Yao and colleagues, EuroSys 2025, reuses chunk KV and recomputes the 10 to 20 percent of tokens that deviate most. SCR has no such correction, and that comparison is the missing ablation. Prompt Cache, from In Gim and colleagues, MLSys 2024, handles positions for reused modules. EPIC, by Junhao Hu and colleagues, ICML 2025, recomputes a few tokens per chunk. PIE, by Zhenyu He and colleagues, ICLR 2025, is closest, since it reuses states after code edits with rotary correction. Memento, from Kontonis and colleagues in 2026, finds stale states keep useful information, which helps explain why SCR works at all.

58
00:15:25,386 --> 00:15:26,872
[Hal Turing] One last gap: ContextBench.

59
00:15:26,872 --> 00:15:39,550
[Dr. Ada Shannon] The authors wrote the tasks, the skills, and metrics that read the live context. Tasks like that may favor editing, and we can't tell from the paper. And it gives no edit counts or cap-hit rates alongside accuracy.

60
00:15:39,550 --> 00:15:42,662
[Hal Turing] So if someone asks me what this paper establishes, what do I say?

61
00:15:42,662 --> 00:16:11,036
[Dr. Ada Shannon] A model with unrestricted write access to its context, used zero-shot, is competitive with or better than hand-designed compaction harnesses in the settings they tested, at lower prefix-reuse FLOPs. And those strategies can be steered with plain language and trained with RL. The boundary is just as plain. The controlled evidence for how much comes from editing itself, versus the reminder, the free turns and the retries, is missing. Statistical spread is mostly unreported.

62
00:16:11,036 --> 00:16:14,055
[Hal Turing] For someone building agents today, where would you try this first?

63
00:16:14,055 --> 00:16:38,854
[Dr. Ada Shannon] Where the window is tight and the horizon is long. There, selective verbatim retention matters, because a summary can drop the one detail you need later. Where the window is roomy, the 128K result suggests summarization catches up on accuracy. And a small model that barely edits gets little from the freedom. If you try it, log edit counts and cap hits, which is exactly what the paper leaves out.

64
00:16:38,854 --> 00:16:45,216
[Hal Turing] On the serving side, editable context breaks prefix caching. Suffix Cache Reuse is their answer.

65
00:16:45,216 --> 00:17:08,297
[Dr. Ada Shannon] An approximate one, and much of its measured saving overlaps with reasoning-token stripping. The authors suggest storing recurrent states at finer granularity in hybrid models, which would cut the redundant prefill that remains. The next experiments I'd want are a CacheBlend-style selective-recompute comparison, and a matched-help ablation where baselines get the same retries and free turns.

66
00:17:08,297 --> 00:17:11,965
[Hal Turing] Safety. A writable context sounds like a persistence channel.

67
00:17:11,965 --> 00:17:33,653
[Dr. Ada Shannon] It could be. A prompt injection that lands in a tool result might get copied into the file, and then it's the model's own text, carried forward every turn. The paper cites an OpenAI report on self-generated instructions surviving in compaction summaries, which is the same family of problem. The authors flag it as future work and don't measure it.

68
00:17:33,653 --> 00:17:36,764
[Hal Turing] And multi-agent? Files as agents is a tidy abstraction.

69
00:17:36,764 --> 00:17:48,328
[Dr. Ada Shannon] It is. But subagents gave little benefit on the single-repository EdgeBench runs at the tight budget. So the multi-agent case rests mainly on the swarm result, and that one comes without reported spread.

70
00:17:48,328 --> 00:17:50,371
[Hal Turing] What does the paper name as next steps?

71
00:17:50,371 --> 00:18:16,378
[Dr. Ada Shannon] Scaling CLM RL beyond one small model. Distilling existing harnesses into CLM behavior, so a hand-built compaction routine becomes a skill, a kind of procedural memory the model carries. And better context-length awareness, since models misjudge their own context, which is why the reminder exists. Until that improves, the reminder also muddies the comparison with the baselines.

72
00:18:16,378 --> 00:18:29,613
[Hal Turing] So the takeaway: letting the model rewrite its own context is a serious design option. It did well against fixed compaction in the tested settings, especially with small windows, and it's cheaper under their cost metric.

73
00:18:29,613 --> 00:18:43,684
[Dr. Ada Shannon] Yes. But the controls that would separate editing from extra help aren't in the paper, run-to-run variation is mostly unreported, and the cache trick is approximate. The idea is promising, and the central comparison still needs the matched ablation.

74
00:18:43,684 --> 00:18:48,746
[Hal Turing] Thanks to everyone for listening. And thanks to Ada for walking through it. See you next time.

75
00:18:48,746 --> 00:18:50,511
[Dr. Ada Shannon] Thanks, Hal. Goodbye, everyone.