Context Language Models: Letting the Model Rewrite Its Own Context
Oct 13, 2026
This episode explores "Context Language Models," a paper that lets a model rewrite its own context window with ordinary Bash or Python edits, instead of relying on harness-defined truncation or a fixed menu of context actions. It covers the three tiers of context control, the serving catch (an edit in the middle or at the start of the context can cost several times more prefill compute than an append, because it breaks the shared prefix), and the ContextBench diagnostic. On the BrowseComp-Plus deep-research benchmark, the approach scores 59.4 percent, about six points above a summary-compaction baseline, with roughly a fifth less compute. The discussion also covers the models' emergent behavior, including an orchestrator that updates a subagent scoreboard in place 163 times and a model that writes and calls its own compaction function. Listeners get a critical look at whether these gains hold up, given that the cost metric is analytic FLOPs rather than measured latency.
Interactive Visualization: Context Language Models: Letting the Model Rewrite Its Own Context
Sources:
1. Context Language Models — Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh, 2026
http://arxiv.org/abs/2609.37725
2. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023
https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems
3. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024)
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
4. Recursive Language Models — Alex L. Zhang, Tim Kraska, Omar Khattab, 2025
https://scholar.google.com/scholar?q=Recursive+Language+Models
5. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (ACE) — Qizheng Zhang et al. (Stanford, SambaNova and collaborators), 2025
https://scholar.google.com/scholar?q=Agentic+Context+Engineering%3A+Evolving+Contexts+for+Self-Improving+Language+Models+%28ACE%29
6. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
7. SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2023 (NeurIPS 2024)
https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs+%28RadixAttention%29
8. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024 (EuroSys 2025)
https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion
9. Prompt Cache: Modular Attention Reuse for Low-Latency Inference — In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, Lin Zhong, 2023 (MLSys 2024)
https://scholar.google.com/scholar?q=Prompt+Cache%3A+Modular+Attention+Reuse+for+Low-Latency+Inference
10. EPIC: Efficient Position-Independent Caching for Serving Large Language Models — Junhao Hu, Wenrui Huang, Weidong Wang, et al., 2025
https://scholar.google.com/scholar?q=EPIC%3A+Efficient+Position-Independent+Caching+for+Serving+Large+Language+Models
11. Let the Code LLM Edit Itself When You Edit the Code (PIE) — Zhenyu He, Jun Zhang, Shengjie Luo, Jingjing Xu, Zhi Zhang, Di He, 2025
https://scholar.google.com/scholar?q=Let+the+Code+LLM+Edit+Itself+When+You+Edit+the+Code+%28PIE%29
12. Memento: Teaching LLMs to Manage Their Own Context — Vasilis Kontonis, Yuchen Zeng, Shivam Garg, et al., 2026
https://scholar.google.com/scholar?q=Memento%3A+Teaching+LLMs+to+Manage+Their+Own+Context
13. Scaling Long-Horizon LLM Agent via Context-Folding — Weiwei Sun, Miao Lu, Zhan Ling, et al., 2025
https://scholar.google.com/scholar?q=Scaling+Long-Horizon+LLM+Agent+via+Context-Folding
14. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents — Zijian Zhou, Ao Qu, Zhaoxuan Wu, et al., 2026
https://scholar.google.com/scholar?q=MEM1%3A+Learning+to+Synergize+Memory+and+Reasoning+for+Efficient+Long-Horizon+Agents
15. Meta-Harness: End-to-End Optimization of Model Harnesses — Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn, 2026
https://scholar.google.com/scholar?q=Meta-Harness%3A+End-to-End+Optimization+of+Model+Harnesses
16. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning — Lakshya A. Agrawal, et al., 2026
https://scholar.google.com/scholar?q=GEPA%3A+Reflective+Prompt+Evolution+Can+Outperform+Reinforcement+Learning
17. Efficient Memory Management for Large Language Model Serving with PagedAttention; and SGLang / RadixAttention — Kwon et al. (vLLM); Lianmin Zheng et al. (SGLang), 2023/2024
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention%3B+and+SGLang+%2F+RadixAttention
18. Self-Generated Prompt Injections in Compaction Summaries — OpenAI, 2026
https://scholar.google.com/scholar?q=Self-Generated+Prompt+Injections+in+Compaction+Summaries
19. Training Language Models to Reason Efficiently; and Think Dense, Not Long (DDCA) — Daman Arora, Andrea Zanette; Keqin Peng et al., 2026
https://scholar.google.com/scholar?q=Training+Language+Models+to+Reason+Efficiently%3B+and+Think+Dense%2C+Not+Long+%28DDCA%29
20. Titans / Mamba / Gated DeltaNet and learned-memory architectures — Behrouz et al.; Gu & Dao; Yang et al., 2024–2025
https://scholar.google.com/scholar?q=Titans+%2F+Mamba+%2F+Gated+DeltaNet+and+learned-memory+architectures
Interactive Visualization: Context Language Models: Letting the Model Rewrite Its Own Context
1. Context Language Models — Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh, 2026
http://arxiv.org/abs/2609.37725
2. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023
https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems
3. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024)
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
4. Recursive Language Models — Alex L. Zhang, Tim Kraska, Omar Khattab, 2025
https://scholar.google.com/scholar?q=Recursive+Language+Models
5. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (ACE) — Qizheng Zhang et al. (Stanford, SambaNova and collaborators), 2025
https://scholar.google.com/scholar?q=Agentic+Context+Engineering%3A+Evolving+Contexts+for+Self-Improving+Language+Models+%28ACE%29
6. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
7. SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2023 (NeurIPS 2024)
https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs+%28RadixAttention%29
8. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024 (EuroSys 2025)
https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion
9. Prompt Cache: Modular Attention Reuse for Low-Latency Inference — In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, Lin Zhong, 2023 (MLSys 2024)
https://scholar.google.com/scholar?q=Prompt+Cache%3A+Modular+Attention+Reuse+for+Low-Latency+Inference
10. EPIC: Efficient Position-Independent Caching for Serving Large Language Models — Junhao Hu, Wenrui Huang, Weidong Wang, et al., 2025
https://scholar.google.com/scholar?q=EPIC%3A+Efficient+Position-Independent+Caching+for+Serving+Large+Language+Models
11. Let the Code LLM Edit Itself When You Edit the Code (PIE) — Zhenyu He, Jun Zhang, Shengjie Luo, Jingjing Xu, Zhi Zhang, Di He, 2025
https://scholar.google.com/scholar?q=Let+the+Code+LLM+Edit+Itself+When+You+Edit+the+Code+%28PIE%29
12. Memento: Teaching LLMs to Manage Their Own Context — Vasilis Kontonis, Yuchen Zeng, Shivam Garg, et al., 2026
https://scholar.google.com/scholar?q=Memento%3A+Teaching+LLMs+to+Manage+Their+Own+Context
13. Scaling Long-Horizon LLM Agent via Context-Folding — Weiwei Sun, Miao Lu, Zhan Ling, et al., 2025
https://scholar.google.com/scholar?q=Scaling+Long-Horizon+LLM+Agent+via+Context-Folding
14. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents — Zijian Zhou, Ao Qu, Zhaoxuan Wu, et al., 2026
https://scholar.google.com/scholar?q=MEM1%3A+Learning+to+Synergize+Memory+and+Reasoning+for+Efficient+Long-Horizon+Agents
15. Meta-Harness: End-to-End Optimization of Model Harnesses — Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn, 2026
https://scholar.google.com/scholar?q=Meta-Harness%3A+End-to-End+Optimization+of+Model+Harnesses
16. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning — Lakshya A. Agrawal, et al., 2026
https://scholar.google.com/scholar?q=GEPA%3A+Reflective+Prompt+Evolution+Can+Outperform+Reinforcement+Learning
17. Efficient Memory Management for Large Language Model Serving with PagedAttention; and SGLang / RadixAttention — Kwon et al. (vLLM); Lianmin Zheng et al. (SGLang), 2023/2024
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention%3B+and+SGLang+%2F+RadixAttention
18. Self-Generated Prompt Injections in Compaction Summaries — OpenAI, 2026
https://scholar.google.com/scholar?q=Self-Generated+Prompt+Injections+in+Compaction+Summaries
19. Training Language Models to Reason Efficiently; and Think Dense, Not Long (DDCA) — Daman Arora, Andrea Zanette; Keqin Peng et al., 2026
https://scholar.google.com/scholar?q=Training+Language+Models+to+Reason+Efficiently%3B+and+Think+Dense%2C+Not+Long+%28DDCA%29
20. Titans / Mamba / Gated DeltaNet and learned-memory architectures — Behrouz et al.; Gu & Dao; Yang et al., 2024–2025
https://scholar.google.com/scholar?q=Titans+%2F+Mamba+%2F+Gated+DeltaNet+and+learned-memory+architectures
Interactive Visualization: Context Language Models: Letting the Model Rewrite Its Own Context