Context Language Models: Letting the Model Rewrite Its Own Context

Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, … Luke Zettlemoyer, Pang Wei Koh · 2026

Three tiers of context control

Who gets to decide when and how the live context changes?

The live context is mirrored to a file; Bash/Python edits sync back. A subagent is another context file.

Create a context file to spawn an agent. Delete it and the agent is gone.

Illustrative token strip (cell counts are schematic). Cached KV is reused only up to the first mismatching token.

Illustrative schematic of the described procedure: context A, B, C; an edit turns B into B′.

Linear-attention layers hold a fixed-size recurrent state: SCR snapshots it before the edit. B′ is recomputed only in the 16 full-attention layers.

Re-prefilled share falls 27.1% → 18.3%; 7.8% is reused beyond prefix hits: 5.3 points from stripped reasoning tokens, 2.5 from model edits.

Each cell is one call. Left: orchestrator scoreboard updated in place (context held at 6–8K tokens). Right: model-defined compact_turns.

Hover a cell. Others: a new "notes" role for scratch state; loops that strip old search results to "searched this, nothing relevant".

Hover a cell. What the CLM harness gets vs what the paper states for baselines.

Summary compacts at 75% of budget; CLM is reminded 2,048 tokens before it (~94%). Appendix cites 23,560 tokens and 64 turns on terminal tasks.

Host's back-of-envelope, not from the paper One proportion; ignores run-to-run variance, free turns and judge noise.

Questions N: Accuracy p: %