Sparse Delta Memory: Scaling Linear RNN State Without Extra Compute

This episode examines "Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity," a Meta FAIR paper (with Inria, ENS-PSL, and Tübingen collaborators) that tackles the memory-capacity wall in linear-attention architectures. The discussion traces the lineage from Katharopoulos-style linear attention through Gated DeltaNet's delta-rule updates to Product-Key Memory's sparse addressing, showing how the paper combines them: applying a gated delta update over a product-key-addressed memory table so only a handful of slots are read and written per token, letting the state size grow a thousandfold with no added FLOPs. It also covers the paper's second contribution — making the initial memory state a learned parameter so pretrained knowledge can live directly in that expanded memory rather than only being built up in-context. Throughout, the hosts stress a key conceptual distinction between compressed fixed-size RNN state and a growing transformer KV cache, framing the paper's core question as whether sparse addressing can close the long-context recall gap with transformers without touching compute or parameter budgets. Listeners interested in efficient long-context architectures will find the mechanistic walkthrough and the 8-billion-parameter validation particularly relevant to real bottlenecks in agentic coding and long-document workloads.