Checkpoint the memory, split the sequence Illustrative
Drag N: the number of segments whose memory is frozen and cached. N = 1 is the plain recurrent model; N = one per token is the gated-attention end of the dial.
What is stored as context grows Illustrative
Conceptual memory units (not bytes). Fixed state vs KV cache vs cached memories with 256-token segments. Hover the chart.
Four ways to combine cached memories Illustrative weights
γᵢ = softmax of the dot product between a projection u of the current token and each segment's mean-pooled tokens (paper Eq. 10). Vectors below are toy 2-D stand-ins.
Section 3.4 design fork From the episode
The paper says each option has pros and cons. The episode could not find which one produced the reported soup numbers.
Cost curves: linear to quadratic Illustrative
Relative operation counts, log y-axis. Recurrent ≈ L; attention ≈ L²; Memory Caching ≈ N·L with N = L / segment; Log-Linear++ ≈ L log L. Constants omitted.
Where the per-token cost goes Illustrative
Per-token reads of cached memory at L = 16K, by variant, in units of "memory reads" (SSC shown with k = 2).
Reported results Numbers from the episode
Values quoted in the discussion; "~" values were given approximately. Single runs, no error bars reported.
How much of GRM's retrieval gain does Soup recover? Derived from reported averages
Why soup equals GRM for linear memory, and not for MLP Illustrative · random weights
Live computation with small random weights. GRM mixes outputs Σγᵢ fᵢ(q). Soup runs one pass through merged weights f(q; Σγᵢθᵢ).
Do the cached memories share a loss basin? Illustrative schematic
Loss along the straight line between two memories' weights. Shapes are conceptual; the paper reports no interpolation-loss check.
Evidence checklist for Memory Soup From the episode