A token's query reads only k rows from a huge learned table. Step through it.
Product keys: search N keys by scoring ~2√N
Hover the grid. Each cell is one key, built from one row sub-key and one column sub-key. Illustrative scores
How the search cost grows
Counts sub-key scores only; at n=1,024 the table has 2^20 ≈ 1M values. Original idea: Lample et al., 2019.
From vanilla memory layer to Memory+
Inside a Memory+ block tap a box
Tap a box to see what changed.
How many memory layers? Illustrative shape
Closed-book QA results
Hover a bar.
Who was compared
Dense and MoE: each trained on 1T tokens. Single runs, no error bars.
What "matched FLOPs" leaves out
Kernel speed: which comparison?
Sharding the table across GPUs
Values split along the embedding dimension; indices shared, partial sums returned. GPU count here is illustrative.
What would settle it
tokens/s at equal qualityMFUstep time vs dense and MoEseveral memory-group sizesiso-parameter / iso-wall-clockoptimizer state of the tablekey usage / dead slotsfine-grained MoE baseline