LeanSparse: Shrinking GPU Memory for Sparse-Attention LLM Serving

Sparse attention cuts reads, not storage

Each query reads a few entries, but a skipped token may be chosen later, so all history stays. Toggle the view and step the query.

Three layer types in V4-Flash

Click a layer type. LeanSparse optimizes CSA only.

The indexer's keys outgrow the hot KV buffer

Per indexed layer. Reported: FP8 indexer keys = 7.2× the hot buffer at 512K tokens. Other lengths: linear scaling, Illustrative

Why not disaggregate prefill and decode?

The paper's objection: separate pools duplicate model weights while HBM is scarce. Layout: Illustrative

Prefill: stream one layer's history at a time

Two alternating GPU buffers: one serves the current layer while the other loads the next from host. Nothing scales with layer count. 6 layers shown: Illustrative

Does the load hide behind compute?

Reported model: 4K chunk ≈ 5.2 ms compute per layer, ≈ 2.9 ms load at 32 GB/s (assumed zero delay, not a measurement). Other chunk sizes: Illustrative scaling

The admission dilemma

Prefill needs a big temporary workspace. Proportions: Illustrative

Product-quantizing an indexer key

128-dim key (132 B with scale) → 32 pieces of 4 dims → one byte each into a per-layer 256-entry codebook = 32 B. Hover a piece. Code values: Illustrative

Coarse-to-fine decode search

Switch the selection strategy. Recall figures are reported.

Keeping host fetches in check

Per CSA layer GPU caches plus a miss filter.

Reported results

DeepSeek-V4-Flash, 4×H100, TP=4. Multi-turn test vs reimplemented ECHO; decode test vs HiSparse (prefix reuse off).

Context length changes the trade

Decode vs HiSparse, reported values.

Quality checks (≤256K, greedy, non-thinking)

Accuracy within half a point of unmodified SGLang on all four tasks. Significance reporting differs.

How firm is each claim?

Hover a row. Qualitative reading of the episode's critique, not a paper score

What comes next (episode's list)