Sparse attention cuts compute, not storage. ECHO moves the KV cache to host DRAM and guarantees an exact top-k. Visuals marked Illustrative are schematics, not paper data.
1 · Capacity wall
Capacity figures are reported: ~655K tokens per worker on 8×H200, at most ~380K per data-parallel worker on 8×H20 with 4-bit AWQ. Host pool 1.8M tokens (~1 TB) is the evaluation setup.
2 · The top-k dependency Illustrative timing
Five fixed-size integer tensors on the GPU
Hover or tap a tensor. Nothing changes length, so the decode step stays in one CUDA graph.
Select a tensor above.
One layer, one step: free → allocate → recall Illustrative
Per-layer GPU cache because each layer's indexer picks different tokens; host copy is per model. Metadata ≈ 10 MB/layer, ~610 MB total. No write-back on eviction since host holds everything. Allocator microbenchmark: ~12 µs (SGLang) vs 6–8 µs (ECHO).
Decode: fetch above a predicted threshold Illustrative scores
Synthetic indexer scores for 40 tokens; K=10 stands in for k=2048. The threshold predictor is an EMA (α = 0.5) of the previous k-th score.
Predicted threshold θ̂:
Early-fetch cap: tokens
Prefill: one-round histogram Schematic
While the indexer scores query block i+1, tokens of block i are fetched. Real kernel: 256 bins over the top 8 bits of each score, shifted by the predicted threshold. Drawn with 32 bins for legibility.
Fused indexer kernel (extends DeepGEMM)
Reported results
Setup
Is "lossless" argued or measured?
Based on the sections discussed in the episode. Hover or tap a row.