Mooncake-style KV stack: swap only the far tier
Pick a backing tier. The runtime policy is unchanged; the package trade rides along.
Restore flow, step by step
KV cache size per request
2 × tokens × layers × KV heads × head dim × bytes (example: 80 layers, 8 KV heads, d=128, FP16)
Three conditions for a faster far tier to pay off
C1 · Exposed fraction f caps the win
Latency change ≤ f·(1 − 1/k) for a medium k× faster
f =
Worked example
10 s request, 0.5 s stalled on restore → f = 5%. Drag k.
k =
C2 · ρ = useful bytes read / byte written
Write 10 GB, 5 GB ever read back → ρ = 0.5. Break-even ≈ 1.
GB read back:
C3 · Thermal limit
3D-ICE single-stack steady state, author-chosen 80 °C limit
HBF vs capacity-matched SSD: every request metric loses
Per-model bars are illustrative within the reported ranges (latency 2–5.5×, throughput −4–34%, goodput 1.1–2.7×). Only GLM/B200 latency (~2.5×) is quoted exactly.
GPU:Metric:
Latency slowdown heatmap: model × offered load (req/s)
Hover cells. Mock gradient: queues build as load rises and the near tier admits fewer requests.
Where did the time go? (GLM · traceA · B200 · 32 rps)
Media-only sweep (HBF latency 3.75× faster)
30/300 µs → 8/80 µs read/write
Writes outweigh reads on every trace
Write-to-read ratio (coder 1.14, traceB 4.90 reported; others interpolated for display)
ρ vs break-even
ρ = 1 / write-to-read ratio. Range 0.2–0.88, all below 1.
Reuse is concentrated, so hot blocks never reach flash
Mock heatmap of block reuse (traceB-like). Hot blocks stay in the near tier; flash gets the write-once cold tail. Hover.
Block size: batching helps SSD, not HBF
Endurance budget (TB/day)
Trace writes 48–140 TB/day exceed both budgets
Near-memory attention: a 43% operator win mostly vanishes
Engine charged no area or power. Only 15.48% of KV sits in HBF; each step pays a 12 µs access.
What the evidence supports
Open experiments (click to expand)
What to do with it: placement test by object type
References
- HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving — Li et al., 2026 (arXiv 2608.11668)
- LMCache — Liu, Cheng et al., 2025
- Strata — Xie, Xu, Zhao et al., 2025
- Tutti — Qiu, Hu et al., 2026
- KVCache Cache in the Wild — Wang, Han, Wei et al., 2025
- H3: HBM + HBF hybrid — Ha, Kim, Kim, 2026
- FlashAccel — Wang, Xue et al., 2026
- HBFSim — Hu, Yang et al., 2026
- Exploring HBF for Modern LLM Inference — Son, Park et al. (with Mutlu), 2026
- CachedAttention — Gao, He et al., 2024
- Data Retention in MLC NAND Flash — Cai, Luo, Haratsch, Mai, Mutlu, 2015