FlatCake: Flat KVCache Storage Beats Tiered Memory Hierarchies

arXiv 2610.38355 Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, Youwei Zhuo · 2026
Read the paper · Interactive page

Tiered vs. flat: where does a block live?

Toggle the architecture. Flat = decide medium at write time, never migrate.

Why delivered hit rate matters

A block on disk that arrives late is recomputed anyway. Step through a request.

Premise 1 fails: DRAM does not filter cold data

Reported in the paper's Table 1 (as quoted in the episode).
Hover a bar.

Premise 2 fails: staging and scatter

Session blocks on the drive. Illustrative block placement, not measured data.

Causal chain behind the collapse

As described by the authors; click a link.
Click a box.

GPUDirect Storage: transport vs. layout

GDS removes the bounce buffer. An NVMe read still needs a contiguous range, so scattered blocks still mean many small commands.

FC-GVA: placement at write time

Toggle signals. Rule order follows the episode's description. Thresholds tau and L_agent: no values published.

64-bit block address

Hover segments. One hash lookup returns the address; medium bits pick DRAM copy vs SSD read.
Hover a field.

FC-CIO: session-contiguous, layer-major extent

Toggle read pattern. Gaps under 64 KiB are bridged. Illustrative block counts.

FC-MAS: two queues

Results on the Claude Code trace

8×B200, 1 TiB DRAM, 4 TiB SSD per system, SGLang, DeepSeek-V4-Flash, ~99,000 requests / 393 sessions, 0.1–1.0 req/s. Reported values only.

Ablation: remove FC-MAS

Only component ablated on the agentic trace: about 10–11% worse mean TTFT, 27% worse P99.

What the experiments test

Coverage matrix from the episode's critique. Hover a cell.
Hover a cell.

Placement risk: a wrong choice sticks

Step a session through time. Illustrative scenario.

Experiments that would settle it

References