Every attention layer produces a key/value pair per token. Restoring a long prior context back into GPU memory sits directly on the path to the first output token — so it dominates Time-to-First-Token (TTFT) for chatbots, coding agents, and retrieval pipelines.
Where restoration sits in the pipeline
Prefill materializes the KV cache; decode reuses it. Restoration happens between the two.
KV cache size grows with context
Real traces regularly exceed 20K tokens — a couple GB for Qwen3-8B, ~10GB for Llama-3.1-405B.
Neither extreme wins alone
Recompute cost is quadratic; load cost depends entirely on real bandwidth. Hover the curves.
CacheFlow's core primitive: a two-pointer "meet in the middle" search, applied along two different axes. One pointer recomputes, the other loads from cache, and an offline-profiled crossover length decides which axis dominates.
Splitting the cached prefix into chunks
512-token chunks, matched to FlashAttention's block size. Recompute pointer advances forward, load pointer advances backward — hover a chunk.
The third axis: when layers are pipeline-split across GPUs, stage two normally waits for stage one to finish. CacheFlow caches only the small boundary activation at each stage seam, so every GPU reconstructs its own shard independently, in parallel.
Sequential handoff vs. parallel restoration
Hover a bar. Caching the boundary activation removes the pipeline dependency entirely.
Harmonic-mean split, linear speedup
Same math as two resistors in parallel — the optimal recompute/load split, divided across S GPU stages.
Real serving batches many requests at once, all fighting over the same GPU and I/O link. Algorithm 1 keeps per-request pointers but hands the shared I/O channel, chunk by chunk, to whoever has the most quadratic recompute cost left to avoid.
Ranking requests by remaining cached length
The longest remaining request always wins I/O priority — hover a row.
GPU vs. I/O utilization, held simultaneously
vLLM: GPU-bound. LMCache: I/O-bound. CacheFlow keeps both busy at once — hover a cell.
Built into vLLM + LMCache, no model changes. Tested on Qwen3-8B, Qwen3-30B-MoE, and Llama-3.1-8B against LMSys-Chat, WildChat, and SWE-Bench traces. Gains widen under contention — P90 to P99 — rather than shrinking, which is the scheduler doing its job.
Lower Time-to-First-Token vs. every baseline
10-62% lower TTFT depending on workload — typically 1.1x to 1.7x, not a cherry-picked headline.
Ablations: gains grow under harder conditions
Lower bandwidth, cheaper GPUs, bigger batches all increase the advantage. Removing multi-GPU restoration is the one regression.
What's still unverified
Two hosts disagreed on the headline number, then agreed the systems story is earlier-stage than the abstract sounds.