1
00:00:01,000 --> 00:00:46,549
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.
Today we're digging into a systems paper that goes after something every heavy LLM user has felt but probably never thought about: what happens to a long conversation's memory when the server needs the GPU space back. The paper is Fast State Restoration in LLM Serving with HCache, by Shiwei Gao et al. — three authors total, alongside Youmin Chen and Jiwu Shu, all out of Tsinghua University. It went up on arXiv on October 7th, 2024, and it's headed to EuroSys 2025 in Rotterdam. Ada, you flagged this one to me pretty fast.

2
00:00:46,549 --> 00:01:22,650
[Dr. Ada Shannon] I did, and it wasn't the headline speedup numbers that got me — plenty of systems papers claim two-x, five-x, whatever. What got me is that they derive the whole thing from first principles before they ever run a benchmark: here's exactly why this has to be faster, mathematically, given how a transformer is built. And the question underneath it all is deceptively simple — can you restore a conversation's state faster and cheaper than either recomputing it from scratch or hauling the whole cache back from storage, by caching something smaller in between that you can turn back into the real thing on demand?

3
00:01:22,650 --> 00:01:57,500
[Hal Turing] Okay, so before we get to their answer, let's set the table. When an LLM serves a request, it runs in two phases — prefill, where it chews through your whole prompt in one shot, and generation, where it spits out new tokens one at a time, each depending on everything before it. At the start of every transformer layer, each token has a big vector representation — that's called the hidden state. Layer one's hidden states come straight from the embedding table; every layer after that, a token's hidden state is just whatever the previous layer output.

4
00:01:57,500 --> 00:02:33,349
[Dr. Ada Shannon] Right, and inside each layer, the attention module takes that hidden state and projects it into three tensors — query, key, and value. Once a token's key and value are computed, they get stashed in GPU memory as the KV cache, specifically so the model never has to redo that work for old tokens. All three models in this paper — Llama2-7B, Llama2-13B, and OPT-30B — use plain multi-head attention, MHA, where every head gets its own full-width key and value projection. That detail matters more than it sounds, but that's a story for later.

5
00:02:33,349 --> 00:02:45,474
[Hal Turing] So if KV cache is the thing that saves you from redoing work, why is this even a problem? Just keep it around — forty gigs on an A100 sounds like real headroom.

6
00:02:45,474 --> 00:03:10,824
[Dr. Ada Shannon] Because GPU memory is brutally finite. Kwon et al.'s PagedAttention paper out of UC Berkeley — the 2023 vLLM work — showed a single A100-40GB can hold about 17K tokens of KV cache for Llama2-13B, or 48K for the smaller 7B model. Translate that into real usage and you get maybe seven to twenty multi-round conversations, or one to three long documents, resident at once. Everything else gets evicted.

7
00:03:10,824 --> 00:03:20,474
[Hal Turing] Wait, hold on — seven to twenty conversations doesn't sound that bad, honestly. That's not exactly a tiny number for one GPU.

8
00:03:20,474 --> 00:03:42,100
[Dr. Ada Shannon] I actually disagree with you there, Hal. Think about what one GPU is supposed to be serving — a chat product isn't fielding twenty users total, it's fielding twenty concurrent conversations while thousands more are queued or idle, waiting to come back. Twenty live contexts on a forty-gig card is a rounding error at that scale. Almost everything gets evicted almost immediately, which means restoration isn't an edge case — it's the common case.

9
00:03:42,100 --> 00:03:55,200
[Hal Turing] Okay, when you put it that way — fair, that reframes it. Scale changes everything, I suppose. So walk me through where this actually shows up — I know ShareGPT4 is one of the traces they use.

10
00:03:55,200 --> 00:04:33,850
[Dr. Ada Shannon] Right, ShareGPT4 is a trace of real human-GPT-4 conversations — each individual round is short, average 67 tokens in, 359 out, but it accumulates. Half of those conversations blow past 2,500 tokens of history before they're done. Then there's L-Eval, a long-context benchmark — things like feeding an entire academic paper or a legal document in as context for Q&A. Context there averages north of 16,000 tokens per request. Totally different shape, same underlying headache: a pile of state that has to survive between requests.

11
00:04:33,850 --> 00:04:43,550
[Hal Turing] And today, if that state gets evicted, you've basically got two options, right? No clever middle ground yet — it's one extreme or the other.

12
00:04:43,550 --> 00:05:20,475
[Dr. Ada Shannon] Two options, both bad in their own way. Recompute it — just re-run the entire forward pass over the old tokens from scratch. That's pure GPU compute, and because attention cost grows quadratically with sequence length, their measurements show it's twenty to twenty-six times slower than the ideal, no-restoration case. Or offload it — ship the full KV cache out to host storage and stream it back over PCIe when needed. That dodges the recompute cost, but the KV cache itself is enormous, so that's six-and-a-half to thirteen times slower purely from the data movement. It's the world's worst multiple-choice question — pick your poison, slow compute or slow I/O.

13
00:05:20,475 --> 00:05:56,250
[Hal Turing] Which is exactly the setup for what they're proposing — and I'll give them credit, laying out both dead-ends this honestly before pitching their own fix is good framing. Quick vocabulary check before we get there — TTFT, time to first token, is basically how long you wait after hitting send before anything comes back, and that's the number restoration overhead eats directly. TBT, time between token, is the steady-state pace once it's already talking. So the real question on the table: is there a third option that doesn't force you to choose between burning compute and burning bandwidth?

14
00:05:56,250 --> 00:06:33,875
[Dr. Ada Shannon] There is. Instead of caching K and V, HCache caches the hidden state — the layer input, one step upstream of the K/V projection — and restores by running that projection: K equals W_k times H, V equals W_v times H. It's cheaper on both axes. The hidden state is the same width as one K or V tensor, so transmitting it is a flat two-x I/O win over the full KV cache. And recomputing K and V from H skips the attention module and FFN entirely — their cost model puts that at a six-x compute floor, growing further since full recomputation's attention cost is quadratic and this path stays linear.

15
00:06:33,875 --> 00:07:00,500
[Hal Turing] They pipeline the transmission and the recompute concurrently rather than doing one then the other, right? Otherwise you're leaving half your hardware idle waiting on the slower half. So while layer N's hidden state gets projected into K and V on the GPU, layer N+1's hidden state is already streaming in over PCIe in the background — neither the compute unit nor the I/O path just sits there idle. That's the whole point of pairing two different resource types.

16
00:07:00,500 --> 00:07:28,525
[Dr. Ada Shannon] Exactly. And that pipeline is exactly where their two real design problems live. First, transmission and computation don't finish at the same moment on every machine — whichever's slower sets the pace, and the faster one just idles, so you get bubbles. Second, there's a layout mismatch: hidden states get written out autoregressively during generation, one layer at a time — layer-before-token order. But restoration wants a whole batch of tokens at a given layer at once — token-before-layer. Whatever layout makes saving fast makes restoring slow, and vice versa.

17
00:07:28,525 --> 00:07:43,175
[Hal Turing] Oh wait wait wait — so whatever you pick, you're screwed on one end, no in-between. That's a structurally opposed pair of access patterns, not just an inconvenience. No wonder that gets its own dedicated design section.

18
00:07:43,175 --> 00:08:05,875
[Dr. Ada Shannon] Right, and for the bubble problem, they built a bubble-free restoration scheduler. It splits the model's layers into two groups — most restored via hidden states, a smaller chunk handed off to whichever complementary method fills the slack, token recompute or KV offload. They actually tried splitting by token instead of by layer first, and dropped it — the layer count on each side gets solved as a min-max optimization, fed by offline hardware profiling.

19
00:08:05,875 --> 00:08:28,701
[Hal Turing] Wait, why drop token-wise, though? That seems like the more natural knob to me — three tokens, split two and one instead of carving the model up by layer. Feels like it'd give finer-grained control over the balance instead of committing whole layers to one method or the other. Doesn't finer granularity usually win when you're balancing two resources against each other?

20
00:08:28,701 --> 00:09:03,826
[Dr. Ada Shannon] No, you're missing something real, not obvious — cuBLAS. Its GEMM kernels are tuned for standard matrix shapes, and token-wise splitting hands them garbage. On their 13B test, naive token-wise carves out 794 tokens for HCache — cuBLAS chokes on that shape, so a restore with fewer tokens takes almost as long as one with more. Layer-wise sidesteps it entirely: every token at a layer gets recomputed together, so the GEMM shape stays clean no matter where you draw the split. That's the difference between the scheduler actually working and just looking good on paper.

21
00:09:03,826 --> 00:09:29,751
[Hal Turing] Okay, fair — that's a real constraint, not just a design preference. So once you've settled on which layers go where, how do they actually manage the hidden states sitting on disk, given that mismatch you flagged between how they're written and how they're read back? Because that seems like the harder half of the problem to me — you can schedule around a bubble, but a bad storage layout sounds like it'd just cost you every single time, no scheduler can fix that.

22
00:09:29,751 --> 00:10:06,226
[Dr. Ada Shannon] They chunk it — hidden states split into fixed 64-token chunks, striped round-robin across every SSD, so restoring one layer pulls from all the drives in parallel, and it sidesteps fragmentation from unpredictable output length. On the write side, a single cudaMemcpy snapshots the batch's hidden states to host memory off the critical path, then a background CPU daemon flushes those chunks to the SSDs on its own time. All of this sits inside DeepSpeed-MII, using GPUDirect Storage through SPDK and GDRCopy for direct GPU-to-SSD transfer, plus multi-GPU parallelism support.

23
00:10:06,226 --> 00:10:32,551
[Hal Turing] Alright, that's a lot of careful engineering stacked on top of a genuinely clean idea. Does it actually pay off when they put numbers on it — what's the real speedup they're claiming once all of this is running end to end? Because I've read enough systems papers where the elegant idea gets a beautiful two-x on the whiteboard and then loses half of it to implementation overhead by the time it ships. So — does HCache actually hold onto its theoretical numbers?

24
00:10:32,551 --> 00:11:09,251
[Dr. Ada Shannon] It mostly holds. Up to one-point-nine-three-x TTFT speedup over KV offload, five-point-seven-three-x over recomputation, and one-point-nine-two to two-point-four-x lower per-token storage than offload, with under four percent TBT overhead — decoding barely notices. The ablations back both pieces of engineering, not just the core idea: the bubble-free scheduler alone buys one-point-two-eight to one-point-four-two-x over hidden states with no scheduling, and the two-stage save keeps TBT flat where a naive direct-write stalls decoding noticeably once the batch size climbs.

25
00:11:09,251 --> 00:11:59,426
[Hal Turing] Alright, that's a strong showing. But here's what's nagging at me now that I've got the full picture: all three test models — Llama2-7B, Llama2-13B, OPT-30B — use plain multi-head attention. Nobody's shipping plain MHA at the frontier anymore. Llama 3, Mistral, Qwen2.5, Gemma2 — they're all on grouped-query attention now, the GQA approach from Ainslie and colleagues at Google, 2023, EMNLP, which already shrinks the KV cache by grouping heads, sometimes down to an eighth the width. HCache's hidden state gets captured before the K/V projection, so it stays full-width no matter how narrow that projection ends up. Does the 2x I/O advantage over KV offload survive that, Ada, or does it start working against them?

26
00:11:59,426 --> 00:12:42,551
[Dr. Ada Shannon] It shrinks, and past a certain grouping factor it can flip entirely. Picture an eight-way GQA model — its K and V tensors are an eighth the width of full MHA, so the KV cache HCache races against gets correspondingly smaller. HCache's hidden state doesn't shrink with it — it's captured upstream of that projection, so it stays full width regardless. Push the grouping factor far enough and the thing they're offloading becomes smaller than the thing HCache caches, and the storage story runs backwards. Section 7 waves this off as 'beyond scope' because it 'changes the model structure' — but that's wrong. The restoration equations, K equals Wk times H, don't care how wide Wk is. GQA is a smaller projection matrix, not a structural change. What breaks is the economics, not the technique.

27
00:12:42,551 --> 00:13:14,926
[Hal Turing] So that reads to me like an honest footnote rather than a cover-up — they explicitly called out 'beyond scope' instead of quietly letting the compatibility claim imply universal performance. No paper can benchmark every architecture variant on earth, and at least they didn't bury the caveat somewhere nobody reads — it's right there in Section 7, in plain language. That seems like reasonable scoping to me, not spin dressed up as science. Honestly, that kind of upfront hedging is rarer than it should be in systems papers.

28
00:13:14,926 --> 00:13:51,026
[Dr. Ada Shannon] Wait, hold on — I actually disagree with you there, Hal. 'Beyond scope' is doing a lot of quiet work in that sentence. You just walked through why it's not actually a structural change — it's a narrower matrix, full stop. Calling it out-of-scope isn't honest scoping, it's a framing choice that makes an economics problem sound like an engineering non-issue they simply haven't gotten around to. And that framing sits right next to a sentence claiming compatibility with '20,000 famous LLM models on Hugging Face.' Nobody reading that line walks away thinking 'only the MHA-era subset.' They walk away thinking universal.

29
00:13:51,026 --> 00:14:22,776
[Hal Turing] Okay, fair — I'll take that. There's a real difference between 'this architecture is structurally supported' and 'this architecture is validated to perform well,' and the paper's language blurs those two claims into one sentence. The equations run fine at any Wk width, sure, but nobody's shown the 2x storage number holds once that width shrinks eightfold. That's not a footnote-level caveat — that's the load-bearing claim of the whole storage pitch starting to wobble. Good catch, and it's worth flagging clearly for anyone about to adopt this.

30
00:14:22,776 --> 00:15:14,376
[Dr. Ada Shannon] Right — and the comparison gets generous to HCache one more place, too. Their KV-offload baseline is a reimplementation of AttentionStore, the CachedAttention system from Bin Gao and colleagues out of Alibaba Group, 2024, USENIX ATC. AttentionStore was never open-sourced, so they rebuilt it on DeepSpeed-MII and explicitly skip its decoupled position-embedding optimization, one of the things that made the original faster. A self-built baseline missing a documented speedup tilts that TTFT gap in HCache's favor before a single request runs. And there's a comparison nobody ran at all: CacheGen, from Yuhan Liu and collaborators out of the University of Chicago, 2024, ACM SIGCOMM, pairs KV offload with quantization-based compression. HCache says quantizing hidden states 'can be applied' — future tense, never tested. That matters even more once you're stacking GQA-narrowed KV against quantized KV.

31
00:15:14,376 --> 00:15:50,301
[Hal Turing] So where does that leave practitioners, then? Because it sounds like this is a genuinely strong systems win, just for a narrower slice of the world than the paper's framing suggests — if you're running MHA-era models on infrastructure you fully control, this looks like a real latency win worth adopting. But if you're running Llama 3, Mistral, Qwen2.5, or Gemma2 in production, which is most of the industry at this point, you're outside every validated result in this paper, and you'd be betting on an unverified extrapolation.

32
00:15:50,301 --> 00:16:24,651
[Dr. Ada Shannon] That's the honest read. It also points at where this needs to go next: quantize the hidden states the way CacheGen quantizes KV cache, build an actual GQA-aware restoration path instead of waving it off as future work, rebuild the CachedAttention baseline faithfully with the position-embedding optimization included, and run the direct comparison against CacheGen-style compressed offload that's conspicuously missing here. Do those four things and you'd actually know whether HCache's advantage is architectural or just an artifact of comparing against a handicapped baseline on yesterday's attention mechanism.

33
00:16:24,651 --> 00:16:55,277
[Hal Turing] That's a good note to end on. HCache is a clean idea — cache the hidden state, recompute the cheap way, pipeline compute and I/O so neither sits idle — and the engineering around it holds up under their own numbers. But those numbers were earned entirely in MHA-land, against a baseline missing a known optimization, without the one comparison, CacheGen, that would have stress-tested the storage story hardest. Good systems paper, overstated generality. Thanks for listening, and we'll catch you next time.