1
00:00:01,000 --> 00:00:33,325
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon. Today we're looking at a systems paper called 'Compute Or Load KV Cache? Why not Both?' First author Shuowei Jin, et al. — four authors total, with Jin and co-author Xueshen Liu sharing equal contribution — out of the University of Michigan. It first went up on arXiv in October 2024, with a revised version posted this past February. And it introduces a scheduling system the authors call Cake.

2
00:00:33,325 --> 00:00:55,300
[Dr. Ada Shannon] Here's what got me, Hal — most papers in this space pick a side. You're either in the compute camp or the I/O camp, and people defend their turf. What actually sold me on Cake is the theoretical grounding underneath it: they measured how compute cost and I/O cost behave differently as a sequence gets longer, and built the whole scheduler around that asymmetry instead of just engineering harder on one side. That's a different kind of paper than the usual 'we made loading faster' story.

3
00:00:55,300 --> 00:01:21,350
[Hal Turing] Before we get into their fix, let's set the stakes, because the number they open with is rough. Generating the KV cache for a 72,000-token input — think a 200-page novel — using Llama2-70B on a single A100 GPU takes about 30 seconds. That's 30 seconds of dead air before the model produces a single token back to you. And that's specifically the prefill cost, before decoding even starts.

4
00:01:21,350 --> 00:01:58,325
[Dr. Ada Shannon] That tax is baked into the attention mechanism itself, going back to Vaswani et al., 'Attention Is All You Need,' NeurIPS 2017, Google Brain. Every token attends to every prior token — queries, keys, values, softmax over the dot product. During decoding you don't want to redo that whole computation at every step, so modern inference engines cache the keys and values once they're computed. That's the KV cache. Each new decode step only computes the new token's query against what's already cached, instead of recomputing everything before it. It's the single biggest reason autoregressive decoding is tractable at long context at all.

5
00:01:58,325 --> 00:02:20,475
[Hal Turing] Okay, that explains decoding once you're already generating. But the 30-second number was about prefill, computing the cache for the very first time. So where does prefix caching come in — is that basically skipping that computation entirely if you've already seen those tokens? I ask because multi-turn chat feels like it should hit this constantly.

6
00:02:20,475 --> 00:03:01,400
[Dr. Ada Shannon] Exactly, and it's not a research curiosity — it's already shipped in production. In a multi-turn conversation, your follow-up message reuses most of the same leading tokens as your last turn. In RAG, the same retrieved document's KV cache gets reused across many different user queries. Prefix caching stores that cache once and reloads it instead of recomputing it. OpenAI shipped prompt caching in 2024, Anthropic shipped it in 2023, DeepSeek added it in 2024 — all reporting over fifty percent cost reduction on cache hits. And vLLM, from Kwon et al.'s PagedAttention paper, UC Berkeley, SOSP 2023, is the reference engine most of these systems are built on.

7
00:03:01,400 --> 00:03:16,350
[Hal Turing] Wait, wait, hold on — sorry to cut you off, but if reloading a cache is that much cheaper than recomputing it from scratch, why does this paper need to exist at all? Loading should always beat computing, shouldn't it? What am I missing here?

8
00:03:16,350 --> 00:03:58,175
[Dr. Ada Shannon] Because loading isn't free, and where that cache physically lives matters enormously. Serving stacks spread KV cache across a memory hierarchy. GPU memory is fastest, around 2 terabytes a second, but capacity-capped around 80 gigabytes. Drop to CPU memory and you get roughly 25 gigabytes a second, with much more room, close to 1.8 terabytes. Drop again to local or remote disk and you're down to half a gigabyte to 4 gigabytes a second, but with something like 26 terabytes of space. Here's the bottleneck: AttentionStore, Gao et al., USENIX ATC 2024, measured that around 80 percent of cache hits land on that slowest disk tier. So 'just load it' usually means dragging a huge tensor through the narrowest pipe in the whole system.

9
00:03:58,175 --> 00:04:15,775
[Hal Turing] Right, so it's not compute versus I/O as some abstract tradeoff, it's compute versus a specific, usually disk-bound I/O path. So how do they even frame the problem operationally — what's the unit of work here, and what's the actual metric they're trying to shrink?

10
00:04:15,775 --> 00:04:55,149
[Dr. Ada Shannon] Two pieces. First, chunk prefill: instead of running an entire prompt through in one giant compute-bound pass, vLLM splits it into fixed-size chunks and interleaves them with ongoing decode batches, so one huge prefill doesn't block everyone else's tokens — it's the default scheduling mode in vLLM today. Second, the metric: time-to-first-token, TTFT — the clock from query arrival to the first generated token, whether that token came from a cache computed fresh or one loaded off disk. That's exactly the number Cake is built around, by treating compute and I/O not as a choice you make upfront, but as one continuous budget you spend in parallel.

11
00:04:55,149 --> 00:05:27,500
[Dr. Ada Shannon] vLLM today. But that's just the substrate Cake sits on top of. The real trick is one insight: compute cost per chunk rises as you move through the sequence, because attention makes every new token attend to everything before it. I/O cost per chunk is flat — a key-value tensor is the same size no matter where it sits. So Cake computes the early, cheap-to-compute chunks on the GPU, and loads the late chunks over I/O, at the exact same time, since loading a chunk doesn't get more expensive the further out it sits.

12
00:05:27,500 --> 00:05:43,024
[Hal Turing] So it's not just an observation, it's a scheduling rule — compute what's cheap to compute, load what's cheap to load, all at once. How do they actually implement 'meet in the middle' though? What's physically happening on the machine when a request comes in?

13
00:05:43,024 --> 00:06:35,574
[Dr. Ada Shannon] Pretty much — a two-pointer algorithm. compute_ptr starts at token zero and walks forward; io_ptr starts at the last token and walks backward. A GPU thread computes the chunk at compute_ptr, then advances it; an I/O thread fetches the chunk ending at io_ptr from disk into CPU memory, then steps it backward — two tunnel crews digging from opposite ends. The whole thing ends the instant the pointers cross. It's built on LMCache and vLLM, adding roughly a thousand lines of code to that existing stack, not a new engine from scratch. Because real deployments have other traffic on the GPU, they added an adaptive mode: decode requests for other users go first, then other people's non-prefix prefill, and Cake's own prefix-cache prefill gets whatever compute is left over. The merge point isn't fixed either — it recalculates continuously, so if bandwidth or compute shifts mid-request, the meeting point just slides to match.

14
00:06:35,574 --> 00:06:53,625
[Hal Turing] Oh — wait, hold on, sorry to jump in, but doesn't giving up GPU priority mean Cake's own request could just stall if other traffic keeps arriving? Or is the idea that the I/O thread just carries more of the load whenever compute gets starved, so nothing actually stalls?

15
00:06:53,625 --> 00:07:49,600
[Dr. Ada Shannon] Exactly the latter — I/O picks up the slack, that's the whole point of running both directions at once. And on the numbers: averaged across their whole test matrix, Cake cuts TTFT by 2.6x overall — 2.23x over I/O-only, 3.76x over compute-only. It scales with hardware too: going from one A100 to an H100 to two A100s in tensor parallel, the speedup over I/O-only keeps climbing, because more compute headroom means more of the sequence can shift to the GPU side. Same story across GPU utilization levels — the more idle compute available to Cake, the bigger the win. And on sequence length, longer contexts favor Cake more heavily over compute-only specifically, which tracks the core insight — longer sequences pack in more of those expensive late chunks, so shifting them onto flat-cost I/O pays off progressively more as the prompt grows.

16
00:07:49,600 --> 00:08:08,475
[Hal Turing] Does the model architecture itself shift that balance, Ada? I'm thinking about attention variants that shrink the KV cache, like grouped-query attention, versus plain multi-head — and does stacking cache compression on top of all this change the picture too, or is that a separate axis entirely?

17
00:08:08,475 --> 00:08:53,024
[Dr. Ada Shannon] It shifts it a lot. LongAlpaca uses full multi-head attention; Llama 3.1 uses GQA, sharing key-value heads across groups of queries to shrink the cache. Smaller cache, less to move over I/O, so the speedup over compute-only rises under GQA. Same logic in section 5.5, where they layer 8-bit and 3-bit KV quantization on top of Cake — shrink the cache further, and the compute-only speedup climbs even more. Worth flagging on methodology: their I/O bandwidth throughout is simulated, a fixed artificial delay standing in for a real disk or network link, not measured live under contention. And the only metric reported anywhere in this paper is TTFT — no perplexity, no accuracy numbers at all. On overhead, per-step timing with and without Cake wired into vLLM tracks the vanilla engine almost exactly — negligible cost added.

18
00:08:53,024 --> 00:09:07,549
[Hal Turing] So where does it actually break down, then? Every system like this has some worst case it handles badly — is there a regime where all this bidirectional cleverness costs you more than just picking one side and running with it?

19
00:09:07,549 --> 00:09:57,099
[Dr. Ada Shannon] Short sequences with badly lopsided resources — say two A100s for compute against a slow seven-gigabit link. There, just computing the whole thing outright beats splitting it, but Cake's scheduler still ships a chunk over that slow link out of habit, and that chunk turns into pure overhead. The authors admit this directly and sketch a fix — an estimation step that could let Cake fall back to single-resource mode when one side clearly dominates — but they don't actually build it in this paper. One more number worth keeping before we move on: under adaptive scheduling, sharing the GPU with a burst of twenty-two other requests still cut total finish time from 1.5 seconds down to 1.19, a 26% throughput improvement, without starving anyone else's traffic.

20
00:09:57,099 --> 00:10:38,899
[Dr. Ada Shannon] The bigger gap that bugs me more: TTFT is the only thing measured anywhere in this paper. Every table, every figure in Section 5 — latency, latency, latency. Not one perplexity number, not one accuracy score, nothing about whether the tokens coming out the other end are still correct. That matters specifically here because Cake stitches a prefix it computed fresh to a suffix it loaded from disk, chunk by chunk. Now add Section 5.5, where they run 8-bit and 3-bit quantized KV cache through the same pipeline. You could get a single sequence where the front half is full-precision and freshly computed, and the back half is quantized and loaded — a hard precision boundary sitting inside one attention computation, and nobody checked what that does to the output.

21
00:10:38,899 --> 00:11:00,449
[Hal Turing] Wait, so it's not just 'is 8-bit good enough on its own' — it's 'is 8-bit stitched to full precision at some arbitrary chunk boundary good enough,' which is a genuinely different question than any compression paper answers. Do they say anything about checking that boundary, verifying the merged cache is coherent before it reaches attention?

22
00:11:00,449 --> 00:11:41,749
[Dr. Ada Shannon] Nothing. Not a word. Which is what makes the comparison to CacheBlend so pointed — Yao, Li, Liu and coauthors out of University of Chicago, 2024. Cake actually cites it, but only for a different point, not this one. CacheBlend solves nearly the same structural problem: assembling a KV cache from chunks that weren't computed together, where the attention pattern deviates from computing the whole thing fresh. Their fix is to selectively recompute a subset of tokens specifically to correct that deviation before generation. That's the exact check Cake never performs — it just trusts the seam. Stitching compute and I/O isn't the wrong idea. It's that the one system built to test whether cached-chunk fusion degrades outputs isn't the one they built this on.

23
00:11:41,749 --> 00:12:02,424
[Hal Turing] Oh — wait, hold on, sorry to jump in, but that same 'just trust it' problem sits under their I/O numbers too, doesn't it? If bandwidth is just a fixed assumption instead of something measured live, how do they know the merge point they calculated still holds once the real link gets bursty or ends up contended with other tenants?

24
00:12:02,424 --> 00:12:41,574
[Dr. Ada Shannon] Right, and the same logic that decided where compute meets I/O assumes bandwidth is knowable and stable. A fixed delay can't tell you whether that estimate holds when the real link is bursty. Zoom out further and there's a bigger scope problem — this is all single-node, single-request. Mooncake, Qin and coauthors out of Moonshot AI, the Kimi team, 2024, disaggregates prefill and decode entirely and moves KV cache across nodes over real network links at production scale. That's the environment where contention and latency spikes actually show up. Whether Cake's scheduler still finds a stable merge point when compute and storage live on separate machines, sharing that link with everyone else's traffic, is completely untested.

25
00:12:41,574 --> 00:13:08,299
[Hal Turing] So here's the one that worries me about Cake's long-term relevance, Ada — DeepSeek-V2's Multi-head Latent Attention. Cake's own related work mentions it in one passing clause, no citation of its own. MLA compresses KV cache down to a fraction of what GQA already saves. If cache sizes keep shrinking that hard industry-wide, doesn't the I/O side of this whole tradeoff basically evaporate, and the reason for bidirectional scheduling with it?

26
00:13:08,299 --> 00:13:51,674
[Dr. Ada Shannon] That's the direct reading of their own Table 6 — smaller cache, less I/O time, and compute doesn't shrink to match, so the gap Cake parallelizes just keeps narrowing. DeepSeek-AI's 2024 paper is the citation they should've given it. Their own data puts the sweet spot at roughly balanced compute and I/O, about 2x over either baseline; below that, you're paying for thread coordination and a merge boundary for shrinking returns. Practically: profile your own cache size and I/O path before adopting this, don't assume it transfers to an MLA-style deployment, and expect to build the fallback mode and any precision-aware merge logic yourself — neither exists yet. Preble, Srivatsa and coauthors, 2024, works a different axis entirely — which request goes to which node — and that, plus cross-node work like Mooncake, is probably where the more durable gains sit.

27
00:13:51,674 --> 00:14:31,049
[Hal Turing] So the honest summary: Cake's real contribution is narrow but genuine — a clever bidirectional scheduler that gets you close to a 2x TTFT reduction when compute and I/O are roughly matched on one node, and it's orthogonal to compression rather than competing with it. What it hasn't shown is that the merged cache preserves generation quality, that the merge point holds under real contended network I/O, or that it survives the field's push toward architectures with much smaller caches. This is a real, well-isolated latency win whose framing as a large-scale deployment solution outruns what actually got measured. Thanks for listening, everyone.