1
00:00:01,000 --> 00:00:36,700
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.
Today we're digging into IMPRESS: An Importance-Informed Multi-Tier Prefix KV Storage System for Large Language Model Inference. That's Weijian Chen et al. — nine co-authors total — out of Zhejiang University and Huawei Cloud. It ran at USENIX FAST 2025, the File and Storage Technologies conference, held in Santa Clara this past February. It's a systems paper, not a modeling paper, and Ada, I know you were genuinely into this one.

2
00:00:36,700 --> 00:01:06,224
[Dr. Ada Shannon] I was — and it's not the usual 'shaved a few percent off a benchmark' systems paper. What got me is the whole design falls out of a measured property of trained attention, not a hunch. They looked at how attention weight is distributed across heads before writing a line of storage code, checked whether it held across model sizes, and only then built the system around it. We're not getting into the mechanics of that today, that's later — but there's real empirical grounding under the plumbing here, not just a headline number.

3
00:01:06,224 --> 00:01:26,899
[Hal Turing] Okay, so before we get into any of that — what's the actual question they're chasing? Because 'storage system' can mean a lot of things, and I want listeners to have the actual target locked in before we start talking about disks, caches, and where all this data actually lives while a model's trying to answer someone in real time.

4
00:01:26,899 --> 00:02:05,799
[Dr. Ada Shannon] Fair. The question is: can you cut the delay before an LLM produces its first output token, when the cached context has spilled out to disk, by loading only the important tokens' data instead of everything — with barely any extra disk trips to figure out which tokens matter? Think about why apps hand the model a huge chunk of context before the actual question shows up. Retrieval-augmented generation prepends retrieved documents. Multi-turn chatbots prepend the whole prior conversation. GPT-style plugin frameworks prepend tool definitions. Few-shot prompting prepends worked examples to steer output format. All of that goes in before the query, every single request.

5
00:02:05,799 --> 00:02:23,299
[Hal Turing] Right, and I assume that's not free — bigger input means slower response, and probably not in a nice linear way either. Walk me through why that specifically hits what you'd call the time-to-first-token, since that term gets thrown around a lot without anyone defining it.

6
00:02:23,299 --> 00:03:02,024
[Dr. Ada Shannon] So a transformer answers in two phases. Prefill processes the whole input prompt at once and produces the first output token — that delay is TTFT, time-to-first-token. Decoding is everything after, one token at a time. Prefill cost grows superlinearly with prompt length, because every token attends to every other token. The paper's own example: a GPT-plugin system called Chameleon prepends over 2,600 tokens of tool definitions before a roughly 750-token query, more than four times the normal length — and on OPT-30B that stretched TTFT by nine times.

7
00:03:02,024 --> 00:03:17,474
[Hal Turing] Wait — hold on, that's nine times TTFT just from the plugin scaffolding, before the model's even answered anything? That seems like it should push everyone toward caching that prefix instead of recomputing it every time.

8
00:03:17,474 --> 00:04:09,899
[Dr. Ada Shannon] Exactly the move the field made. If requests share the same leading text — same documents, same tool prompt, same few-shot block — you compute the keys and values once and reuse them, skipping that part of the forward pass. A radix tree tracks this, quickly finding the longest shared prefix already cached. On disk, those KVs aren't stored token by token — they're grouped into chunks, batches of consecutive tokens bundled into one object, because that's what makes disk reads and PCIe transfers efficient. That's also the tier where vLLM's PagedAttention, from Kwon et al. at UC Berkeley in 2023, stops — it assumes everything still fits in GPU or CPU memory. Once traffic is big enough, prefix KVs outgrow both, and the only place left is disk. AttentionStore, from Gao and colleagues in 2024, pushed prefix KV onto that disk tier and tried to prefetch it back based on scheduler predictions.

9
00:04:09,899 --> 00:04:28,500
[Hal Turing] And I'm guessing disk isn't exactly known for being fast, especially compared to GPU memory. So does adding a disk tier on top of GPU and CPU just trade one bottleneck, compute, for another, I/O? That sounds like it could cancel out whatever you gained from not recomputing.

10
00:04:28,500 --> 00:05:12,524
[Dr. Ada Shannon] That's precisely the finding that motivates this paper. I/O latency getting a prefix KV from SSD to GPU is rarely hidden by the query's own computation, and accounts for 51 to 98 percent of total TTFT. Disk-tiered caching alone barely helps. Separate research — H2O, from Zhang and colleagues in 2023 — found that not every token's key and value matters equally to output quality, and you can discard plenty of them with little accuracy loss. That work applied the idea during decoding. IMPRESS's premise: apply that same importance signal during prefill, so you never load the unimportant KVs from disk at all. They're claiming up to 2.8x lower TTFT with comparable accuracy. How they find 'important' tokens without paying the I/O cost to look — that's where it gets interesting.

11
00:05:12,524 --> 00:05:42,099
[Dr. Ada Shannon] That's the H2O trick, yes — sum a token's attention weight down its column across the sequence, and that column sum is a solid importance score you can use to safely evict the low scorers during decoding without hurting quality much. IMPRESS takes that exact metric and points it somewhere else entirely: instead of deciding what to throw away mid-decode, it decides what's even worth loading off the SSD before prefill starts. Neat reuse of an existing signal — but repurposing it at load time exposed two problems nobody had actually solved.

12
00:05:42,099 --> 00:05:58,199
[Hal Turing] Wait, hold on — isn't that circular, though? To know which tokens are important you need the attention weights, and to get attention weights you need the keys loaded onto the GPU, which is exactly the disk read you're trying to avoid in the first place.

13
00:05:58,199 --> 00:06:40,225
[Dr. Ada Shannon] That's challenge one, exactly. Existing importance methods load every key into GPU memory first to compute the scores, so you pay the full I/O cost before you've decided anything's skippable. Challenge two is independent of that — once you do know which tokens matter, prefix KVs are stored in fixed chunks for disk-read efficiency, and importance doesn't respect chunk boundaries. They measured it directly: mark half a prefix as important, and only 46% of the KVs inside a loaded chunk turn out to be the ones you wanted — 2.2x read amplification. And a chunk's access frequency tells you nothing about how many important KVs it's carrying. So caching by recency or frequency, which is what every existing system does, is optimizing for the wrong variable.

14
00:06:40,225 --> 00:06:51,800
[Hal Turing] Oh wait wait wait — so your busiest, most frequently-hit chunk could be almost entirely dead weight importance-wise, and a frequency-based cache would keep it around anyway?

15
00:06:51,800 --> 00:07:33,800
[Dr. Ada Shannon] Exactly that disconnect. Take challenge one first, since it's more fundamental. Their finding: within a given transformer layer, the set of important token indices — the top attention scorers — is nearly identical across different heads. They quantify the overlap with the Jaccard index, intersection over union of the two index sets, and on OPT-6.7B's middle layer the average comes out above 0.95. It generalizes too — holds across OPT-6.7B, 13B, and 30B, and across different selection ratios, though the similarity drops the lower the retention percentage — around 0.48 at a 10% ratio on OPT-30B versus 0.68 at 40%. Smaller models and deeper layers show weaker agreement as well, but still well above what random token selection would produce.

16
00:07:33,800 --> 00:07:47,300
[Hal Turing] So the move is: don't check every head individually — sample a few, and if those few agree with each other, assume the rest of the heads in that layer would land on the same answer without ever reading their keys.

17
00:07:47,300 --> 00:08:29,425
[Dr. Ada Shannon] That's the algorithm — similarity-guided identification. They designate a small set of probe heads, settled on three, using just the first three heads in each layer, since which three you pick doesn't matter once the similarity holds. Compute attention from only those heads, measure their mutual Jaccard agreement, and if it clears a threshold, propagate that one answer to every other head in the layer — skip computing attention for them entirely. If agreement falls short, fall back to the full computation for that layer, which happens under 20% of the time on average. The threshold itself isn't arbitrary — they derive the Jaccard score you'd expect from pure random token selection at a given retention ratio, then scale it up by a factor they tuned to 0.6, balancing how much TTFT drops against how much accuracy holds.

18
00:08:29,425 --> 00:08:41,650
[Hal Turing] Okay, that handles finding the important tokens cheaply. What about challenge two, then — the chunks mixing important and unimportant KVs, and caching that can't tell the difference?

19
00:08:41,650 --> 00:09:24,500
[Dr. Ada Shannon] Two fixes there. KV reordering runs periodically — every ten minutes or so, asynchronously, off the critical path — and repacks each radix-tree node's tokens by importance, so important KVs land densely in the same chunks instead of one per chunk. Since reordering would normally scramble the token order the radix tree needs for prefix matching, they scope it strictly inside each tree node and keep a small mapping list to translate back during prefix lookups. On the caching side, instead of ranking chunks by recency or frequency alone, they score each chunk by access frequency times the fraction of important KVs it holds, and manage GPU and CPU caches with a dual min-heap — so a less-hot but importance-dense chunk can outrank a hot, mostly-irrelevant one for the scarce GPU slot.

20
00:09:24,500 --> 00:09:33,650
[Hal Turing] Alright, tie it together for me — what did they actually build this on, and does the whole stack deliver where it counts, at the TTFT line?

21
00:09:33,650 --> 00:10:50,325
[Dr. Ada Shannon] Built on FlexGen — Sheng and colleagues' system from ICML 2023 — modified for prefix reuse and importance scoring, with new classes for the reordered KV layers and the scored cache. Test rig: one A100 with 80 gigs of HBM, one 2-terabyte SSD, OPT at 6.7, 13, and 30 billion parameters plus Llama2-7B and 13B, on PIQA, RTE, COPA, and OpenBookQA, using prepended few-shot exemplars as the shared prefix. Baselines: plain recomputation, a reimplementation of AttentionStore since it isn't open-sourced, and two AttentionStore-plus-H2O variants — one LRU-cached, one LFU-cached. Headline: up to 2.8x lower TTFT than the best baseline, from a 3.8x cut in I/O time, accuracy down under 1%, often closer to 0.2%. Tail latency improves sharply too — p99 on RTE with OPT-30B drops from 9.3 seconds to 2.95. The three techniques' contributions shift by workload, sometimes split 60/30/10, sometimes 36/8/56 — and score-based caching alone lifts GPU hit ratio from 68% to 80%, holding up across chunk sizes, dataset sizes up to 400 gigs, and both model families.

22
00:10:50,325 --> 00:11:10,325
[Dr. Ada Shannon] ...and Llama2-13B added afterward just for a TTFT sensitivity check, not the core similarity claim. One node, one GPU, one drive — that's the whole rig. On that rig the stack delivers exactly what we just walked through. But Hal, "delivers on that rig" is where I want to get harder on this paper — there's a real gap between what IMPRESS is sold as and what it's actually been shown to do.

23
00:11:10,325 --> 00:11:38,000
[Hal Turing] Let's get into it. First thing that bugs me: every eval here — PIQA, RTE, COPA, OpenBookQA — is short, few-shot, multiple-choice, graded on picking the right letter. And they admit there's no open-source real-world prefix-reuse dataset, so they fake one by stapling few-shot examples in front of each query. That's two shortcuts stacked together, and both happen to flatter the system relative to the RAG and chatbot scenarios the intro actually opens with.

24
00:11:38,000 --> 00:12:18,850
[Dr. Ada Shannon] They compound. Multiple-choice accuracy just needs the model to rank four candidates — it tolerates dropped or reordered KVs fine, since the signal for picking a letter is redundant. Open-ended generation and exact-fact retrieval aren't as forgiving: drop the token holding the one date or name, and the answer's wrong. IMPRESS never reports a generation metric — no F1, no factuality score. Stack on the synthetic-prefix problem: a few-shot block is fixed and repetitive by design; a RAG-retrieved passage isn't. RAGCache, from Chao Jin and colleagues, 2024, was built to cache exactly those real retrieved documents. IMPRESS calls it "orthogonal" and never runs the comparison.

25
00:12:18,850 --> 00:12:41,676
[Hal Turing] And a systems concern on top — everything ran on a single A100 and a single SSD, basically one request at a time. But the intro's own motivation is heavy request volumes and preemptive scheduling. What happens to that score-based cache when a dozen concurrent requests are fighting over the same disk and the same GPU slots?

26
00:12:41,676 --> 00:12:58,201
[Dr. Ada Shannon] Not addressed — there's exactly one p99 tail-latency number in the whole paper, single-stream. Compare that to Mooncake, the KV-cache-centric architecture behind Moonshot AI's Kimi in production, from Ruoyu Qin and team, 2024 —

27
00:12:58,201 --> 00:13:04,526
[Hal Turing] Hold on — that's a system actually serving real concurrent traffic, not a lab benchmark?

28
00:13:04,526 --> 00:13:39,551
[Dr. Ada Shannon] Exactly — built because single-node numbers don't survive real multi-tenant load. IMPRESS gives no evidence its score-based eviction holds under contention, so I'd call 2.8x a best-case, uncontended number. Same problem shows up in their own baseline choice: AttentionStore, from Bin Gao and colleagues, 2024, isn't open-source, so they reimplemented it themselves and explicitly disabled its scheduler-aware prefetch prediction to make it "more general." That prediction is a core part of AttentionStore's design. Strip it out, and the 2.8x is measured against a weakened version of the strongest disk-tiered baseline available.

29
00:13:39,551 --> 00:13:48,001
[Hal Turing] And the importance signal underneath all of this, the column-sum attention score — that's not new either. That's H2O's idea.

30
00:13:48,001 --> 00:14:31,676
[Dr. Ada Shannon] Right, from Zhenyu Zhang and colleagues, H2O, NeurIPS 2023, originally used for decode-time eviction. IMPRESS's real contribution is repurposing that oracle for load-time prefill selection, not inventing a new signal. That assumption runs into trouble elsewhere too: retrieval-head research. Wenhao Wu and colleagues, 2024, showed only a small, specific subset of heads actually do long-range factual retrieval — not interchangeable with the rest of the layer. IMPRESS assumes any three heads represent a whole layer, and short multiple-choice questions never stress long-context recall enough to expose that. Every model here is also OPT, dense 2022-style attention. Modern models use grouped-query attention, where KV heads are already shared by construction — nobody's shown this even applies there.

31
00:14:31,676 --> 00:14:36,976
[Hal Turing] So practically — what's safe to ship today, and where should someone be nervous?

32
00:14:36,976 --> 00:14:58,701
[Dr. Ada Shannon] Safe: short, classification-style, cost-sensitive serving — exactly what they tested. Get nervous the moment it's RAG over real documents, long-document QA, or a chatbot where one wrong fact matters, because none of that's been measured. Before deploying there, I'd want F1 or factuality numbers against real RAG and multi-turn traces, and a run under genuine concurrent, multi-GPU load instead of one A100 talking to one SSD.

33
00:14:58,701 --> 00:15:20,751
[Hal Turing] Fair place to land. The engineering — the probe-head shortcut, the reordering, the score-based cache — is genuinely clever, and the head-similarity insight is real. Just don't let that 2.8x headline travel further than the short, multiple-choice, single-GPU world it was actually measured in. Thanks for listening, everyone — we'll catch you next time.