1
00:00:01,000 --> 00:00:36,600
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.
Today's paper is DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference — Bodon Jeong et al., seven authors total, out of Sogang University, Florida State University, and Samsung Electronics Co. It hit arXiv on April 29th, 2026. Ada, this one's pure systems engineering — no new architecture, just a very deep look at what happens when you try to push a KV-cache onto an SSD.

2
00:00:36,600 --> 00:01:17,775
[Dr. Ada Shannon] And that's exactly why it grabbed me. Most KV-cache offloading work reads like trial-and-error engineering. This team actually diagnosed, layer by layer, why file-backed offloading breaks down before writing a line of the real system — the kernel page cache, the block layer, the SSD's own queueing, all treated as variables in the problem rather than background plumbing. That level of upfront diagnosis is what actually won me over — solid theoretical grounding before a single line of implementation. And the setting matters too: this isn't a cloud cluster where you throw another GPU at a growing KV-cache. It's a single edge box. Once that cache outgrows memory, an SSD is the only place left to go.

3
00:01:17,775 --> 00:02:01,300
[Hal Turing] Right — and you genuinely can't just add DRAM at runtime the way a cloud operator adds a node. We're talking single-GPU boxes, often unified memory where CPU and GPU share one 8-to-32-gigabyte pool — a Jetson Orin, or your own workstation card. Load a 7 or 8 billion parameter model and that budget's mostly spoken for already. What keeps growing on top is the KV-cache, session after session. So the paper's actual question is: how do you offload that overflow to NVMe without hitting the exact traps that make naive file-based offloading fall apart under real memory pressure — cache thrashing, kernel overhead, and broken sequential access?

4
00:02:01,300 --> 00:02:40,950
[Dr. Ada Shannon] Let's ground the cache itself. Every transformer layer, for every token, produces a key and value vector. Instead of recomputing those on every generation step, the model caches them — that's the KV-cache — so decoding doesn't mean reprocessing the whole sequence from token one each time. It grows linearly with context length and batch size, and at long contexts it can dwarf the model weights. It also stresses storage differently by phase: prefill processes the whole prompt at once, so you get a burst of large, write-heavy KV writes. Decode is the opposite — one token at a time, but every step rereads the entire accumulated cache so far. That's read-heavy, and relentless.

5
00:02:40,950 --> 00:03:25,500
[Hal Turing] Once that cache outgrows GPU and host memory, the default move — the one FlexLLMGen takes, the baseline this paper builds on — is to memory-map a file on disk and let the OS page cache tier it automatically. FlexLLMGen descends directly from FlexGen, the single-GPU offloading system Ying Sheng and colleagues published out of Stanford back in 2023. The kernel decides what stays in DRAM and what gets evicted using its own general LRU-style policy. The problem is that policy has no idea it's serving an autoregressive KV-cache. It's tuned for generic file access, not for reread-this-exact-growing-region-on-every-decode-step.

6
00:03:25,500 --> 00:04:01,125
[Dr. Ada Shannon] And that mismatch produces real failures. First, decode-side thrashing — because decode keeps rereading the accumulated cache, and the page cache doesn't know that region is precious, the pages you need again next step get evicted anyway. Your hit ratio collapses and you're hitting the SSD constantly instead of DRAM. Second, prefill has the opposite problem: that burst of writes pressures the page cache into synchronous write-back to free space, which stalls you right when you're trying to process the prompt fast. So you've got thrashing on one side of the phase cycle and write-stalls on the other —

7
00:04:01,125 --> 00:04:27,600
[Hal Turing] Oh wait wait wait — hold on. So even before the SSD itself does anything, the kernel's already working against you twice over — write-side during prefill, read-side during decode? I think people hear "disk offloading" and picture one clean, predictable read. They don't picture a whole eviction and write-back machine underneath, deciding — badly, for this workload — what actually survives in DRAM. That's a pretty different mental model of what offloading even means here.

8
00:04:27,600 --> 00:05:13,475
[Dr. Ada Shannon] Precisely. There's a third bottleneck too — pure software tax. Every I/O request crosses the virtual file system, the filesystem layer, the generic block layer, and the device driver before it reaches the NVMe SSD, and each hop adds latency none of those layers actually needs for this pattern — which is exactly why the usual fix in this space is kernel bypass, issuing I/O straight to the device with mechanisms like io_uring_cmd instead of walking that whole stack. Then there's a sneakier issue: NVMe SSDs are addressed by logical block addresses, LBAs, independent of any filesystem, and Linux spreads requests across multiple hardware queues — the blk-mq layer — for parallelism. That fragments and reorders what was originally a contiguous KV-cache stream by the time it lands at the controller. You lose sequential locality, exactly the access pattern flash is fastest at.

9
00:05:13,475 --> 00:05:38,850
[Hal Turing] So three separate failures, all stacking out of one decision: treating the KV-cache like any other memory-mapped file. Thrashing, kernel-stack tax, and broken sequential locality. That's the board DUAL-BLADE is playing on — an edge box, a fixed memory ceiling, a cache that keeps growing, and a storage stack actively fighting you the whole way down. Next, we get into how they actually route tensors around all of that.

10
00:05:38,850 --> 00:06:18,600
[Dr. Ada Shannon] So here's the fix. At initialization, DUAL-BLADE wraps every K or V tensor, per layer, into what they call a KV Placement Unit — a KPU. A budgeter reads cgroup memory stats, subtracts the pinned DMA buffers each copy-thread reserves for GPU transfers, and gets a usable page-cache budget, B_pc. Algorithm 1 walks the layers in order and assigns KPU pairs to Group 1 — the page-cache path, memory-mapped exactly like FlexLLMGen already did — until that budget's spent. Everything past the cutoff goes to Group 2, the NVMe-direct path. It's not a smarter cache. It's triage — stop lying to yourself about how much DRAM you actually have free.

11
00:06:18,600 --> 00:06:38,350
[Hal Turing] Okay — Group 2 is the kernel-bypass side. What does 'bypass' actually mean here in implementation terms, not just conceptually? Because 'skip the filesystem' is easy to say and historically brutal to build correctly — is this SPDK-style user-space polling, or something lighter than that?

12
00:06:38,350 --> 00:07:37,275
[Dr. Ada Shannon] Lighter — io_uring_cmd. It issues native NVMe commands as passthrough: no VFS, no block-layer scheduling, just a submission and completion queue talking almost directly to the device. In their own comparison it matched SPDK's latency without SPDK's busy-polling and core-pinning cost. The mapping is a hash table — tensor ID to starting LBA and block count — and since Group 2 tensors get bound in access order, each tensor's start LBA is just the previous one's end. Algorithm 2 takes a tensor's shape and offset, computes that anchor LBA, then chunks the transfer to the device's max transfer size — 256 kilobytes on SSD A — keeping every chunk a clean multiple of the 4-kibibyte LBA size. That alignment is exactly where OPT-13B gets awkward — its per-token KV tensor is ten kibibytes, not a 4K multiple on its own. Their fix: force an even batch size, so two tensors together land on a clean 4K boundary.

13
00:07:37,275 --> 00:07:56,150
[Hal Turing] Oh wait, hold on — that's a neat trick, but it's also a little fragile, isn't it? Doubling B to satisfy alignment means the fix is quietly shape-dependent, not some fixed device constant. So what happens on a model where doubling the batch still doesn't land the tensor on a clean 4K boundary?

14
00:07:56,150 --> 00:08:37,650
[Dr. Ada Shannon] For OPT-13B specifically it always works — ten kibibytes times two is twenty, exactly five 4K blocks. It's a parity fix, not a general solver, and I want to come back to how far that generalizes. But layout's only half the story — you still have to keep the GPU fed while that lane runs. DUAL-BLADE picks between two overlap strategies. Overlap-Intra runs both copy-threads' storage reads in parallel, maximizing raw bandwidth when the SSD isn't saturated. Overlap-Cross staggers the second thread's start so its storage read overlaps the first thread's GPU DMA instead — different hardware, no contention. Each decode phase trials one iteration of each, compares throughput, and locks in the winner for the rest of generation.

15
00:08:37,650 --> 00:08:55,625
[Hal Turing] Self-tuning per workload instead of hard-coding one strategy for their test rig — that's a nice touch. So does all of this, the routing plus the bypass plus the pipeline choice, actually add up to something you'd notice, or is it a thousand micro-optimizations canceling out?

16
00:08:55,625 --> 00:09:56,525
[Dr. Ada Shannon] It adds up. Across the memory sweep, on SSD A — their PCIe Gen5 drive — DUAL-BLADE cuts prefill latency by up to 33.1% and decode by as much as 42.4%. SSD B, the older Gen4 drive, shows the same pattern. NVMe busy time — actual device utilization — jumps 2.2x on prefill writes, meaning the drive's finally doing work instead of idling between kernel round-trips. The hit-ratio chart tells the story cleanest: Baseline collapses into the thrashing zone, CachePolicy-Only recovers linearly but leans on posix_fadvise to proactively evict, which still costs a memcpy through the cache. DUAL-BLADE recovers further because Group 2 never touches the page-cache at all. And the per-tensor table is almost striking — a 256-kilobyte decode write drops from 4.91 milliseconds to 0.06 on SSD A, with device busy time pinned at 100% the entire time. Same hardware work, ninety-eight percent less software tax.

17
00:09:56,525 --> 00:10:17,800
[Hal Turing] That 4.91-to-0.06 number is exactly the kind of result that makes me want to see the fine print immediately. What model is generating every one of those figures — three through sixteen, every table? If it's one or two checkpoints across the whole evaluation, I want to know how representative that choice really is.

18
00:10:17,800 --> 00:10:58,500
[Dr. Ada Shannon] And that's the crack I'd poke at. Every experiment — all of it — runs on OPT-6.7B or OPT-13B, 2022-era dense multi-head-attention models, no grouped-query attention anywhere in the eval. But their own framing leans on Llama 3, Mistral, and Qwen-class models as what's actually dominant on edge hardware today — and those are all GQA. GQA shrinks the per-token KV tensor, fewer KV heads in that same B-times-H-times-D-times-2 formula from Table II, which means the alignment problem that already forced a special parity fix for OPT-13B's ten-kibibyte tensor gets tighter, not looser, on a GQA model. And they never test it. Not once, on any GQA checkpoint.

19
00:10:58,500 --> 00:11:15,225
[Dr. Ada Shannon] —hardware. And I don't want to just retread what I said a second ago — that gap is real, but it's actually a preview of something bigger. The GQA mismatch is a symptom. The disease is an assumption baked into the whole NVMe-direct design, and once you see it, it's hard to unsee.

20
00:11:15,225 --> 00:11:35,025
[Hal Turing] Don't leave me hanging, Ada — what's the assumption? Everything we walked through last part, the budgeter, the contiguous LBA extents, Algorithm 2 chunking against MDTS, sounded like solid systems engineering. So what's the one thing all of that quietly requires to be true before it even starts?

21
00:11:35,025 --> 00:12:30,325
[Dr. Ada Shannon] It requires the entire KV footprint and its on-disk layout to be knowable before execution begins — that's stated outright as a precondition in the paper. FlexLLMGen fixes batch size and sequence length up front, so DUAL-BLADE can plan one contiguous LBA extent per tensor and materialize it once. Production stacks don't work that way anymore: vLLM, llama.cpp, and LMCache on top of vLLM use PagedAttention-style block allocators — Kwon and colleagues out of UC Berkeley, SOSP 2023 — where requests join and leave dynamically and nobody knows the total footprint in advance. The paper cites PagedAttention exactly once, in related work, as an orthogonal GPU-memory technique, and never engages with the conflict between dynamic block allocation and a design that needs one static contiguous extent per tensor.

22
00:12:30,325 --> 00:12:43,575
[Hal Turing] Wait, hold on — they explicitly say the design is 'framework-agnostic' and could plug into something like LMCache. Did they ever actually try that, or is that just a sentence in related work?

23
00:12:43,575 --> 00:13:24,275
[Dr. Ada Shannon] Just a sentence. Asserted, not demonstrated — no integration, no benchmark against a PagedAttention-based server, nothing. That's from the LMCache paper itself, Liu and colleagues, 2025, out of the University of Chicago's LMCache project, exactly the dynamic backend that claim gestures at. Same gap shows up in the comparisons generally. The only baseline anywhere in this paper is vanilla FlexLLMGen's own mmap path. KVSwap, Zhang, Xia, and Wang, 2025, is the closest actual competitor: a low-rank predictor that fetches only the important KV groups from flash instead of DUAL-BLADE's uniform layer-order split. Cited in related work. Never benchmarked.

24
00:13:24,275 --> 00:13:39,100
[Hal Turing] And that same pattern repeats with InfiniGen, doesn't it? I remember a line admitting prior work ranks KV entries by reuse or importance, and calling their own uniform policy 'pluggable' to something smarter.

25
00:13:39,100 --> 00:14:11,725
[Dr. Ada Shannon] Exactly — Lee, Lee, Seo, and Sim, InfiniGen, OSDI 2024, out of Seoul National University. It ranks KV entries by how much they'll matter for attention instead of treating every layer the same. The paper name-checks it, calls its own scheme pluggable to that kind of ranker, and never implements or tests one. So the honest claim here is narrower than it sounds: the I/O-path engineering, the kernel bypass, the sequential LBA work, genuinely earns its evidence. Whether a smarter, importance-aware placement policy would beat the naive layer-order split, or make the whole path question moot, is simply unverified.

26
00:14:11,725 --> 00:14:23,125
[Hal Turing] Stepping outside the architecture fight for a second — does any of this matter outside a chat-style benchmark? I noticed a data-wrangling evaluation buried in the results.

27
00:14:23,125 --> 00:15:01,425
[Dr. Ada Shannon] It's one of the more grounded parts of the paper. They run entity matching, data imputation, and error detection with OPT-6.7B under a tight 4-gigabyte limit — long inputs, short outputs, the workload an edge box actually runs all day. DUAL-BLADE wins most of them, down to about 0.85 times baseline latency on the Buy dataset. Honest exception: on the Hospital error-detection dataset it's a wash, basically 1.0 times, because that workload's KV-cache is only 1.58 gigabytes, small enough to fit entirely in page-cache already. I like that they left in a case where their own system doesn't help.

28
00:15:01,425 --> 00:15:06,775
[Hal Turing] That's a fair thing to leave in. So where does this need to go next, in your view?

29
00:15:06,775 --> 00:15:47,575
[Dr. Ada Shannon] Four things. Validate the alignment and B-parity scheme against a real GQA model — Llama 3, Mistral, Qwen — where per-token KV units are smaller and that 4-KiB boundary problem gets tighter, not looser. Integrate with a dynamic block allocator, vLLM plus LMCache, and see what survives once the static-footprint assumption is gone. Run the comparisons that are conspicuously missing, KVSwap and InfiniGen, head to head, not just cited. And longer term, compose this with GPU-initiated storage access — BaM, Qureshi and colleagues out of the University of Illinois Urbana-Champaign, ASPLOS 2023 — for the datacenter tier, though the paper itself admits that needs hardware edge boxes don't have.

30
00:15:47,575 --> 00:16:24,175
[Hal Turing] That's a fair place to land. DUAL-BLADE genuinely fixes something real — page-cache thrashing and kernel overhead were measurably killing NVMe offloading, and the dual-path residency plus kernel-bypass design earns its 33 and 42 percent numbers on its own test rig. Where it gets ahead of itself is generalization: untested on GQA models, untested against its closest competitors, resting on a static-footprint assumption production serving is moving away from. Solid systems paper, oversold framing. That's DUAL-BLADE. Thanks for listening, everyone — we'll catch you next time.