1
00:00:01,000 --> 00:00:08,569
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:08,569 --> 00:00:51,480
[Hal Turing] Today we're looking at "Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving." It's from Junhong Wu, Zedong Liu and colleagues, with Dingwen Tao as senior author. They're at the Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences. It's arXiv 2611.04382, November 2026. The system is called FastKVComp. The question is whether KV compression still pays off on 100 to 400 gigabit networks, and whether the headline throughput numbers describe what the whole system does.

3
00:00:51,480 --> 00:00:59,467
[Dr. Ada Shannon] The abstract claims compression throughput over 500 gigabytes per second. Hold onto that number. We'll spend the episode checking what it measures.

4
00:00:59,467 --> 00:01:01,975
[Hal Turing] First, why is the KV cache even on a network?

5
00:01:01,975 --> 00:01:33,786
[Dr. Ada Shannon] In disaggregated serving, one GPU does prefill and another does decode. Prefill reads the prompt and builds the KV cache, the stored keys and values for every earlier token. Decode reads that cache to generate tokens. If they sit on different GPUs or nodes, the cache has to cross a link. A long-context request can produce around a gigabyte of KV state. Remote KV offload has the same shape. So you either ship the cache as is, or you shrink it, ship it and rebuild it.

6
00:01:33,786 --> 00:01:36,201
[Hal Turing] And shrinking costs GPU time on both ends.

7
00:01:36,201 --> 00:01:53,291
[Dr. Ada Shannon] Right, and that's the lens for this paper. Their Equation 1 says compression pays only if encode time plus decode time is smaller than the wire time you saved. As networks get faster, the wire time saved gets smaller, so the codec has to get faster too. Ratio alone doesn't decide it.

8
00:01:53,291 --> 00:01:55,149
[Hal Turing] What does the compression pipeline do?

9
00:01:55,149 --> 00:02:23,199
[Dr. Ada Shannon] It builds on KVServe, by Liu, Ma, Luo and others, 2026, arXiv 2605.13734. KVServe picks a compression policy per request and runs the pipeline. FastKVComp keeps those policies and changes how the pipeline executes. Note the author overlap: Liu, Ma, Tan and Tao appear on both papers. So the baseline here is the same group's earlier system, not an independent one.

10
00:02:23,199 --> 00:02:48,183
[Dr. Ada Shannon] The pipeline has three steps. First, a randomized Hadamard transform reshapes the value distribution so outliers are spread out. Second, low-bit quantization turns values into small integer codes. Keys are scaled per channel, values per token, and more important heads get more bits. Third, a lossless codec squeezes those integer codes. The receiver reverses all three.

11
00:02:48,183 --> 00:02:50,087
[Hal Turing] So which step loses information?

12
00:02:50,087 --> 00:03:07,641
[Dr. Ada Shannon] Only quantization. The transform is invertible and the codec is lossless. That means output quality rests entirely on KVServe's quantization policy. This paper reports no accuracy or quality numbers. It's purely about speed.

13
00:03:07,641 --> 00:03:09,406
[Hal Turing] Then where does the speed come from?

14
00:03:09,406 --> 00:03:32,487
[Dr. Ada Shannon] Two places. One is kernel fusion. Unfused, the transform kernel writes a full-size intermediate tensor to GPU memory, the HBM. Then the quantize kernel reads it straight back. Fusion keeps each tile on chip, so that write, that read, the temporary buffer and extra kernel launches disappear. It saves memory traffic and launch overhead. It doesn't change the math.

15
00:03:32,487 --> 00:04:05,738
[Dr. Ada Shannon] The other is the lossless codec. The common choice is NVIDIA's nvCOMP ANS, a general-purpose GPU library that ships as a closed binary. It compresses harder but runs slower. The authors swap in LC, from Azami, Fallin and Burtscher, ASPLOS 2025. LC isn't one compressor. It's a set of small, parallel, invertible GPU building blocks, and you search offline for a pipeline that fits your data. It was designed for scientific floating-point data, not quantized KV codes.

16
00:04:05,738 --> 00:04:08,524
[Hal Turing] So the trade is compression ratio against codec time.

17
00:04:08,524 --> 00:04:18,555
[Dr. Ada Shannon] Yes. A lower ratio means more bytes on the wire, but a faster codec costs less time. Equation 1 says which side wins at a given bandwidth.

18
00:04:18,555 --> 00:04:20,227
[Hal Turing] What are the headline claims, then?

19
00:04:20,227 --> 00:04:32,952
[Dr. Ada Shannon] The paper claims a fused kernel up to 5.9 times faster, lossless throughput over 500 gigabytes per second, and up to 2.04 times better job completion time. We'll check what each of those measures.

20
00:04:32,952 --> 00:04:38,896
[Hal Turing] One thing to flag now. What does "gigabytes per second" mean here? Compressed bytes out, or raw bytes in?

21
00:04:38,896 --> 00:05:07,874
[Dr. Ada Shannon] The paper never defines it. But in Table 1, throughput times latency comes to about 1.07 gigabytes in every row. For example, 139 gigabytes per second times 7.7 milliseconds is about 1.07. So it looks like throughput is measured on raw, uncompressed input, for a payload of roughly a gigabyte. That's our inference from the table, not a stated definition. The paper also doesn't say whether scratch allocation and staging are counted.

22
00:05:07,874 --> 00:05:14,051
[Hal Turing] Fine. I'll hold the 500 figure loosely until we see which GPU and which stage it belongs to.

23
00:05:14,051 --> 00:05:18,138
[Dr. Ada Shannon] Good policy. Next we'll look at how the kernel and the codec actually work.

24
00:05:18,138 --> 00:05:22,875
[Hal Turing] Let's start with the kernel. What does the fused kernel do that the two separate kernels didn't?

25
00:05:22,875 --> 00:06:05,042
[Dr. Ada Shannon] It's one tiled kernel, written in TileLang, a tile-based GPU programming framework. It loads a tile of keys or values once. On chip, it applies the randomized Hadamard transform, computes the min-max statistics, and converts to low-bit codes. Then it writes the codes straight into a preallocated buffer that covers all layers. The signs, precision assignments and workspaces are cached across requests. So the full-size intermediate tensor never exists, and the per-layer splitting, stacking and layout conversion mostly go away. The receiver gets the mirror image: dequantize, inverse Hadamard, normalize, restore signs, in one pass. That part doesn't depend on which lossless codec you pick.

26
00:06:05,042 --> 00:06:07,085
[Hal Turing] And the result is the same bits as before?

27
00:06:07,085 --> 00:06:35,785
[Dr. Ada Shannon] The paper says it keeps KVServe's policy and the min-max rule. It reports no accuracy numbers, so that is their claim rather than something we can check. The speedup is large, though. The transform-and-quantize step goes from roughly 27 to 34 milliseconds down to about 6 to 7. That's four to nearly six times faster. And the limit is clear. Fusion removes memory traffic, allocation and launch overhead. It doesn't make the Hadamard arithmetic cheaper.

28
00:06:35,785 --> 00:06:39,407
[Hal Turing] Okay, now the lossless side. Why swap out nvCOMP at all?

29
00:06:39,407 --> 00:07:07,922
[Dr. Ada Shannon] nvCOMP ANS compresses harder, but it's slow, and it's a closed binary with a fixed interface. LC lets you assemble a codec from small GPU primitives. The authors search two- and three-stage combinations offline on quantized KV buffers. They measure encode time, decode time and size, keep the Pareto-efficient candidates, and rank them for the high-bandwidth setting. The winner is TUPL8_1, then BIT_8, then RZE_2.

30
00:07:07,922 --> 00:07:09,222
[Hal Turing] Decode that for me.

31
00:07:09,222 --> 00:07:34,067
[Dr. Ada Shannon] The first stage regroups the bytes by position within each tuple. The second shuffles bits so the same bit position from neighboring values sits together. Low-bit codes waste most of each byte, so after that shuffle you get long runs of zeros and repeated patterns. The third stage, zero elimination, strips those out. The first two stages don't shrink anything. They rearrange the data so the last stage can.

32
00:07:34,067 --> 00:07:36,807
[Hal Turing] Is the integration something they had to engineer too?

33
00:07:36,807 --> 00:07:53,944
[Dr. Ada Shannon] Yes. LC normally runs through a command-line, file-based flow. They built an in-process runtime that takes device pointers, keeps persistent scratch space, and returns only the compressed size to the host. The payload never leaves the GPU. A good share of the speedup is that plumbing.

34
00:07:53,944 --> 00:07:56,126
[Hal Turing] So back to the 500 gigabytes per second.

35
00:07:56,126 --> 00:08:30,817
[Dr. Ada Shannon] Here's what it measures. On an RTX 5090, the LC codec alone runs at about 517 to 524 GB/s. nvCOMP ANS is around 70 to 86. Codec latency is under half a millisecond to about half a millisecond, against about three. That's real. But that GPU isn't in the two-node deployment. There, the full path on the RTX PRO 6000 compresses at about 139 GB/s. The H100 decodes at about 103. So the end-to-end figure is roughly a quarter of the headline, or a fifth on the decode side.

36
00:08:30,817 --> 00:08:33,325
[Hal Turing] Do those figures count bytes the same way as Table 1?

37
00:08:33,325 --> 00:08:53,619
[Dr. Ada Shannon] We can't tell. Table 1 suggests raw input bytes, but whether scratch allocation and staging sit inside the timing is unstated. And the codec-only numbers imply a payload of roughly a quarter gigabyte, against about a gigabyte in Table 1. Either the requests differ or the definitions do, and the text doesn't say which.

38
00:08:53,619 --> 00:08:55,616
[Hal Turing] Is the codec still the bottleneck, then?

39
00:08:55,616 --> 00:09:08,665
[Dr. Ada Shannon] No. After fusion, the codec is only 6 to 8 percent of the encode path with LC. So LC did its job: the codec stopped being the problem. The six or seven milliseconds that remain are mostly outside the codec.

40
00:09:08,665 --> 00:09:12,241
[Hal Turing] Then the end-to-end numbers should show fusion doing most of the work.

41
00:09:12,241 --> 00:09:44,378
[Dr. Ada Shannon] They do. Over the host-staged link, two RTX 5090s with up to 306 Gbps measured, FastKVComp cuts job completion time by 45 to 51 percent versus raw transfer. That's about half the wait. Output throughput rises 1.83 to 2.04 times. The ablation then splits the credit. Fusion alone gives 1.87 to 2.01 times over original KVServe. LC alone stays within 2.1 percent of original KVServe. And fusion alone versus the full system differs by at most 1.3 percent.

42
00:09:44,378 --> 00:09:47,443
[Hal Turing] Why does a codec that is six times faster not move the result?

43
00:09:47,443 --> 00:10:02,489
[Dr. Ada Shannon] Because it compresses less. The ratio falls from about 9x with nvCOMP to about 6.3x with LC, so there are roughly 40 percent more bytes on the wire. That extra transfer time cancels the time saved at this bandwidth.

44
00:10:02,489 --> 00:10:04,765
[Hal Turing] And on the real 200 gigabit InfiniBand link?

45
00:10:04,765 --> 00:10:18,093
[Dr. Ada Shannon] The gain is smaller. JCT drops 30 to 39 percent, and throughput rises 44 to 63 percent. There is no ablation on that setup, so we can't say how fusion and LC split the credit there.

46
00:10:18,093 --> 00:10:19,997
[Hal Turing] What should I keep in mind about the setup?

47
00:10:19,997 --> 00:10:38,759
[Dr. Ada Shannon] It's small. Llama-3.1-8B and Qwen2.5-7B, three LongBench subsets, 50 requests, inputs up to 20K tokens. Generation is 8 tokens locally and up to 64 remotely. That makes transfer a big slice of job time. And there's no variance or repeat count reported.

48
00:10:38,759 --> 00:10:49,394
[Hal Turing] Let's start with the number you told me to hold onto. Why does the abstract say over 500 gigabytes per second if the full path on the real GPUs is far below that?

49
00:10:49,394 --> 00:11:12,010
[Dr. Ada Shannon] Because the 500 is true for exactly one stage on one GPU: the lossless codec alone, on an RTX 5090 that isn't in the two-node deployment. The abstract and conclusion say "compression throughput" without that scope. Each number is real for its own scope, but the paper never reconciles them, and a reader who remembers 500 will overestimate what the system delivers.

50
00:11:12,010 --> 00:11:15,586
[Hal Turing] And the missing throughput definition and the mismatched payload sizes?

51
00:11:15,586 --> 00:11:29,982
[Dr. Ada Shannon] They mean you can't line the two figures up. I'd rather leave that as unknown than guess. What we can say is narrower: LC stops the codec from being the bottleneck. That's a fair claim. "The system compresses at 500" is not.

52
00:11:29,982 --> 00:11:34,533
[Hal Turing] Now the paper's own break-even test. Does it pass? Nobody seems to plug the numbers in.

53
00:11:34,533 --> 00:12:07,134
[Dr. Ada Shannon] This is our calculation, not the paper's. It uses the roughly one gigabyte payload we inferred, and it assumes the full 25 gigabytes per second of a 200 gigabit link is usable. The paper doesn't report effective bandwidth. With those assumptions, raw transfer is about 43 milliseconds. Compressed, the wire time is about 7 milliseconds. Encode on the prefill GPU plus decode on the H100 adds about 18. That's about 25 against 43, so it pays, with real margin.

54
00:12:07,134 --> 00:12:08,574
[Hal Turing] And at twice the link speed?

55
00:12:08,574 --> 00:12:28,821
[Dr. Ada Shannon] Wire savings halve while codec time stays put. Break-even lands near 400 gigabit, so the margin is roughly gone. The old nvCOMP path fails the test even at 200: its codec cost is about 71 milliseconds against roughly 38 saved. That matches the Fig. 3 story.

56
00:12:28,821 --> 00:12:30,261
[Hal Turing] What would settle it properly?

57
00:12:30,261 --> 00:12:43,775
[Dr. Ada Shannon] A per-request timeline of encode, wire, decode and reconstruct. If the system streams layer by layer, compression partly hides behind transfer, and the break-even moves in its favor. The paper gives us no way to check that.

58
00:12:43,775 --> 00:12:51,530
[Hal Turing] Then the ablation. Fusion drives the gain, and LC adds at most 1.3 percent. Why does the title spotlight the codec?

59
00:12:51,530 --> 00:13:10,617
[Dr. Ada Shannon] I think the framing outran the evidence. At the tested bandwidths, LC's speed and its lower ratio cancel out. The best backend likely depends on bandwidth, but they tested two regimes, so the claim that LC wins on faster links is untested. A sweep of link speed would show the crossover.

60
00:13:10,617 --> 00:13:12,196
[Hal Turing] What about the comparison set?

61
00:13:12,196 --> 00:13:53,063
[Dr. Ada Shannon] Three gaps. First, SplitZip, by Guo and Joshi, 2026, a GPU-friendly lossless KV codec. It's cited but never benchmarked, and it's the most direct competitor to LC. Second, KVServe is the same group's earlier system. Fusion alone gives nearly two times over it, so much of that baseline's cost was fixable engineering, not a limit. Third, LC was designed for scientific floating-point data. KV codes are already quantized integers, and the offline search used a number of requests the paper doesn't state. We don't know if the chosen pipeline holds across models, bit widths or distributions.

62
00:13:53,063 --> 00:13:53,853
[Hal Turing] And quality?

63
00:13:53,853 --> 00:14:04,998
[Dr. Ada Shannon] Not evaluated. The lossless stage is safe by construction, but the fused kernel has to match KVServe's quantization, and there's no accuracy table. We also don't see long generations.

64
00:14:04,998 --> 00:14:07,088
[Hal Turing] So who should care, and what should they do?

65
00:14:07,088 --> 00:14:27,522
[Dr. Ada Shannon] Anyone shipping large KV payloads over fast links, where codec time rivals wire time. Before adopting, measure the whole path with your own throughput definition, not codec-only. The transferable lesson is that execution efficiency can matter more than compression ratio. Naive compression can erase its own savings.

66
00:14:27,522 --> 00:14:28,869
[Hal Turing] And what would you want next?

67
00:14:28,869 --> 00:14:40,943
[Dr. Ada Shannon] SplitZip head to head, per-request breakdowns with overlap, 400 gigabit and above, longer generation, more models and bit widths, and the LC search run on the real deployment distribution.

68
00:14:40,943 --> 00:15:05,742
[Hal Turing] So here's where I land. The work shows an efficient execution path keeps KV compression worthwhile on fast links, and fusion does most of that. The boundary is that 500 gigabytes per second covers the codec only. The full path runs about 100 to 140, and LC's own end-to-end contribution isn't shown to matter. Thanks for listening, and we'll catch you next time.