FastKVComp: Checking Whether High-Throughput KV Compression Pays Off

Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving — Junhong Wu, Zedong Liu, Xinyang Ma, Yuhan Chen, Guangming Tan, Dingwen Tao · 2026
Episode companion · hosts Hal Turing & Dr. Ada Shannon
arXiv 2611.04382 Interactive page

Where compression sits

Prefill GPU builds the KV cache, compresses it, sends it over a 100–400 Gb link; the decode GPU reverses every step. Click a box or use Next.

only quantization loses information

Equation 1: does it pay?

Compression pays only if encode + decode time < wire time saved. Drag the link speed.

Episode's back-of-envelope calculation, not paper data ~1.07 GB payload (inferred) · assumes full link usable · LC ratio 6.3× · nvCOMP 9×
—LC path total vs raw
—nvCOMP path total vs raw
~400 GbLC break-even (computed)
~107 GbnvCOMP break-even (computed)
LC codec cost assumed ≈18 ms (encode + decode); nvCOMP ≈71 ms. Streaming layer by layer could hide part of this; the paper gives no per-request timeline.

Kernel fusion: no full-size intermediate

Toggle between the two designs. Fusion changes memory traffic and launches, not the math.

Schematic

Transform + quantize time

Reported: ~27–34 ms down to ~6–7 ms (about 4× to nearly 6×; paper headline up to 5.9×).

Why a Hadamard transform helps

Randomized signs + Hadamard spread an outlier channel across the row, so min-max quantization wastes fewer levels.

Toy 8×8 data, computed in-page
Hover a cell.

Inside an LC-style pipeline

Chosen stages: TUPL8_1 → BIT_8 → RZE_2. The first two only rearrange; the last removes zeros.

Toy 8 bytes of 4-bit codes · exact stage internals simplified

nvCOMP ANS vs LC (RTX 5090, codec alone)

~40%more bytes on wire with LC (9× → 6.3×)
6–8%codec share of encode path after fusion

What does "500 GB/s" measure?

Table 1 sanity check

139 GB/scompress × 7.7 mslatency ≈ ~1.07 GBsame in every row Inference: throughput is likely raw input bytes. Not stated by the paper.

Audit checklist

Tap a chip.

Pick a question above.

References