Cross-Model KV Cache Transfer for Fast LLM Prefill Reuse

This episode examines a NVIDIA paper on transferring KV cache between different-sized models within the same architecture family — for example Qwen3 14B and 32B — without any gradient training. The hosts explain the core finding: a single layer of a smaller model's cache can explain over half the variance in a larger model's keys, and stacking source layers pushes that correlation even higher. They break down the two practical payoffs — using a small model's cache to bootstrap a larger model mid-conversation for quality upgrades, and the reverse direction, prefilling once on an expensive large model then handing the cache down to a cheap model to skip decode costs entirely. A skeptical exchange probes whether a closed-form ridge-regression mapping can really generalize across the nonlinear depth of transformer layers, with the paper's authors measuring rather than assuming the linear structure holds, and only for "matched-KV pairs" with identical head counts and per-head dimensions. Listeners interested in inference cost reduction, model routing, and cache reuse across model families will find the comparison to trained alternatives like Cache-to-cache and LatentAlign particularly relevant.