1 00:00:01,000 --> 00:00:07,269 [Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon. 2 00:00:07,269 --> 00:00:42,703 [Hal Turing] Today's paper is "KV-Lingo: Learning KV-Cache Translators with Distillation," from Apple. The first authors are Valérie Castin and Keitaro Sakamoto, who did the work as interns, along with colleagues. It went on arXiv in September 2026. The claim is that a small set of linear maps can turn one frozen model's memory of a document into memory a different frozen model can read. The reported speedups are 9.6x on a laptop-class chip and up to 29x at long context. 3 00:00:42,703 --> 00:00:54,823 [Dr. Ada Shannon] Those speedups are the headline, and what they actually measure matters a lot. We'll check that later. The question underneath is this: when you switch models over text that's already been read, can you skip reading it again? 4 00:00:54,823 --> 00:00:56,635 [Hal Turing] Start with why you'd have to read it again. 5 00:00:56,635 --> 00:01:24,220 [Dr. Ada Shannon] The KV cache. While a transformer reads a prompt, every layer computes a key vector and a value vector for every token. Later tokens attend to those stored vectors, so the model never recomputes the past. But the vectors are a product of that model's weights, and their shape follows its layer count, KV heads and head dimension. Hand model B the cache from model A, even when the shapes match, and generation falls apart. 6 00:01:24,220 --> 00:01:26,960 [Hal Turing] So if model B takes over, it rebuilds everything. 7 00:01:26,960 --> 00:01:50,273 [Dr. Ada Shannon] Right. That rebuild is called prefill: one forward pass over the whole prefix. Time to first token, TTFT, is how long the user waits before the first generated word, so lower is better. Prefill cost grows roughly quadratically with prefix length. So a long document makes the re-prefill after a switch expensive, even though another model already read it. 8 00:01:50,273 --> 00:01:53,152 [Hal Turing] When do two models end up sharing one context in practice? 9 00:01:53,152 --> 00:02:30,582 [Dr. Ada Shannon] Routing. FrugalGPT, by Lingjiao Chen, Matei Zaharia and James Zou in 2023, is the canonical cascade. A cheap model answers first, and you escalate to a pricier one if a scorer distrusts the answer. RouteLLM, by Isaac Ong and colleagues in 2024, trains a router to pick the strong or weak model per query. Then there's token-level routing, like R2R from Fu and colleagues in 2025, where a small model generates by default and a larger one steps in for selected tokens. Each of those creates a switch point over a shared prefix. 10 00:02:30,582 --> 00:02:32,533 [Hal Turing] Now the translator. What is it, concretely? 11 00:02:32,533 --> 00:02:51,434 [Dr. Ada Shannon] For each layer of the target model, one linear map for keys and one for values. It takes the source model's cache entries, from one or a few source layers, and produces target-shaped entries. It works token by token, with no mixing across tokens, so the cost is linear in prefix length. Both models stay frozen. 12 00:02:51,434 --> 00:02:56,124 [Hal Turing] Direction matters too, right? The paper talks about small-to-large and large-to-small. 13 00:02:56,124 --> 00:03:19,205 [Dr. Ada Shannon] Yes. Small-to-large, S to L, means the small model prefills and the large model decodes from the translated cache. That's the cascade-escalation case. Large-to-small is the reverse. One use is agent hand-offs, where a strong model sets up context and a cheaper one continues. Another is privacy, where the large model only sees the non-sensitive part of the context. 14 00:03:19,205 --> 00:03:23,478 [Hal Turing] And the Apple laptop number involves a mixture-of-experts model, doesn't it? 15 00:03:23,478 --> 00:03:42,564 [Dr. Ada Shannon] A Qwen MoE. It decodes about as fast as a small dense model, because only a few experts fire per token. But prefill is slow, since a long prompt spreads across many experts. So the model is cheap to talk to and expensive to hand a long context. That's where skipping the prefill looks most attractive. 16 00:03:42,564 --> 00:03:47,858 [Hal Turing] So the headline says the switch costs one linear map plus one decode step, not a prefill. 17 00:03:47,858 --> 00:03:55,707 [Dr. Ada Shannon] Yes, and the thing to watch is whose prefill is missing from that comparison. We'll come back to it. First, how these maps get trained. 18 00:03:55,707 --> 00:03:59,561 [Hal Turing] A linear map per layer sounds almost too simple. How do you fit it? 19 00:03:59,561 --> 00:04:17,673 [Dr. Ada Shannon] Two stages. Stage one is closed-form. For each target layer, solve a least-squares problem: find the matrix that turns the source model's keys into the target's actual keys, and the same for values. No gradient steps. It's cheap, and a few concurrent papers stop there. 20 00:04:17,673 --> 00:04:18,927 [Hal Turing] And that isn't enough? 21 00:04:18,927 --> 00:04:36,853 [Dr. Ada Shannon] That's the paper's central finding. A translator can match the target's cache closely and still generate badly. Matching activations is not the same as producing the right next-token distribution. With the closed-form fit alone, on some benchmarks up to 96 percent of items had no parseable answer at all. 22 00:04:36,853 --> 00:04:38,942 [Hal Turing] So what's stage two optimizing instead? 23 00:04:38,942 --> 00:05:02,487 [Dr. Ada Shannon] Self-distillation. The target model is its own teacher. Run it on its native cache and record its next-token distribution. Run it again on the translated cache. Train the maps to minimize the KL divergence between the two. The translated cache never has to equal the native one entry by entry. It only has to make the target behave the same way. 24 00:05:02,487 --> 00:05:06,806 [Hal Turing] That dodges a lot of wasted precision. Some cache entries barely matter. 25 00:05:06,806 --> 00:05:18,370 [Dr. Ada Shannon] Right. A token nobody attends to can be badly reconstructed at no cost, but a reconstruction loss charges for it anyway. The distillation loss only charges for errors that change the output. 26 00:05:18,370 --> 00:05:22,921 [Hal Turing] Two variants keep coming up: head-wise and head-mixing. What's the difference? 27 00:05:22,921 --> 00:05:46,327 [Dr. Ada Shannon] Head-wise means target head h reads only source head h. That works when the two caches have the same shape, like the Qwen3 pairs. Head-mixing leaves the maps unconstrained, so any target head can read any source head. It's costlier but needed when shapes differ. Cross-layer means the models have different depths, so each target layer reads a small number of source layers. That number is called nu. 28 00:05:46,327 --> 00:05:49,856 [Hal Turing] What about multi-turn? Hand the context back and forth a few times. 29 00:05:49,856 --> 00:06:11,683 [Dr. Ada Shannon] The naive version translates the whole prefix at every switch, including text that was already translated once. It holds for a few turns, then collapses. Their fix is alternation. Each model keeps its own cache and translates only the tokens generated since the last switch. On QuAC that stays within 0.05 of full re-prefill. 30 00:06:11,683 --> 00:06:14,748 [Hal Turing] Now the headline numbers. What exactly is being compared? 31 00:06:14,748 --> 00:06:38,061 [Dr. Ada Shannon] On an H100 with vLLM, the numerator is the target's own time to first token after a full re-prefill. The denominator is translation plus one decode step. At 2k tokens that's about five times faster. At 32k it's 12 to 29 times, depending on the target, and larger targets gain more. The gap widens because prefill grows quadratically and translation grows linearly. 32 00:06:38,061 --> 00:06:39,500 [Hal Turing] Give me one that sticks. 33 00:06:39,500 --> 00:06:58,076 [Dr. Ada Shannon] Qwen3-8B at 32k tokens: about 1.4 seconds of waiting becomes under 50 milliseconds. Two caveats ride along. The source model's prefill is not in the denominator. And the translation is timed as plain PyTorch matrix multiplies outside vLLM. 34 00:06:58,076 --> 00:06:59,330 [Hal Turing] Fine. Does quality hold up? 35 00:06:59,330 --> 00:07:19,810 [Dr. Ada Shannon] Partly, with a ceiling. Take the 0.6B to 8B pair, averaged over nine benchmarks scored from 0 to 1, reasoning off. The 8B reading the translated cache of the 0.6B beats the 0.6B alone by about seven to nine points. It doesn't reach what the 8B gets from its own cache. And the error bars are over evaluation items only. 36 00:07:19,810 --> 00:07:21,111 [Hal Turing] And large to small? 37 00:07:21,111 --> 00:07:32,256 [Dr. Ada Shannon] Slightly below the small model's own score. On the shape-preserving Qwen3 pairs, the linear maps roughly match a nonlinear MLP translator, which is the justification for keeping them linear. 38 00:07:32,256 --> 00:07:34,067 [Hal Turing] Does any of this travel beyond Qwen? 39 00:07:34,067 --> 00:07:45,491 [Dr. Ada Shannon] A bit. They try Gemma pairs, a pair with different tokenizers, and MLA caches. Those work to varying degrees, so it's evidence of range rather than a settled result. 40 00:07:45,491 --> 00:07:48,046 [Hal Turing] So whose prefill is missing from the 9.6x? 41 00:07:48,046 --> 00:08:14,377 [Dr. Ada Shannon] The small model's. On the Apple laptop, the big mixture-of-experts model takes 152 milliseconds to prefill a 64-token prompt. The translated path is 1.8 milliseconds of translation plus a 14-millisecond decode step, so 15.8 milliseconds. That's the 9.6x. But the 15.8 does not include the 68 milliseconds the 4B model spent building the cache it's translating. 42 00:08:14,377 --> 00:08:15,352 [Hal Turing] So add it back. 43 00:08:15,352 --> 00:08:33,975 [Dr. Ada Shannon] 68 plus about 16 is roughly 84 milliseconds, against 152. The wait drops by a bit under half, about 1.8x. The paper's own appendix says as much. At 1024 tokens the 4B prefills about as slowly as the MoE, so there's no gain at all. 44 00:08:33,975 --> 00:08:37,876 [Hal Turing] And the H100 numbers? Twelve to twenty-nine times at 32k. 45 00:08:37,876 --> 00:09:08,340 [Dr. Ada Shannon] Same structure. The source prefill isn't in the denominator, and the paper reports no end-to-end H100 number. As a rough estimate of mine, not a paper result: at 32k the 4B takes about 1065 milliseconds in vLLM and the 8B about 1384. Add roughly 48 milliseconds for translation and a step, and you get about 1113 against 1384. That's around 1.2x, not 29x. 46 00:09:08,340 --> 00:09:09,733 [Hal Turing] So when is the saving real? 47 00:09:09,733 --> 00:09:40,988 [Dr. Ada Shannon] When the small model would have prefilled anyway. A cascade like FrugalGPT fits. The small model answers first, a verifier rejects it, and you escalate. The prefix is already cached, and that's exactly when a re-prefill would have happened. A query-level router like RouteLLM decides before generation, so the small model might never prefill. Then translation adds a prefill you wouldn't otherwise run. Token-level routing in R2R switches constantly, so the multi-switch behavior matters more there. 48 00:09:40,988 --> 00:09:43,077 [Hal Turing] What escalation rate makes it pay off? 49 00:09:43,077 --> 00:09:55,755 [Dr. Ada Shannon] Nobody knows. The paper builds no router and measures no escalation rate, discarded answers, or router overhead. To settle it you'd run the whole system on one serving stack and count the source prefill. 50 00:09:55,755 --> 00:10:01,282 [Hal Turing] Then the quality side. Does the big model reading a translated cache behave like the big model? 51 00:10:01,282 --> 00:10:23,294 [Dr. Ada Shannon] Mostly no. Translated S→L lands near the small model, not the large one. On Gemma it only reaches the small model's level, within one standard error. The dense-to-MoE pair fails on long context, mostly on the RULER retrieval tests. So if escalation buys you roughly the small model's quality, the reason to escalate gets thin. 52 00:10:23,294 --> 00:10:24,455 [Hal Turing] Is there a defence? 53 00:10:24,455 --> 00:10:43,171 [Dr. Ada Shannon] A partial one. On Qwen3 0.6B to 8B it clearly beats the small model. But the evidence is one training corpus of chat data, one translator per ordered pair, and error bars over evaluation items only. And the strongest results come from Qwen3, where heads line up one to one. 54 00:10:43,171 --> 00:10:44,703 [Hal Turing] What else does the paper skip? 55 00:10:44,703 --> 00:11:05,973 [Dr. Ada Shannon] The setting is batch size 1, memory-bound. Under continuous batching the picture could change, and nothing here tests it. On-device, the cache has to move between models, and that communication cost isn't evaluated. Reading several source layers costs extra. And cross-tokenizer and cross-architecture pairs add constraints on top. 56 00:11:05,973 --> 00:11:09,734 [Hal Turing] The privacy pitch also assumes the translated cache leaks nothing. 57 00:11:09,734 --> 00:11:20,322 [Dr. Ada Shannon] Right. The paper states that as a motivation but doesn't test it. The method is a real contribution, but the claim that this is a general routing primitive runs ahead of the evidence. 58 00:11:20,322 --> 00:11:23,852 [Hal Turing] So if I'm an engineer, where does a cache translator earn its place? 59 00:11:23,852 --> 00:11:42,892 [Dr. Ada Shannon] Where the small model's prefill is work you were doing anyway. A cascade where the small model answers first and only escalates on failure is the clean case. The cache already exists, and translating it replaces a full prefill by the big model. Long contexts help too, since re-prefill keeps getting more expensive while translation stays linear. 60 00:11:42,892 --> 00:11:44,193 [Hal Turing] And the on-device story? 61 00:11:44,193 --> 00:12:00,586 [Dr. Ada Shannon] A mixture-of-experts target at batch size one is the best fit in the paper. It decodes cheaply but prefills slowly, so a small dense model can do the slow part. L→S hand-off is the other use, passing a long working context from a strong model to a cheaper one. 62 00:12:00,586 --> 00:12:02,118 [Hal Turing] And when is it the wrong tool? 63 00:12:02,118 --> 00:12:16,236 [Dr. Ada Shannon] When the small model prefills only to feed the large one. Then you pay for two passes where you could have paid for one, and the saving can vanish. Short prompts on a fast serving stack are also poor candidates, because there's little prefill to avoid. 64 00:12:16,236 --> 00:12:19,115 [Hal Turing] So what's the follow-up measurement that would convince you? 65 00:12:19,115 --> 00:12:35,648 [Dr. Ada Shannon] End-to-end time to first token inside a real router, with the source prefill counted. Sweep the escalation rate, and run it all on the same serving stack, translation included. Then you can read off where translation pays. I won't guess that number, because the paper doesn't give one. 66 00:12:35,648 --> 00:12:37,738 [Hal Turing] Does the paper point anywhere beyond that? 67 00:12:37,738 --> 00:13:03,744 [Dr. Ada Shannon] A few directions. One is training models jointly so their caches are compatible from the start, instead of bolting a translator onto two frozen models. Another is training the translators on continuations the target model generated itself, rather than one generic chat corpus. The appendix also sketches memory-budget decoding, where you decode without ever materialising the full target cache. That one is a proof of concept with random weights. 68 00:13:03,744 --> 00:13:05,788 [Hal Turing] Random weights. So it's a plumbing test. 69 00:13:05,788 --> 00:13:12,243 [Dr. Ada Shannon] About as much as it can be. It shows the idea can run, not that it works well, so it shouldn't carry weight in a decision. 70 00:13:12,243 --> 00:13:24,085 [Hal Turing] Let me try the takeaway. A linear map, trained by distillation against the target's own predictions, can make one frozen model's cache readable by another, at far lower cost than a prefill. 71 00:13:24,085 --> 00:13:35,973 [Dr. Ada Shannon] That's what's established. The boundary is the part to remember. The big speedups leave out the small model's prefill, and a big model reading a translated cache mostly performs like the small model, not the large one. 72 00:13:35,973 --> 00:13:44,008 [Hal Turing] So the method is sound and cheap. Whether it saves you time depends on your deployment, and the paper hasn't measured that yet. 73 00:13:44,008 --> 00:13:44,704 [Dr. Ada Shannon] Right. 74 00:13:44,704 --> 00:13:48,419 [Hal Turing] That's our time. Thanks to everyone for listening, and thank you, Ada. 75 00:13:48,419 --> 00:13:50,602 [Dr. Ada Shannon] Thanks, Hal. And thanks to the listeners.