KV-Lingo: Translating KV Caches Between Frozen Models
Oct 1, 2026
This episode examines a paper on training small linear "translators" that convert one frozen model's KV cache into one a different frozen model can read. The aim is to skip the expensive re-prefill when a request switches models, as in cascade routing, token-level routing, agent hand-offs, and privacy-split setups. The discussion explains why prefill cost grows quadratically with prefix length. It also explains why a mixture-of-experts model, which is fast to decode but slow to prefill, benefits most from the approach. The central finding is that a closed-form least-squares fit can match the target's cache closely yet generate badly, and in some benchmarks up to 96 percent of answers were unparseable. The fix is a second stage of self-distillation, where the target model's own next-token distribution on its native cache supervises the translated cache through a KL loss. The episode also covers head-wise versus head-mixing maps and cross-layer translation. It questions what the headline speedups, up to 9.6x on a laptop chip and 29x at long context, actually leave out of the comparison. Listeners get a clear walkthrough of KV-cache mechanics and a reason to distrust reconstruction-based objectives.
Interactive Visualization: KV-Lingo: Translating KV Caches Between Frozen Models
Sources:
1. KV-Lingo: Learning KV-Cache Translators with Distillation — Valérie Castin, Keitaro Sakamoto, Anastasiia Filippova, João Monteiro, Marco Cuturi, Pierre Ablin, 2026
http://arxiv.org/abs/2609.32610
2. KV-Lingo: Learning KV-Cache Translators with Distillation — Valérie Castin, Keitaro Sakamoto, Anastasiia Filippova, João Monteiro, Marco Cuturi, Pierre Ablin (Apple), 2026
https://scholar.google.com/scholar?q=KV-Lingo%3A+Learning+KV-Cache+Translators+with+Distillation
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
4. Model stitching / relative representations line of work (for example 'Revisiting Model Stitching to Compare Neural Representations' by Bansal, Nakkiran, Barak, and 'Relative representations enable zero-shot latent space communication' by Moschella et al.) — Yamini Bansal, Preetum Nakkiran, Boaz Barak (2021); Luca Moschella et al. (2023), 2021 / 2023
https://scholar.google.com/scholar?q=Model+stitching+%2F+relative+representations+line+of+work+%28for+example+%27Revisiting+Model+Stitching+to+Compare+Neural+Representations%27+by+Bansal%2C+Nakkiran%2C+Barak%2C+and+%27Relative+representations+enable+zero-shot+latent+space+communication%27+by+Moschella+et+al.%29
5. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — Lingjiao Chen, Matei Zaharia, James Zou, 2023
https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance
6. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica et al., 2024 (ICLR 2025)
https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data
7. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023
https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding
8. R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing — Tianyu Fu, Yi Ge, Yichen You, Enshu Liu, Zhihang Yuan, Guohao Dai, Shengen Yan, Huazhong Yang, Yu Wang, 2025
https://scholar.google.com/scholar?q=R2R%3A+Efficiently+Navigating+Divergent+Reasoning+Paths+with+Small-Large+Model+Token+Routing
9. Cache-to-Cache: Direct Semantic Communication Between Large Language Models — Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, Yu Wang, 2026
https://scholar.google.com/scholar?q=Cache-to-Cache%3A+Direct+Semantic+Communication+Between+Large+Language+Models
10. Latent Space Communication via KV Cache Alignment — Lucio M. Dery, Zohar Yahav, Henry Prior, Qixuan Feng, Jiajun Shen, Arthur Szlam, 2026
https://scholar.google.com/scholar?q=Latent+Space+Communication+via+KV+Cache+Alignment
11. See What I See, Know What I Think: Dense Latent Communication across Heterogeneous Agents — Siyi Chen, Xiaoyan Zhang, Meng Wu, et al., 2026
https://scholar.google.com/scholar?q=See+What+I+See%2C+Know+What+I+Think%3A+Dense+Latent+Communication+across+Heterogeneous+Agents
12. DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — Yuhan Liu et al., 2024
https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+Cache+Sharing+for+Cross-LLM+Communication+and+Multi-LLM+Serving
13. Revisiting Model Stitching to Compare Neural Representations — Yamini Bansal, Preetum Nakkiran, Boaz Barak, 2021
https://scholar.google.com/scholar?q=Revisiting+Model+Stitching+to+Compare+Neural+Representations
14. Transferring Linear Features Across Language Models with Model Stitching — Alan Chen, Jack Merullo, Alessandro Stolfo, Ellie Pavlick, 2025
https://scholar.google.com/scholar?q=Transferring+Linear+Features+Across+Language+Models+with+Model+Stitching
15. Similarity and Matching of Neural Network Representations — Adrián Csiszárik, Péter Kőrösi-Szabó, Ákos Matszangosz, Gergely Papp, Dániel Varga, 2021
https://scholar.google.com/scholar?q=Similarity+and+Matching+of+Neural+Network+Representations
16. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan-Kelley, 2024
https://scholar.google.com/scholar?q=Reducing+Transformer+Key-Value+Cache+Size+with+Cross-Layer+Attention
17. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Saul Ehrlich, Simran Arora, et al., 2026
https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study
18. Stochastic KV Routing: Enabling Adaptive Depth-wise Cache Sharing — Anastasiia Filippova, David Grangier, Marco Cuturi, Joao Monteiro, 2026
https://scholar.google.com/scholar?q=Stochastic+KV+Routing%3A+Enabling+Adaptive+Depth-wise+Cache+Sharing
Interactive Visualization: KV-Lingo: Translating KV Caches Between Frozen Models
1. KV-Lingo: Learning KV-Cache Translators with Distillation — Valérie Castin, Keitaro Sakamoto, Anastasiia Filippova, João Monteiro, Marco Cuturi, Pierre Ablin, 2026
http://arxiv.org/abs/2609.32610
2. KV-Lingo: Learning KV-Cache Translators with Distillation — Valérie Castin, Keitaro Sakamoto, Anastasiia Filippova, João Monteiro, Marco Cuturi, Pierre Ablin (Apple), 2026
https://scholar.google.com/scholar?q=KV-Lingo%3A+Learning+KV-Cache+Translators+with+Distillation
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
4. Model stitching / relative representations line of work (for example 'Revisiting Model Stitching to Compare Neural Representations' by Bansal, Nakkiran, Barak, and 'Relative representations enable zero-shot latent space communication' by Moschella et al.) — Yamini Bansal, Preetum Nakkiran, Boaz Barak (2021); Luca Moschella et al. (2023), 2021 / 2023
https://scholar.google.com/scholar?q=Model+stitching+%2F+relative+representations+line+of+work+%28for+example+%27Revisiting+Model+Stitching+to+Compare+Neural+Representations%27+by+Bansal%2C+Nakkiran%2C+Barak%2C+and+%27Relative+representations+enable+zero-shot+latent+space+communication%27+by+Moschella+et+al.%29
5. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — Lingjiao Chen, Matei Zaharia, James Zou, 2023
https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance
6. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica et al., 2024 (ICLR 2025)
https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data
7. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023
https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding
8. R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing — Tianyu Fu, Yi Ge, Yichen You, Enshu Liu, Zhihang Yuan, Guohao Dai, Shengen Yan, Huazhong Yang, Yu Wang, 2025
https://scholar.google.com/scholar?q=R2R%3A+Efficiently+Navigating+Divergent+Reasoning+Paths+with+Small-Large+Model+Token+Routing
9. Cache-to-Cache: Direct Semantic Communication Between Large Language Models — Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, Yu Wang, 2026
https://scholar.google.com/scholar?q=Cache-to-Cache%3A+Direct+Semantic+Communication+Between+Large+Language+Models
10. Latent Space Communication via KV Cache Alignment — Lucio M. Dery, Zohar Yahav, Henry Prior, Qixuan Feng, Jiajun Shen, Arthur Szlam, 2026
https://scholar.google.com/scholar?q=Latent+Space+Communication+via+KV+Cache+Alignment
11. See What I See, Know What I Think: Dense Latent Communication across Heterogeneous Agents — Siyi Chen, Xiaoyan Zhang, Meng Wu, et al., 2026
https://scholar.google.com/scholar?q=See+What+I+See%2C+Know+What+I+Think%3A+Dense+Latent+Communication+across+Heterogeneous+Agents
12. DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — Yuhan Liu et al., 2024
https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+Cache+Sharing+for+Cross-LLM+Communication+and+Multi-LLM+Serving
13. Revisiting Model Stitching to Compare Neural Representations — Yamini Bansal, Preetum Nakkiran, Boaz Barak, 2021
https://scholar.google.com/scholar?q=Revisiting+Model+Stitching+to+Compare+Neural+Representations
14. Transferring Linear Features Across Language Models with Model Stitching — Alan Chen, Jack Merullo, Alessandro Stolfo, Ellie Pavlick, 2025
https://scholar.google.com/scholar?q=Transferring+Linear+Features+Across+Language+Models+with+Model+Stitching
15. Similarity and Matching of Neural Network Representations — Adrián Csiszárik, Péter Kőrösi-Szabó, Ákos Matszangosz, Gergely Papp, Dániel Varga, 2021
https://scholar.google.com/scholar?q=Similarity+and+Matching+of+Neural+Network+Representations
16. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan-Kelley, 2024
https://scholar.google.com/scholar?q=Reducing+Transformer+Key-Value+Cache+Size+with+Cross-Layer+Attention
17. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Saul Ehrlich, Simran Arora, et al., 2026
https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study
18. Stochastic KV Routing: Enabling Adaptive Depth-wise Cache Sharing — Anastasiia Filippova, David Grangier, Marco Cuturi, Joao Monteiro, 2026
https://scholar.google.com/scholar?q=Stochastic+KV+Routing%3A+Enabling+Adaptive+Depth-wise+Cache+Sharing
Interactive Visualization: KV-Lingo: Translating KV Caches Between Frozen Models