Skip the re-read
Switch models over a shared prefix: re-prefill, or translate the cache. Pick a scenario and a mode.
Why model B can't read model A's cache
Each cell is one (layer, token) key vector summarized to a single value. Hover for details.
Prefill grows quadratically; translation stays linear
Illustrative curves shaped like the discussion (8B target; ~1.4 s at 32k, ~48 ms translate + step). Not paper data points.
Two-stage training
Stage 1 fits a closed-form least-squares map. Stage 2 distills the target's own next-token distribution through the translated cache.
Reconstruction charges for everything; distillation only for what matters
Mock per-token data. Importance = how much the target's output depends on that cache entry.
Head-wise vs head-mixing
Rows = target KV heads, columns = source KV heads (8 heads, d=128).
Cross-layer: ν source layers per target layer
Hover a target layer. Example depths 28 → 36.
Multi-turn: translation depth per segment
Computed from the rule. Naive re-translates the whole prefix at every switch; alternation translates only new tokens.
What the headline speedup leaves out
Time to first token on the target. Toggle the setup, then add the source model's prefill back in.
Speedup vs context length (H100, vLLM)
Paper: ~5× at 2k, 12–29× at 32k (target-dependent), source prefill excluded. The ~1.2× point is the host's own rough estimate, not a paper result.
Does the big model act big?
Illustrative scores on a 0–1 scale. Paper reports gaps: 0.6B→8B beats the small model by ~7–9 points but stays below native.
Where a cache translator earns its place
Fit meter is an editorial reading of the discussion, not a measurement. Hover for the reason.
References
- KV-Lingo: Learning KV-Cache Translators with Distillation — Castin et al., 2026 (arXiv 2609.32610)
- PagedAttention — Kwon et al., 2023
- Revisiting Model Stitching — Bansal, Nakkiran, Barak, 2021
- Relative representations — Moschella et al., 2023
- FrugalGPT — Chen, Zaharia, Zou, 2023
- RouteLLM — Ong et al., 2024
- Speculative Decoding — Leviathan, Kalman, Matias, 2023
- R2R — Fu et al., 2025
- Cache-to-Cache — Fu et al., 2026
- Latent Space Communication via KV Cache Alignment — Dery et al., 2026
- Dense Latent Communication across Heterogeneous Agents — Chen et al., 2026
- DroidSpeak — Liu et al., 2024
- Transferring Linear Features via Model Stitching — Chen et al., 2025
- Similarity and Matching of NN Representations — Csiszárik et al., 2021
- Cross-Layer Attention — Brandon et al., 2024
- Cartridges — Eyuboglu et al., 2026
- Stochastic KV Routing — Filippova et al., 2026