Cartridges++: Compressed KV Memory Without Off-Context Derailment

Sonia Laguna, Joao Monteiro, Marco Cuturi, Pierre Ablin, Eleonora Gualdoni (Apple) · 2026
Episode companion · visuals marked Illustrative are schematics, not measured results.
arXiv 2609.35621 Interactive page

What gets compressed

A document of L tokens becomes m virtual KV positions. Drag ρ = m/L across the paper's swept range (0.1% to 20%).

Three ways to build m positions

Eviction selects, summaries rewrite, learned methods optimize new vectors.

Why a learned memory can derailIllustrative

Hypothesis: optimized vectors can act like attention sinks. The paper tests symptoms, not this mechanism directly.

A one-sided training objective

Self-study distills the teacher only on document queries (Q+). Q− is unconstrained.

The trade-off, as orderingIllustrative

Bar heights are ordinal ranks matching the paper's reported ordering, not measured values. The paper's plots put off-context accuracy in the 60s–80s % and interference up to about 20%.

How it is measured

Two sides: Q+ context fidelity, Q− capability preservation. Hover a row.

Fix 1 · C_D++ data mixingIllustrative curves

γ = fraction of supervised assistant tokens from Dolci-Instruct-SFT, taught against the no-Cartridge base model. Paper: 1% already helps; 5% chosen.

Fix 2 · C_R++ routing

The trained Cartridge is untouched; each query is accepted or rejected.

accepts >90% on-contextrejects ≈89–92% off-contextconformal α = 0.05kNN k = 10

Threshold calibrationIllustrative

Threshold set so ≈5% of Dolci-like negatives are falsely accepted. Move it to see the trade.

Where the evidence stops

Tested zones are solid; untested zones are dashed. Click a zone.

What to ship

References

  1. Cartridges++: KV Cache Compression without Off-Context Derailment — Laguna et al., 2026
  2. Cartridges — Eyuboglu et al., 2025
  3. Overcoming catastrophic forgetting — Kirkpatrick et al., 2017
  4. LoRA Learns Less and Forgets Less — Biderman et al., 2024
  5. LLMs Can Be Easily Distracted by Irrelevant Context — Shi et al., 2023
  6. Fast KV Compaction via Attention Matching — Zweiger et al., 2026
  7. Cartridges at Scale — Hardalov et al., 2026
  8. Prefix-Tuning — Li, Liang, 2021
  9. Attention Sinks — Xiao et al., 2024
  10. Conformal Risk Control — Angelopoulos et al., 2024
  11. Lost in the Middle / Distraction — Liu et al.; Shi et al., 2023
  12. Plug-and-play knowledge modules — Caccia et al., 2025
  13. Doc-to-LoRA — Charakorn et al., 2026
  14. The pitfalls of KV cache compression — Chen et al., 2026
  15. KV-Distill — Chari et al., 2025