Step through the ICAE pipeline
Encoder = LLM + LoRA. Decoder = the untouched LLM.
Figure 2 idea: same passage, three sizes
Bar lengths are linear in the quantities the episode quotes.
Pick the number of slots k Illustrative colors
Cell counts and ratios are arithmetic (512 / k). Slot colors are decorative, not real activations.
Two-stage training
Only the LoRA weights and memory-token embeddings receive gradients.
Reported results
Numbers are those quoted in the episode. The judge is GPT-4 only.
Hover or tap a bar.
Where does the evidence stop?
Context-length axis is logarithmic. Markers come from the episode's discussion.
Hover or tap a marker.
Multi-span extension: chunk and concatenate
Each 512-token chunk is compressed independently, then slots are concatenated.
Evidence map Qualitative reading
A reading of the hosts' discussion, not a paper-reported score.
Hover or tap a row.
References
- In-context Autoencoder for Context Compression in a Large Language Model — Ge, Hu, Wang, Wang, Chen, Wei (2023)
- Learning to Compress Prompts with Gist Tokens — Mu, Li, Goodman (2023)
- Adapting Language Models to Compress Contexts (AutoCompressors) — Chevalier, Wettig, Ajith, Chen (2023)
- Compressive Transformers for Long-Range Sequence Modelling — Rae et al. (2019)
- Perceiver: General Perception with Iterative Attention — Jaegle et al. (2021)
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al. (2023)
- LLMLingua — Jiang et al. (2023)
- Language Modeling Is Compression — Delétang et al. (2023)
- StreamingLLM — Xiao et al. (2023)
- H2O — Zhang et al. (2023)
- KIVI — Liu et al. (2024)
- PagedAttention (vLLM) — Kwon et al. (2023)
- FlashAttention-2 — Dao (2023)
- xRAG — Cheng et al. (2024)
- In-Context Former / 500xCompressor — Li, Su, Collier (2024)
- Perceiver Resampler / Q-Former — Jaegle; Alayrac; Li et al.
- Prompt Cache / Cartridges — Gim; Eyuboglu et al.