Gist Tokens: Compressing Prompts Up to 26x

Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman (Stanford, 2023; NeurIPS 2023)
arXiv:2304.08467
1 tokenof cached state per prompt
≈26 ÷ 1ratio of averages
45.8LLaMA win rate, Human split
0.11%FLOPs saved vs KV cache

The trick, step by step

Sequence: prompt → gist tokens → user input. Step through what the mask changes.

What gets stored per reused prompt

Reported: ≈1 MB per cached token in fp32 for LLaMA-7B → 26-token prompt ≈ 27 MB, one gist token ≈ 1 MB.

Hover a bar. The 26x storage saving assumes every prompt is about 26 tokens.

Attention mask explorer

Rows = query tokens, columns = keys. Red × marks positions that are in the past but masked by gisting.

gist tokens k = 2
Hover a cell to see what that query token can attend to.

Reported results

Judge score: ChatGPT-3.5 picks the better of compressed vs uncompressed output; 50 = as good as uncompressed.

Where does 26x come from?Illustrative zones

Compression = average prompt length ÷ number of gist tokens. Slide the prompt length; zones are conceptual.

gist tokens:

Where the paper shows it breaks

Two reported failure examples.

Landscape of related approaches

Filter by how the compressed prompt is obtained.

References

  1. Mu, Li, Goodman — Learning to Compress Prompts with Gist Tokens (2023)
  2. Lester, Al-Rfou, Constant — The Power of Scale for Parameter-Efficient Prompt Tuning (2021)
  3. Li, Liang — Prefix-Tuning (2021)
  4. Chevalier et al. — Adapting Language Models to Compress Contexts (2023)
  5. Ge et al. — In-context Autoencoder for Context Compression (2023)
  6. Kim et al. — Compressed Context Memory for Online LM Interaction (2023)
  7. Xiao et al. — Efficient Streaming Language Models with Attention Sinks (2023)
  8. Beltagy, Peters, Cohan — Longformer (2020)
  9. Jiang et al. — LLMLingua (2023)
  10. Rae et al. — Compressive Transformers (2020)
  11. Snell, Klein, Zhong — Learning by Distilling Context (2022)
  12. Wingate, Shoeybi, Sorensen — Prompt Compression and Contrastive Conditioning (2022)
  13. Kwon et al. — PagedAttention / vLLM (2023)
  14. Ainslie et al. — GQA (2023)
  15. Zheng et al. — Judging LLM-as-a-Judge with MT-Bench (2023)