Sequence: prompt → gist tokens → user input. Step through what the mask changes.
Reported: ≈1 MB per cached token in fp32 for LLaMA-7B → 26-token prompt ≈ 27 MB, one gist token ≈ 1 MB.
Rows = query tokens, columns = keys. Red × marks positions that are in the past but masked by gisting.
Judge score: ChatGPT-3.5 picks the better of compressed vs uncompressed output; 50 = as good as uncompressed.
Compression = average prompt length ÷ number of gist tokens. Slide the prompt length; zones are conceptual.
Two reported failure examples.
Filter by how the compressed prompt is obtained.