LLMLingua: Compressing Prompts with Small-Model Token Pruning

Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu · Microsoft · EMNLP 2023

arXiv 2310.057365×–20× prompt compression

The four-step pipeline

Click a step to light it up. A small local LM edits the text; the big model stays a black box.

Budget controller: where the cuts landIllustrative sizes

Only the 2,366 total and the 85% / 90% keep rates are reported. Instruction and question sizes (40 tokens each) and the demo budget arithmetic are a schematic, not paper data.

Compression ratio20×

Instruction (keeps 85%)DemonstrationsQuestion (keeps 90%)

Surprisal = code length

Surprisal of a token is −log p(token | prefix). Slide the probability and watch the bits.

p(token | prefix)

Perplexity = exp(average surprisal) over a text. Predictable tokens cost few bits (droppable); rare names and numbers cost many.

Pruning by thresholdIllustrative values

Per-token surprisal values below are invented for illustration. Hover a token; raise the threshold to drop more.

Drop tokens below

Hover a token.

Why independent scoring failsIllustrative

One recurring phrase in all eight demonstrations. Toggle how it is scored.

Reported results

Pick a view. Values from the episode; "~" marks figures given as approximate.

How many questions is a difference?

GSM8K test set: 1,319 questions, one dot each. Single greedy run, no seeds, no significance test.

The denominator matters

Every ratio is measured against hand-picked 8-shot prompts. Prompt lengths in tokens.

Hand-picked 1-shot (422 tok) scores 77.10; the method at 446 tok scores 79.08: about 2 points (~26 questions) with no compressor. Much of "20×" is dropping redundant demonstrations.