Can prompts be compressed without losing model performance?

LLMLingua is a "coarse-to-fine prompt compression method" built to cut the cost of inference when prompts have grown "increasingly lengthy, even exceeding tens of thousands of tokens" from chain-of-thought and in-context learning. Tested on GSM8K, BBH, ShareGPT, and Arxiv-March23, it "achieves state-of-the-art performance across all datasets, with up to 20x compression with only a 1.5 point performance drop," and the paper reports LLMs "can effectively restore compressed prompts" from the output.

The mechanism has three parts. A budget controller first splits the compression ratio unevenly across a prompt's instruction, demonstrations, and question, on the reasoning that "the instruction and the question in a prompt have a direct influence on the generated results" while "multiple demonstrations... may be redundant" — so demonstrations absorb more compression, instructions and questions less. At the demonstration level this is coarse-grained: a small model such as GPT-2 or LLaMA scores each demonstration's perplexity and lower-information ones are dropped whole, preserving "a certain degree of linguistic integrity" rather than discarding scattered tokens. A second, token-level pass then prunes within what remains, modeling "the interdependence between compressed contents" to improve on Selective-Context's independence assumption, which the paper likens to "the shortcomings of the Mask Language Model." Because the small scoring model and the target black-box LLM see language differently, an instruction-tuning step aligns the small model's distribution to the target's — all "without any gradient flow through the LLMs," so the method works through APIs alone.

This sits alongside other token-economy approaches already in the vault, but compresses a different object. Can minimal reasoning chains match full explanations? shrinks the tokens a model generates during reasoning; LLMLingua shrinks the tokens fed in before generation starts — complementary targets for the same inference-cost problem. Can models think longer by forgetting intermediate reasoning? shares LLMLingua's premise that token informativeness is uneven, but applies it to live generation memory rather than a fixed prompt. And where Can thinking traces be made reliably budget-controllable? reaches for RL reward shaping to keep compression controllable, LLMLingua gets budget control from an explicit allocation rule and perplexity scoring instead, with no reward signal or fine-tuning of the target model at all.

The excerpt gives compression ratios and accuracy-point drops, not dollar costs or latency numbers — the inference-cost savings are an inference from fewer tokens being sent to a metered API, not a measured cost figure. It also names a real ceiling: "a notable performance drop when trying to achieve excessively high compression ratios such as 25x-30x on GSM8K," and flags that mismatched tokenizers between the small and target models "may result in an underestimation of the prompt's token length." So the 20x figure should be read as a validated-but-task-dependent ceiling, not a general compression rate, and its cost benefit is strongest where API pricing scales directly with input tokens.