If an AI tool compresses your prompt using one 'word-splitter' and your model reads with another, does the compression still work as advertised?
How much does tokenizer mismatch affect prompt compression estimates?
This explores what happens when the tokenizer used to measure or perform prompt compression, often belonging to a small helper model, isn't the same one the target LLM uses, and whether that gap makes compression ratios and accuracy claims less reliable.
This explores whether prompt compression numbers still hold when the model doing the compressing splits text into tokens differently from the model that receives the compressed prompt. The short answer is that the collection doesn't measure this directly. No note here quantifies how much tokenizer mismatch shifts compression estimates. What the collection does have is a clear picture of why the mismatch matters, plus some close neighbors that show where the risk sits.
The main entry point is LLMLingua Can prompts be compressed without losing model performance?. It reports up to 20x compression with only about a 1.5-point accuracy drop. It works by having a small model score each token by perplexity (roughly, how surprising that token is) and then deleting the least surprising ones. The important detail is that the small model has to be *aligned* to the target LLM before its scores can be trusted. That alignment step amounts to admitting that the small model's sense of which tokens are "predictable" can differ from the big model's. If the two models also cut text into different units, a "20x" ratio counted in one model's tokens may not mean 20x in the other's. And the tokens marked as safe to delete may not be the ones the target model actually finds redundant.
A useful way to see why this matters: compression is really about predictability, and predictability depends on the model. One note shows that language modeling is mathematically the same thing as lossless compression Can text-trained models compress images better than specialized tools?. A model compresses well exactly where it predicts well. So how compressible a prompt is depends on which model you ask, and the tokenizer is part of that model. The Byte Latent Transformer Can byte-level models match tokenized performance with better efficiency? takes this further. It drops fixed tokenizers entirely and groups raw bytes into chunks by how predictable they are. That makes it more robust to typos and better at moving between languages, which suggests fixed tokenizers bake in assumptions that don't carry over cleanly from one model to another.
There's also a hint that even small surface changes matter more than you'd expect. Prompts that mean the same thing but are worded differently produce measurably different outputs, depending on how common each phrasing was in training data Why do semantically identical prompts produce different LLM outputs?. A compressed prompt is an extreme rewording: grammatically broken, with function words removed. If wording alone shifts results, the output of a compressor built on a different tokenizer is a plausible source of hidden accuracy loss that a headline ratio wouldn't show.
The less obvious takeaway: a compression ratio isn't a property of the prompt. It's a property of the prompt *and* a particular model, and the token is the unit of measurement. The collection also points to approaches that avoid the question altogether. Recursive Language Models Can models treat long prompts as external code environments? leave long prompts uncompressed and let the model query them with code. If you want a measured answer to how big the mismatch effect is, this collection doesn't have it yet.
Sources 5 notes
LLMLingua achieves up to 20x prompt compression with only 1.5-point performance drops by allocating less compression to instructions and questions, more to demonstrations, then pruning tokens using a small model's perplexity scoring aligned to the target LLM.
Chinchilla models trained exclusively on text achieve better compression rates on images and audio than FLAC and PNG by using their context window to adapt as task-specific compressors. This demonstrates that generalization operates through compression, not specialization.
The Byte Latent Transformer (BLT) dynamically segments bytes into patches based on next-byte entropy, allocating more compute to high-entropy regions and less to predictable ones. At 8B parameters, BLT matches tokenized baselines while reducing inference cost and improving robustness to typos and cross-lingual transfer.
Cao et al. and Adam's Law show that semantically identical prompts with different sentence-level frequencies produce systematically different output quality. Higher-frequency phrasings win because models register statistical mass from pre-training, not meaning.
Recursive Language Models store long prompts in a Python REPL and query them via code execution, avoiding attention degradation. RLMs outperform base models even on shorter prompts while handling inputs two orders of magnitude beyond context windows.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- ACON: Optimizing Context Compression for Long-horizon LLM Agents
- Recursive Language Models
- Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
- Byte Latent Transformer: Patches Scale Better Than Tokens
- Language Modeling is Compression