Trimming a long document down to 'just the relevant parts' can backfire by wrecking the cost savings of reusing cached work.

This explores a trade-off between two ways of handling long inputs: cutting them down to only the relevant parts for each request, or keeping them stable so the provider can reuse cached work across requests. The corpus has no paper that studies this collision directly, but several notes cover each side, and together they explain the tension. One piece comes from how prompt caching works rather than from the corpus. Providers generally reuse cached work only when the start of a new prompt matches, token for token, the start of one they have already processed. Change something early in the prompt and everything after it has to be computed again.

The cost side is clear. A benchmark across OpenAI, Anthropic and Google found that caching cut costs for multi-turn agent tasks by 41–80% and made the first token arrive 13–31% sooner. It also found that naive full-context caching can make latency worse, because it caches content that changes and will never be reused Does prompt caching actually reduce agentic LLM costs?. That is the core of the answer. Caching pays off only for the parts of a prompt that stay the same. Trimming for relevance turns stable material into material that changes with every question, because each query keeps a different slice of the long input. Once that slice differs, the prompts no longer share a starting sequence, so there is nothing to reuse.

Prompt compression shows the same pattern. LLMLingua shrinks prompts up to 20x with little accuracy loss by scoring each token and pruning those a small model considers least useful, while compressing instructions and questions less than examples Can prompts be compressed without losing model performance?. The result depends on the question, so the compressed prompt changes whenever the question does. That is the opposite of what a cache can use. You can save tokens or save cached compute, but rarely both on the same content unless you deliberately split the prompt into a fixed part and a changing part.

The less obvious point is that the field is partly moving away from trimming. LongRAG argues that the old design of precise retrieval feeding a small reader was a leftover from short context windows. Feeding long, coarse chunks to a long-context reader now works better Can long-context models resolve retriever-reader imbalance?. Long-context models can also match standard RAG on meaning-based lookup, though they still fail at database-style queries that join tables Can long-context LLMs replace retrieval-augmented generation systems?. If you can afford to keep a large, stable context, it becomes cheap to reuse. The catch is that reasoning gets worse as inputs grow, even far below the context limit Does reasoning ability actually degrade with longer inputs?. So trimming saves money per call and keeps the model sharper, while a stable full context saves money across calls.

Two designs in the corpus try to get both. Recursive Language Models keep the long prompt outside the model in a code environment and query it with code Can models treat long prompts as external code environments?. The Thread Inference Model prunes its internal cache with fixed rules as it works through subtasks, instead of rebuilding the prompt each time Can recursive subtask trees overcome context window limits?. Both deal with relevance after the input is in place rather than by changing the input. That suggests a design lesson: put fixed material first, add the changing, relevance-picked material at the end, and let the model choose what to use instead of rewriting the input for every request.