Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

Compressing a prompt 7x cut reasoning accuracy from 70% to 57%

Long prompts are often trimmed by eye, which is risky. In the LLMLingua study, 7x compression on reasoning tasks (BBH) cut the share of correct answers from 70% to 57%, while on math (GSM8K) even 20x compression cost only 1.5 points. If the express-delivery exception drops out of a shipping instruction, the answer will look plausible but the price will be wrong. That is why every shortened prompt should be tested on a set that includes rare cases.

AI-processed from Habr AI; edited by Hamidun News
Compressing a prompt 7x cut reasoning accuracy from 70% to 57%
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Selective Contextcutting context by 50% reduced time by 32% and memory by 36%
LLMLingua-2the cycle from compression to answer is 1.6–2.9 times faster
Key metric #3Cutting input by 40% of tokens lowers the cost of a request by only about 18%

Prompts usually grow the same way: first the task, then a role, constraints, a couple of examples, and one more paragraph "just in case." Half a page later, deleting anything feels dangerous: that very phrase might be holding the whole logic together. Research shows that text really can be cut severalfold, but the outcome depends on the task.

When LLMLingua compressed a prompt 7x on the BBH set (reasoning tasks), the share of answers matching the reference fell from 70.07% to 56.85%, a drop of 13.

22 percentage points. On the GSM8K math set, even 20x compression cost only 1.52 points.

Imagine an instruction for calculating delivery costs: if trimming drops the line about the express exception, the model's answer will still look correct, but the price will be wrong. So a prompt should only be shortened together with a test.

What the studies showed

In Selective Context (EMNLP 2023), an auxiliary model estimates how unexpected each fragment of text looks to it and removes the most predictable ones. When context was cut by 50%, memory use fell by 36% and request processing time (inference) by 32%. The BERTScore similarity-to-reference metric dropped by 0.023, and the faithfulness score (fidelity to the source text) by 0.038. A text half as long is processed about a third faster, with a small loss on these metrics.

LLMLingua (EMNLP 2023) is more aggressive: a small model picks which parts of the prompt to keep and distributes the token budget among the instruction, the examples and the question. The test ran on GPT-3.5-Turbo-0301, and quality was measured as the share of answers that exactly matched the reference (Exact Match). On GSM8K, the prompt was compressed from 2366 to 446 tokens (5x), and the result even rose slightly: from 78.85% to 79.08%. At compression to 117 tokens (20x), it fell to 77.33%. On BBH, compression from 774 to 110 tokens (7x) sent the result down from 70.07% to 56.85%. The same technique behaves differently on different tasks.

LLMLingua-2 (Findings of ACL 2024) was trained to decide, for each token, whether to keep or delete it, and the model looks at context on both sides. GPT-4 prepared the training data, and evaluation covered five benchmarks, including MeetingBank and LongBench. The compressor itself runs 3–6 times faster than the methods it was compared with, and the whole cycle from compression to answer speeds up by 1.6–2.9 times when the prompt is shortened 2–5x.

Where compression breaks meaning

The words that remain come from the original, but if a negation or part of a condition disappears, the rule changes. The short version "Delivery is free on orders over 5000 rubles" loses the clarification that this applies only to standard delivery, while express costs 790 rubles at any order total. A single correct answer will not reveal such a loss: you need an order with express. Text length and task accuracy must be checked separately.

The author of the analysis proposes a method. Take three versions of the instruction: the original, a compact statement of the same conditions, and a dense one in which words are replaced with arrows and a ternary expression. Run them on GPT-4.

1 Mini via BotHub: 12 orders, three repeats for each version, 108 requests in total at temperature=0, top_p=1 and max_tokens=512. The order of requests is shuffled, and each one runs in a new chat without history. An answer counts only if the JSON is valid, the status matches and the price is correct.

The prompt texts are fixed before the run so they are not tuned to familiar answers. The article contains no final results; it is purely a method. Even 36 correct answers out of 36 on 12 cases will not show the future error rate, so the set should be expanded with real orders.

How to calculate savings honestly

Input and output tokens are priced differently. Example: a 2000-token prompt versus 1200, with a 600-token answer in both cases and an output token four times as expensive as an input token. That comes to 4400 conventional units versus 3600. Input shrank by 40%, but the cost of the request fell by only about 18%. If the shorter instruction triggers a longer answer or a repeat request, the gain is smaller still.

The calculation should also include the compressor's work: if the context is new for every request, compression is paid for each time. Count the time spent on manual editing and on testing the template, as well as the cache: a long prefix served from cache can cost less than a fresh short one. The final metric is the cost of an accepted result: the cost of all requests plus the cost of compression, divided by the number of answers that passed the check. Failed attempts go in the numerator.

In practice, do shorten your prompt, but with a test: the same data, with exceptions and missing fields in the set, one model, fixed settings. Compare four outcomes pairwise: both versions succeeded, only the original, only the short one, both failed. A version that loses a rule is unfit, no matter how many tokens it saves.

Story

Key Questions Answered

What the studies showed?

In Selective Context (EMNLP 2023), an auxiliary model estimates how unexpected each fragment of text looks to it and removes the most predictable ones. When context was cut by 50%, memory use fell by 36% and request processing time (inference) by 32%. The BERTScore similarity-to-reference metric dropped by 0.023, and the faithfulness score (fidelity to the source text) by 0.038. A text half as long is processed about a third faster, with a small loss on these metrics.

Where compression breaks meaning?

The words that remain come from the original, but if a negation or part of a condition disappears, the rule changes. The short version "Delivery is free on orders over 5000 rubles" loses the clarification that this applies only to standard delivery, while express costs 790 rubles at any order total. A single correct answer will not reveal such a loss: you need an order with express. Text length and task accuracy must be checked separately. The author of the analysis proposes a method.

How to calculate savings honestly?

Input and output tokens are priced differently. Example: a 2000-token prompt versus 1200, with a 600-token answer in both cases and an output token four times as expensive as an input token. That comes to 4400 conventional units versus 3600. Input shrank by 40%, but the cost of the request fell by only about 18%. If the shorter instruction triggers a longer answer or a repeat request, the gain is smaller still.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…