Spectral-LSH: сжатие промптов до 16× без дообучения для ускорения LLM-инференса

Researchers have introduced Spectral-LSH — a compression method for long prompts for large language models that works without fine-tuning and shrinks the input text by 8–16×, preserving quality where simple methods lose it. The preprint appeared on arXiv (number 2607.19368) in July 2026.

Why compress prompts

Long prompts are expensive to process because attention costs at the prefill stage grow quadratically — as O(N²) of sequence length. Spectral-LSH addresses this before the prompt even reaches the model: it finds tokens that are close in meaning and merges them into "macro-tokens," shrinking the input length before computation even begins.

Key facts from the preprint:

• The method is training-free — it does not require fine-tuning the model
• It works as a preprocessor, before input reaches the LLM
• It was tested on Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct, and Qwen2.5-14B-Instruct
• The evaluation dataset is C4
• Compression ratios ρ were tested in the range from 4× to 16×

How the method works

Spectral-LSH approximates the principal components of the implicit attention-kernel operator using the Krylov subspace method together with random features. This makes it possible to avoid explicitly constructing the O(N²)-sized attention matrix — the most expensive part when handling long context. Then, in the resulting "spectral" attention space, SimHash is applied: similar tokens are hashed into the same buckets and aggregated into macro-tokens while preserving the causal order of positions, so the model does not lose the sequence structure.

Where the phase transition occurs

The main result is a phase transition with respect to the compression ratio, which the authors describe. Below ρ=4× the local redundancy of tokens is low, and light chunking (simple splitting into blocks) gives the best balance of speed and quality. Above ρ=8× the spectral path begins to preserve quality that chunking loses.

At ρ=16× the gap is especially pronounced: according to the preprint, Qwen2.5-7B in adaptive mode reduces the perplexity (PPL) coefficient from 353,409 to 196,963, and Qwen2.5-14B — from 9,533 to 3,427. The lower the perplexity, the less the model's prediction is distorted after compression.

"Our experiments reveal a phase transition with respect to the

compression ratio," the study's abstract on arXiv states.

What the stress test showed

In a small stress test with long context made of structured data — JSON-style fragments, code, and tables — local LSH improved every metric compared to chunking at 8× compression. The adaptive backend combines both modes: chunking at low compression and spectral clustering at high compression. At the same time, chunking remains the fastest in terms of end-to-end latency — quality at high ratios comes at the cost of time.

What this means

Spectral-LSH shows that aggressive prompt compression (8× and above) can be done without loss of quality and without fine-tuning the model — if tokens are grouped by attention spectrum rather than the text being cut into blocks. For long-context inference, this is a direct path to lower compute costs.

Frequently asked questions

What is Spectral-LSH?

It is a training-free method for compressing input prompts for language models: it groups similar tokens via SimHash hashing in the spectral attention space and merges them into macro-tokens, shrinking the input length by 4–16×. No model fine-tuning is required.

Which models was the method tested on?

The authors evaluated Spectral-LSH on Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct, and Qwen2.5-14B-Instruct on the C4 dataset, as well as on a stress test with structured data in JSON-style, code, and tables.