Spectral-LSH: Up to 16× Prompt Compression Without Fine-Tuning to Speed Up LLM Inference

Researchers have introduced Spectral-LSH, a method for compressing long prompts for large language models. It works without fine-tuning and shrinks the input text 8–16×, keeping quality where simple methods lose it. The preprint appeared on arXiv (number 2607.19368) in July 2026.

Why compress prompts

Long prompts are expensive to process because the attention cost during the prefill stage grows quadratically, as O(N²) in sequence length. Spectral-LSH tackles this before the prompt ever reaches the model: it finds tokens that are close in meaning and merges them into "macro-tokens," cutting the input length before any heavy computation begins.

Key facts from the preprint:

• The method is training-free: it requires no fine-tuning of the model
• It works as a preprocessor, ahead of the LLM input
• It was tested on Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct
• The evaluation dataset is C4
• Compression ratios ρ were tested from 4× to 16×

How the method works

Spectral-LSH approximates the principal components of the implicit attention-kernel operator using the Krylov subspace method together with random features. This avoids explicitly building the O(N²) attention matrix, the most expensive part of long-context processing. Then SimHash is applied in the resulting "spectral" attention space: similar tokens are hashed into the same buckets and aggregated into macro-tokens, with the causal order of positions preserved so the model does not lose the structure of the sequence.

Where the phase transition lies

The main result is a phase transition in the compression ratio, as the authors describe it. Below ρ = 4×, local token redundancy is low, and lightweight chunking (simply splitting the text into blocks) gives the best balance of speed and quality. Above ρ = 8×, the spectral path starts to preserve quality that chunking loses.

At ρ = 16× the gap is especially visible. According to the preprint, in adaptive mode Qwen2.5-7B cuts its perplexity (PPL) from 353.409 to 196.963, and Qwen2.5-14B from 9.533 to 3.427. The lower the perplexity, the less the model's predictions are distorted after compression.

"Our experiments reveal a phase transition in the compression ratio,"

the study's abstract on arXiv states.

What the stress test showed

In a small long-context stress test built from structured data (JSON-style fragments, code and tables), local LSH improved every metric over chunking at 8× compression. The adaptive backend combines both modes: chunking at low compression and spectral clustering at high compression. Chunking nonetheless remains the fastest in total latency, so the quality gain at high ratios costs time.

What this means

Spectral-LSH shows that aggressive prompt compression (8× and above) can be done without quality loss and without fine-tuning the model, provided tokens are grouped by attention spectrum rather than by cutting the text into blocks. For long-context inference, this is a direct route to lower compute costs.

Frequently asked questions

What is Spectral-LSH?

It is a training-free method for compressing input prompts for language models. It groups similar tokens via SimHash hashing in attention's spectral space and merges them into macro-tokens, shortening the input 4–16×. No model fine-tuning is required.

Which models was the method tested on?

The authors evaluated Spectral-LSH on Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct using the C4 dataset, and also on a stress test with structured data in the form of JSON-style fragments, code and tables.